Washington (CNN) Donald Trump is coming to Capitol Hill to rally the GOP troops roughly two weeks before he's slated to accept his party's nomination for president at the GOP convention in Cleveland, but the crowd might be a little thin.

"I've got a long-standing appointment downtown and I don't know at what time I can get out of that and get back to the Hill," Rep. Fred Upton, R-Michigan, told reporters Wednesday.

Colorado GOP Rep. Mike Coffman, one of the House Democrats' top targets in November, told CNN, "I'm not going to be able to make it. I've got a conflict."

Sen. Pat Toomey, who is in a difficult re-election race in Pennsylvania and has not endorsed Trump, shrugged off questions about the meeting Trump has scheduled with at the National Republican Senatorial Committee Thursday.

"I haven't checked my schedule yet," Toomey said Wednesday evening when asked if he'd attend the Trump session. "I don't know."

Asked if he felt the need to hear what his party's nominee had to say, Upton, who hasn't endorsed Trump and isn't planning to attend the convention, said about his own meeting, "This has been set up for some time."

For those who are going, there's a group that is still skeptical, but hopeful they will learn something new, especially since the vast majority of Hill Republicans haven't met Trump personally yet.

Illinois Republican Rep. Adam Kinzinger has publicly criticized Trump but has left open the possibility he could eventually back him and told CNN he's going to the meeting.

"For me, I've always said I want to get there. I'm not there, so if he's going to come talk to us, I'm happy to hear what he has to say," Kinzinger said.

But the Iraq war veteran said that Trump's comments praising Saddam Hussein at a rally in North Carolina are making it harder for him to come around to his candidacy.

"It was ludicrous," Kinzinger said. "I've always had a deep concern that he has a fundamental lack of knowledge about foreign policy and the president has almost unchecked power on foreign policy."

Rep. Duncan Hunter, R-California, one of Trump's earliest supporters in Congress, downplayed those who decided against attending the morning session at the Capitol Hill Club hosted by House GOP Conference Chair Cathy McMorris Rodgers, R-Washington.

"If they are skipping it, they are skipping it -- they've got stuff to do," Hunter said.

After weeks of infighting within their party and lingering fears about Trump's candidacy, some clearly hope that Trump's Thursday meeting will give the party a chance to unite. And Republicans say one thing has helped in that regard -- the common desire to beat Clinton, something many believe is quite possible after her handling of classified intelligence was sharply criticized by the FBI.

Rep. Raul Labrador, an Idaho conservative who has been critical of Trump, said he was more concerned about beating Hillary Clinton than he was worried about Trump.

"We have a very beatable candidate in Hillary Clinton," Labrador said. "When he calls her 'Crooked Hillary,' the FBI just said that it was true. ... So I think we need to make sure that he continues to prosecute the case against Hillary, and I think he can win."

Still, Republican leaders are uneasy with Trump.

House Majority Leader Kevin McCarthy, who is among the California delegates for Trump at the convention, declined to say if he's now more comfortable with Trump as the nominee.

"I always look to constant improvement," McCarthy told reporters.

And Senate Majority Leader Mitch McConnell would not say if Trump should receive classified briefings -- even as Republican leaders say Clinton should now be denied access to highly sensitive intelligence.

"The question here is Hillary Clinton," McConnell said when asked about Trump, "and her public explanations compared to her private representations to the FBI."

Hunter said Trump "needs to be himself" at the meeting and said he believes the chance to talk with members will help the business mogul expand support among those who haven't yet embraced his candidacy.

But when asked about his colleagues worried about what Trump's comments about Hussein, Hunter, who served three combat tours in Iraq and Afghanistan, said he didn't see the comments, so couldn't speak to the context and added, "I'm not a surrogate, I'm not going to answer for him."

Pennsylvania GOP Rep. Charlie Dent said he remains concerned about some of the controversial statements the businessman has made "dealing with issues like POWs, the disabled, to Mexicans, to Muslims, to women, to the David Duke debacle, to the Chris Matthews exchange on abortion, to the Indiana judge, to yesterday and Saddam Hussein."

Initially, Dent told reporters he planned to go, but after a few minutes appeared to be on the fence, saying, "I intend to go, but I could change my mind between now and then."