Washington (CNN) House Speaker Paul Ryan and his leadership team continue to work around the clock to pass their bill to repeal and replace the Affordable Care Act, realizing that they may need to make significant changes to their legislation if they are going to pass it on the House floor as soon as next week.

As of Wednesday night, CNN's own whip count recorded 19 Republican House members against or leaning against the bill as it was written, a rude awakening that Ryan can't ignore. He can only lose 21 of his own members in order to pass the bill in the House without any Democratic support.

Ryan insisted to reporters the bill was still on a path to pass the House and that Republicans have been working "hand in glove" with the White House on the repeal plan.

"We're going to keep our word," Ryan said Wednesday night, dodging specifics about what may be incorporated into the bill in upcoming days.

Ryan called it "premature to get into the conclusion to those things" and said he was aiming for the "sweet spot" that gets the bill to pass.

"We're all collaborating to make sure we get this exactly right," Ryan said.

Pressure is mounting for the speaker and his fellow Republican leaders to hold their party together as members of the Budget Committee prepare to vote on the bill Thursday and send it on its way to the Rules Committee where it could be amended next week.

Under the rules, there are few changes that can be made in the Budget Committee. The job of committee members is to simply package legislation approved last week from the House Ways and Means Committee with language from the House Energy Committee and move it on its way.

That doesn't mean, however, there won't be some fireworks. A handful of Budget panel members, including Reps. Mark Sanford of South Carolina and Dave Brat of Virginia, have been very vocal about their problems with the bill. Any defections at the committee level -- even if the bill ultimately passes -- could be a setback for legislation that's struggling to maintain support.

The top Democrat on the Budget Committee, Rep. John Yarmuth of Kentucky, admitted the bill will still likely pass, but that he thought "it's going to be a close call."

"There are a number Republicans I've spoken to who have very serious problems with the legislation and understand the predicament that they find themselves in, but that doesn't necessarily mean they are going to vote against it," Yarmuth said. "They'll be under a lot of pressure to actually move this bill out of Budget and off to Rules."

Sales pitch

While Ryan works for votes, the White House is working on its sales pitch. A Fox News poll released Wednesday showed that only 34% of Americans backed the GOP's plan to repeal and replace Obamacare.

Trump held a campaign-style rally in Nashville and Tom Price, secretary of Health and Human Services, appeared on a CNN Town Hall, to promote the effort.

"We believe strongly, strongly, that the plan we put together is so much better than the one that's there now," Price said at the CNN event

All the while, Trump continues to say the bill needs work to get past Congress, adding to the uncertainty for lawmakers.

"I think we're going to have negotiation," he told Fox News' Tucker Carlson. "The only way you're going to get it passed is with Republican votes."

Conservatives present wish list

Going forward, there's little margin for error for the speaker who has the almost herculean task of trying to please both moderates in his conference who fear the legislation doesn't do enough to help low-income Americans and conservatives who want to end the Medicaid expansion allowed under Obamacare sooner than later as well as reduce tax credits.

Members of the conservative Republican Study Committee huddled with Vice President Mike Pence Wednesday to discuss changes they'd like to see including giving states more "flexibility" on Medicaid, cutting the program's expansion off earlier and ensuring that the tax credits offered under the House's American Health Care Act were "fiscally responsible and promote work," according to a handout from the meeting.

While Republican Study Committee Chairman Mark Walker said he felt like the White House was open to the changes, it was just another example of the tension point between conservatives and moderates.

"For every action it creates an equal and opposite reaction sure some of that is going to be able to be moved back and forth. But I believe there is still enough middle ground, enough breathing space that it keeps everybody on board in the asks that we have made," Walker told reporters.

Pence also met with moderates hear their concerns.

Moderates like Rep. Charlie Dent, a Republican from Pennsylvania, have specifically said that rolling back Medicaid expansion before 2020 could be a major problem for himself and others.

At this point it's clear the bill will need to be changed, but pressed on what kind of changes the leadership is prepared to make, a senior House GOP leadership aide said they are limited to those that would "net votes" -- meaning that they are carefully evaluating those amendments conservatives and moderates are pressing and are only planning to sign off on those that up the ultimate vote total.

That message appeared to conflict with the message that members were getting from Pence Thursday, who left many with the impression that the tweaks they wanted would be added to the final version before it went to the House floor.

First official whip count

On Wednesday night, Republican leaders also got underway with its first official whip count. though leaders and whips have had ongoing conversations with members for weeks and since the bill has come out.

Chris Bond, spokesman for the chief GOP vote counter, Rep. Steve Scalise, said Wednesday evening "we feel very comfortable that we have a path to passage here."

The job of the whip team has only been made more difficult this week by a non-partisan Congressional Budget Office score showing that 24 million more Americans could be uninsured in the next decade if the Republican bill to repeal Obamacare is passed.

The public outcry from Senate Republicans on both sides of the ideological spectrum has also spooked some. Sens. Rand Paul and Ted Cruz attended a rally Wednesday at the Capitol and railed against the Paul Ryan-led House bill.

House Republicans from swing districts aren't interested in taking a risky vote on legislation that may be dead on arrival in the Senate.

Republican Rep. Leonard Lance, a moderate from New Jersey who may be vulnerable in 2018, told CNN Tuesday that "I do not want to vote on a bill that has no chance of passing over in the Senate," Lance said. "The CBO score has modified the dynamics."

Dent, the moderate from Pennsylvania, compared it to Democratic House members' vote in 1993 on the controversial BTU energy tax, which passed the House and then died in the Senate after a robust lobbying effort was waged against it. The vote was blamed by some for the Democrats' historic loss the next year.

"Members of the House don't want to be BTUed on health care," Dent said.