The legislation that House Speaker Paul Ryan and President Trump embraced to repeal and replace Obamacare was already being dogged by criticism from deeply conservative members in the House. But now it faces an emerging challenge: more moderate Republicans distancing themselves from the bill after the Congressional Budget Office concluded it would result in 24 million more people being uninsured than under the Affordable Care Act.

In the two days since the CBO report, two members from districts where Hillary Clinton outdid Trump last fall, Florida’s Ileana Ros-Lehtinen and and New Jersey’s Leonard Lance, have suggested they will be reluctant to back the current version of the legislation, the American Health Care Act. A third member from a liberal-leaning district, New York’s John Katko, said the CBO report left him with “serious questions that need to answered.”

Some senators are already suggesting this bill has little chance of passing in Congress’ upper chamber. But it may not get even there. An informal FiveThirtyEight count suggests two huge challenges for getting the AHCA through the House and avoiding what would be an embarrassing defeat for Republicans: spending seven years campaigning to end Obamacare and then not being able to get a bill through even one house of Congress when the party controls both houses and the presidency.

First, even before the full House considers the bill, it must move through another committee. And the Budget Committee, which meets on Thursday to consider the AHCA, is a potential problem since at least three of its GOP members had expressed deep concerns about the Obamacare repeal effort even before the CBO report was released. Republicans have a 22-14 majority on that committee, so the legislation would stall if four Republicans joined with the 14 Democrats, who are expected to vote en masse against the bill.

If the bill emerges from that committee, the full House will vote on it. With five seats in the chamber currently vacant, there are 430 members of the House. A bill must get a majority of those voting to be approved, so Republicans need 216 votes, not the traditional 218. And there are currently 237 House Republicans.

So for Republicans, the AHCA will fail if 22 GOP members oppose the bill. And public comments and votes suggest the party already faces at least 16 plausible defections.

The group that has already voted against the process to write this health care bill (nine members)

In January, House Republicans held a vote that called for using the so-called reconciliation budget procedure to repeal Obamacare. The vote was approved largely along party lines; all Democrats opposed it. But nine Republicans voted against that provision, as well. That opposition included four members of the House who are generally considered moderates. Two of them, Pennsylvania’s Charlie Dent and New Jersey’s Tom MacArthur, are co-chairs of the Tuesday Group, a bloc of more centrist House Republicans. Pennsylvania’s Brian Fitzpatrick, a freshman in the House, has created his own group, the Congressional Citizen Legislature Caucus, that is designed to work across party lines. The fourth of this group, New York’s Katko, was endorsed by the left-leaning New York Times editorial board last fall, with the paper casting him as an “independent thinker” and praising him for blasting Trump and promising not to vote for his party’s presidential nominee.

Two founders (Michigan’s Justin Amash and Idaho’s Raúl Labrador) of the House’s Conservative Freedom Caucus, a bloc of some of the chamber’s most conservative members, also opposed the reconciliation bill. So did three members who are not closely tied to the party’s moderate wing or the Freedom Caucus: Walter Jones of North Carolina, Thomas Massie of Kentucky and California’s Thomas McClintock.

Again, this January vote was procedural, not on the formal policy of the ACA. Dent and Katko, according to Roll Call, both said their opposition to the resolution was in part because congressional Republicans had not laid out a detailed plan to replace Obamacare. Obviously, the situation has now changed, with the AHCA fully written.

That said, the vote in January was a necessary first step in repealing Obamacare through reconciliation. And these nine members voted against it, suggesting they might do so again. Amash, Labrador, Jones and Massie, in particular, are known for either publicly questioning the views of Republican congressional leadership or voting against the GOP on issues that otherwise split along party lines.

Indeed, according to FiveThirtyEight’s Trump Score, which tracks how often members of Congress align with the position of the Trump administration on issues, only 10 Republicans have voted against Trump at higher rates than might be expected, based on Trump’s vote share in their districts in the 2016 election. That group includes Amash, Jones, Labrador and Massie.

McClintock, in comments at a closed-door meeting of House Republicans in January that were leaked publicly, warned his party of the political perils of trying to repeal Obamacare. Dent has remained critical, too, expressing deep concerns over how the AHCA would limit Medicaid, which was greatly expanded in his state under Obamacare. Katko has also been noncommittal about voting for the AHCA.

Ryan probably can’t count on these nine votes.

House conservatives who have sharply criticized the AHCA (five members)

Only two of the Freedom Caucus’ members (Amash and Labrador) opposed the January reconciliation bill. But four others, Virginia’s Dave Brat, Ohio’s Jim Jordan, North Carolina’s Mark Meadows and South Carolina’s Mark Sanford, have criticized various parts of the House proposal to repeal Obamacare or suggested the bill does not go far enough in completely revamping the law. And when Brat, Jordan, Meadows and Sanford held a press conference recently to slam the AHCA, Texas’ Louie Gohmert, who has not been that closely associated with the Freedom Caucus in the past, joined them to criticize the bill.

Some of these five members have suggested that Ryan needs to change parts of the AHCA to get their votes. Ryan does not seem inclined to do so.

If these five members can’t be brought on board, we’re up to 14 “no” votes.

Blue-district Republicans swayed by the CBO (two members)

Writing on her congressional website on Tuesday, Ros-Lehtinen announced she could not vote for the AHCA in its current form, arguing that “too many of my constituents will lose insurance and there will be less funds to help the poor and elderly with their health care.”

New Jersey’s Lance told CNN, “I do not want to vote on a bill that has no chance of passing over in the Senate.”

“The CBO score has modified the dynamics,” he added.

This was a reversal for Lance, who voted for the AHCA in committee just last week and defended it in an primetime interview on MSNBC.

A number of polls show Obamacare becoming more popular and that parts of the repeal effort, like making major changes to Medicaid and cutting funding for Planned Parenthood, are opposed by a majority of the public. That data suggests members in districts that backed Clinton in 2016 might be more politically vulnerable if they vote for the AHCA.

The other moderates and conservatives (at least 50)

If those 16 are likely against the AHCA, Ryan can only afford five more defections. And those defections are most likely to come from where the 16 do: the poles of the Republican caucus — its most moderate members and its most conservative members.

Along with Katko, Lance and Ros-Lehtinen, 20 other House Republicans live in congressional districts where Clinton won. About 50 House Republicans are associated with the moderate Tuesday Group, which includes Dent and MacArthur. (There is substantial overlap between those associated with the Tuesday Group and members from Clinton districts, so the bloc of House moderates numbers about 60.)

But while “no” votes may come from the more moderate wing of the party, worried the legislation is too conservative, there is a large bloc of members who are skeptical that the bill is conservative enough. There is no public, formal list of the members of the House Freedom Caucus. But at least 30 House members have publicly associated themselves with the group, and it is a hot bed of reservations about this bill. For example, Mo Brooks of Alabama, a Freedom Caucus member, called the legislation, “the largest welfare program sponsored in the history of the Republican Party.” And as noted above, four Freedom Caucus members are already very resistant to the bill.

That’s a lot of members — at least 50 moderates and at least 25 solid conservatives — to keep in line.

The good news for Paul Ryan

Many Freedom Caucus members live in districts that backed Trump overwhelmingly. White House officials are for now trying to court Freedom Caucus members through outreach. But at Tuesday’s White House briefing Press Secretary Sean Spicer sidestepped the question when asked if Trump would support primary challenges to Republicans who didn’t back the party’s health care push. Even the threat of a primary challenge could force some House conservatives to get behind this bill.

Meadows, who is chairman of the Freedom Caucus, has suggested that there is a broad consensus within the group to oppose the AHCA. But even as they express reservations about it, few Freedom Caucus members have said they will definitely vote against the AHCA.

Comments from other members of the group, such as Arizona’s David Schweikert and Ted Yoho of Florida, have been more equivocal than those of Meadows.

In addition, it will be interesting to see if the CBO report, which says this bill will result in a huge reduction in the number of Americans on Medicaid, helps move some Freedom Caucus members to be more supportive of the legislation. That CBO report cast the AHCA as more conservative than its portrayal from some on the right as “Obamacare-lite.” (In other words, if liberal groups are saying a bill is too conservative, citing the Washington-based CBO, does that move anti-establishment, conservative Republicans toward it?)

Also, the AHCA has already received 54 votes from House Republicans, combining the votes in the Ways and Means and Energy and Commerce committees. That’s exactly 25 percent of the 216 that Ryan needs. Members associated with the Tuesday Group, such as Chris Collins of New York and Adam Kinzinger of Illinois, have already voted once for the AHCA, through the committee process. The CBO report may have spooked Lance, but it may not shake other House Republicans, many of whom campaigned on repealing Obamacare.

Finally, some House Republicans are sticking by Trump in his early days, potentially at their political peril. FiveThirtyEight’s Trump Tracker shows that Virginia’s Barbara Comstock and California’s Darrell Issa have voted 100 percent of the time with the president’s position so far, even as those two members live in districts Clinton won in 2016. Partisanship exerts a strong pull. That, combined with the chance to do away with Obamacare, finally (at least in part), gives Ryan some powerful forces working in his favor. But it’s looking like a close vote.