It should have been a crowning moment for House Speaker Paul Ryan. The Wisconsin Republican had campaigned, both as a congressman and as a vice presidential nominee, on repealing and replacing Obamacare, and he finally had what he needed to do it: control of both chambers of Congress and a Republican president equally determined to kill President Barack Obama's landmark health care law.

Instead, Ryan ended up traveling to the other end of Pennsylvania Avenue to tell President Donald Trump that he did not have enough votes in his own caucus to pass a health care package that Republicans had seven years to write. And the high-profile failure threatens to make even more difficult a job Ryan never wanted to begin with.

With the GOP-crafted American Health Care Act in trouble, Trump came in heavy-handed, with threats to "come after" conservative lawmakers who voted no and a Thursday night ultimatum: Stop negotiating and vote Friday, even though it was clear Ryan did not have the 216 votes needed to pass it. Instead of voting on Friday, though, the president called Robert Costa, a political reporter for The Washington Post, and told him they were pulling the bill.

"We just pulled it," Costa tweeted the president told him. "I don't blame Paul."

Now, Ryan is left with the empty line on his list of legislative victories – not to mention the task of re-establishing his authority in a divided caucus and increasingly distrustful Capitol environment. In a press conference after the decision to pull the bill, Ryan attributed the loss to the "growing pains" of trying to make the transition from being the opposition party to the governing party.

But the defeat, combined with questions over who was making the strategic decisions, creates a cloud over Ryan's office.

"On any given day, I can't figure out who's in charge on the other side," Rep. Hakeem Jeffries, D-N.Y, co-chairman of the Democratic Policy and Communications Committee, told reporters. "Is it Speaker Ryan? Donald Trump? The Freedom Caucus? The Heritage Foundation? None of the above? I can't figure [it] out."

The loss on the AHCA could have a ripple effect, hampering Ryan as congressional Republicans try to go forward with other measures on tax reform, infrastructure, the budget and perhaps immigration. The debacle is compounded by other in-House troubles Ryan is dealing with, including anger that the GOP Intelligence Committee chairman, Rep. Devin Nunes of California, revealed intelligence information to Trump without sharing it with Democratic colleagues or revealing his sources.

"We need to go back and have kind of an introspective meeting with ourselves about what this does to the rest of the legislative agenda," says Rep. Steven Womack, R-Ark., ticking off items like the debt ceiling, appropriations and tax reform. These are "issues we all want to do, so much of which I think was hitched to our ability to get something like this done."

Adds Alabama GOP Rep. Bradley Byrne: "I don't think tax reform is going to be easier than health care. This [defeat] didn't make it any easier."

Most Republicans -- including Trump -- say they do not blame Ryan for the loss, praising the wonky speaker of the House for his hard work on the bill. They bemoan the inability of their own caucus to come together, especially on something as prominent as health care.

"There's no question that it's a loss for leadership. Not a loss of leadership, but a loss for leadership," Womack says.

If there was a winner on the health care standoff, it was the Freedom Caucus, a conservative GOP group of House members who didn't think the AHCA went far enough to take government out of the health care system. Former Republican House Speaker John Boehner had battled with the caucus, a conflict that eventually drove a frustrated Boehner out of the job.

But if Boehner was irritated over the Freedom Caucus' refusal to compromise on spending and other matters, Ryan has it worse. He had been perfectly happy as a Budget Committee wonk; it took lots of effort to get him to accept the speaker's role after Boehner stepped down. And without Obama, or even a Democratic-led Senate, to serve as a galvanizing common enemy, Ryan has been dealing with an ideological civil war in his caucus at a time when he ought to be scoring massive legislative victories.

The Freedom Caucus formed because a few dozen Republicans felt the Republican Study Committee, the policy arm of the caucus, was not conservative enough, notes Rep. Chris Collins, a New York Republican who is loyal to both Ryan and Trump.

"You could say that the conference hasn't figured out yet how to be the governing party. They are still acting as though they are the opposition party. They [the Freedom Caucus] are still acting as though we are the opposition party," Collins says. Instead of being a group pushing bills to the right, "they are simply three dozen members who don't compromise," Collins adds.

Ryan has always been seen as more of a policy guy than than a political animal, and his decision to schedule a vote on the AHCA for Thursday evening – the seventh anniversary of the signing of Obama's Affordable Care Act – turned out to be a misread of his caucus, which was deeply divided over the hastily moved bill.

Mistake One was followed by what some on Capitol Hill saw as Mistake Two: giving in to Trump when the president insisted a vote needed to be held Friday, as well as making repealing and replacing Obamacare a priority over other issues. That annoyed key Republican lawmakers, who felt the vote was being artificially hastened so Trump could claim an early victory and deflect attention from the investigation of his team's alleged ties to Russia.

"People say it's got to be done by the anniversary of the signing" of the Affordable Care Act, says Rep. Mark Amodei, a Nevada Republican who intended to vote no on the GOP bill. "Really? Are we in the Hallmark card business?"

Some observers say having Trump participate in the negotiations helped doom the legislation – real estate deals and business-world arm-twisting are different from the complicated science of working with House members from disparate districts, ideologies and varying levels of political vulnerability ahead of the 2018 mid-term elections.

Others point out Ryan may not have had the clout, the whip experience or the intellectual arguments to defend a bill that, according to a Quinnipiac University poll, had the support of just 17 percent of the American public.

"The fact that the president was taking the role on the negotiations that he was tells me the speaker has drained the fuel in his negotiations tank," Amodei says.

"I think it's embarrassing to the speaker, and it demonstrates that maybe the speaker doesn't control the House right now," says Rep. Joseph Crowley, D-N.Y. "It's controlled by Donald Trump."

That's hardly a ringing endorsement for the leader as Republicans embark on what they hoped would be a string of legislative victories. But Ryan, Collins and others say, is safe in the job, for better or for worse.