If Paul Ryan thought his final months as House Speaker would be easy, he was mistaken.

Ryan, who is walking away from Congress when his term ends early next year, will have to steer a divided Republican caucus through an internal debate over immigration, where moderates and hardliners alike are battling the Speaker and his Leadership team. On top of that, he has to manage a government-funding deadline shortly that falls shortly before the midterm elections, oversee the ongoing investigation into President Donald Trump’s campaign and figure out how to pick up the pieces of a farm bill that failed at the end of last week.

If that weren’t enough, there’s this looming question: Who will replace Ryan atop a largely ungovernable GOP conference? And a corollary that many Republicans outside of Congress are asking: Who would want to?

With the House majority in revolt, Ryan could only watch powerlessly as an enormous piece of legislation — the $867 billion farm bill — imploded on Friday. As the vote got closer, he closed his eyes for a beat as he stood in the rear of the House chamber, waving his hand dismissively as if to say he had done his best.

Ryan saw it coming, those close to him said. “What more could we have done?” asks one adviser to House Leadership. “Tell me, and we’ll try it next time.”

In simpler times, it would have been a layup: a bill to aid rural farmers while imposing new work requirements on food stamp recipients. “I really thought we had the votes,” says one senior Republican aide. “I don’t know how you can say no to what we offered.”

But the farm bill fell apart in a stunning 213-198 defeat not just because Republicans didn’t care for its content, but also because of what it represented. A bloc of conservative insurrectionaries, led by Rep. Mark Meadows of North Carolina, the House Freedom Caucus chairman, rallied votes against it because it was a piece of legislation they could use to get their way on another matter.

For months, House conservatives have been clamoring for a vote on a hard-line immigration bill sponsored by Rep. Bob Goodlatte of Virginia and backed by the White House. The bill would reduce legal immigration by roughly 40% and offer no protections for the young immigrants who were brought to the country illegally as children, known as Dreamers. As such, it would almost certainly never make it through Congress. But that didn’t stop these conservatives from using the farm bill vote to spite the leadership for its inaction on immigration.

At the same time, moderates and pragmatists on immigration were pushing another immigration measure designed to be more palatable in districts with heavy Hispanic and Latino populations, as well as swing districts where the GOP majority hangs on a knife’s edge.

“We failed to get one, and so we’re forcing one,” GOP Rep. Jeff Denham of California said on CNN about the desire for a debate on the issue. He is among the group of lawmakers looking to defy Republican Leadership and force a floor vote on a bipartisan immigration plan using a parliamentary gambit that could prove embarrassing to Ryan and his team.

Such embarrassment would be nothing new. At this stage, Ryan should be planning his exit with an eye toward burnishing his legacy. (The work-requirement component of the farm bill was a piece of his welfare overhaul crusade.) Instead, he is fighting his own restive conference and trying to manage relations with the White House—tasks, those close to him say, that are proving more difficult now that he has made known his plans to leave.

Ryan’s critics, emboldened now that he is heading toward the door and increasing in numbers and volume, say the Speaker is running his own agenda that is apart from the party’s base. Meanwhile, the fight to replace him brewing just below the surface is starting to boil, while the midterm elections draw near.

Ryan has been warning his colleagues that Democratic enthusiasm is growing. Democrats need to net only 23 House seats in November’s elections to claim the majority for the first time since 2011, and 47 Republican incumbents in the House — including Ryan — have already announced that they’re calling it quits, either heading into retirement or seeking other offices. Republicans are bearish on their odds, and all the internal squabbling does little to unite the party behind a message they can sell to voters.

Consider the farm bill. The measure was a rare moment of potential achievement that could help Republicans with their rural and suburban bases. The largest piece of the agriculture bill is typically aid for poor people to buy food, a two-for-one proposition to feed the working poor and to indirectly subsidize farmers. Republicans inserted a requirement for those poor Americans to either work 20 hours a week or to be in job-training programs that same amount of time to qualify.

That work-requirement proved unacceptable for Democrats, who voted in unison against the measure. Republicans couldn’t match their discipline, and defections led the bill to fail. (The temporary political setback may have actually helped Republicans, given the party’s base of support is in rural areas and steady jobs are often harder to find there than in cities.) House rabble rousers defended derailing the bill, saying the 2016 election had given them a mandate to crack down on immigrants and immigration. And behind the scenes, some White House advisers applauded these lawmakers’ pluck.

The fight over immigration may play well with the hard-line base of the party, but it doesn’t bode well for districts or states with large Hispanic and Latino populations. For instance, GOP Rep. Will Hurd represents a sprawling district in Texas that is about 70% Hispanic. Hurd is among those looking to force Leadership to schedule a floor vote on a bipartisan immigration bill — all while running in a district he has twice won by the barest of margins. If Tuesday’s runoff goes as expected, Hurd is likely to face Democrat Gina Ortiz Jones, a female Air Force intelligence officer who served in Iraq.

At the same time, Ryan and his deputies cannot seem to control a House Intelligence panel that seems determined to give the White House cover. A prematurely released report from the committee indicated members found no evidence of collusion with Russia — a conclusion the President heralded wrongly as the final word. On Monday, as he visited CIA headquarters, Trump twice singled out House Intelligence Committee Chairman Devin Nunes for his work on that verdict.

Meanwhile, few lawmakers seem eager to protect Special Counsel Robert Mueller’s ability to finish his probe, which has secured indictments against 19 people and five guilty pleas stemming from an ongoing look at foreign meddling in the 2016 election. Heeding the White House’s talking points, many rank-and-file Republican lawmakers have Mueller probe a waste of time—a stark counterpoint to Ryan’s position that Mueller should have time to conclude his investigation.

Like Mueller, Ryan and his lieutenants are finding that no amount of logic or method can overcome the raw emotion of the Republican majority in the House. If that gut-driven approach to governing continues, however, the caucus may find itself appreciably smaller after the 116th Congress takes power in January.

Get our Politics Newsletter. The headlines out of Washington never seem to slow. Subscribe to The D.C. Brief to make sense of what matters most. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Philip Elliott at philip.elliott@time.com.