Republican control of the House during President Obama’s era was marked by high-stakes showdowns, intra-party sniping and the persistent threat of a coup against the GOP House speaker. A Republican House at the outset of a President Clinton administration could be more of the same–or even worse–if November losses erode the GOP majority, giving the Freedom Caucus types increased leverage in a more closely divided chamber.

While Democrats are signaling they’re playing to win back the House, most forecasters still see a flip of the lower chamber to be a long shot. But that doesn’t mean Speaker Paul Ryan (R-WI) and the other GOP leaders committed to steering the party away from the recent dysfunction have any reason to breathe easily. An election that preserves Republican control of the House but shrinks GOP’s margins significantly will exacerbate the challenges Ryan was already facing in navigating a fractured party.

Here are 5 points on the headaches awaiting Ryan if Republicans’ margin over Democrats in the House shrinks.

Pragmatic Republicans are the members most likely to lose their seats.

If Trump does prove to be a drag on the ticket, the GOP incumbents likely to be affected are those who in the past could be depended on to cooperate with leadership, congressional experts tell TPM.

“If you look at a couple of instances where getting a must-pass bill done required Republicans to go to Democrats for votes, the Republican members who are most likely to lose this year made up a big share of the Republicans that actually voted for those things,” Molly Reynolds, a fellow in governance studies at the Brookings Institution, told TPM.

These members tend to be from purple districts, whereas the hardliners are usually from bright red districts that will be safe, even if Trump suffers a major loss.

“The people who are likely to be more fractious with Paul Ryan are unlikely to lose,” John Fortier, the director of Bipartisan Policy Center’s Democracy Project, told TPM.

A smaller GOP majority shifts the power to the House Freedom Caucus.

Time and time again, under former Speaker John Boehner’s watch, the chamber’s legislative proceedings were gummed up by a group of far-right representatives known as the House Freedom Caucus. Those roughly three dozen Republicans often wielded coup threats against Boehner. (Boehner ultimately announced his resignation — in a sense, falling on his sword — to escape their threats before pushing through a major budget deal).

The smaller the size of the entire GOP conference, the larger portion of it is made up by this group of hardliners. If Ryan loses their support, particularly on must-pass legislation to keep the government running or raise the debt ceiling, he will have to rely increasingly on Democrats, which will in turn enrage conservatives and their political base.

“That fraction of Republicans that might give leadership a hard time, it’s a larger fraction of the whole,” Fortier said.

Hardliners won’t have many reasons to temper their approach.

Even before GOP was bracing for Trump-induced down-the-ballot carnage, the House Freedom Caucus was making moves to expand their ranks. The hardliners have backed candidates in races for open seats that appear to be aligned with their agenda. Politico reported last week that establishment types are fretting that Club For Growth is bankrolling the effort (the conservative advocacy group denies any official coordination with the House Freedom Caucus). Already, hardliner-approved candidates have won primaries for open seats in Indiana and in Georgia, while capturing Boehner’s old district in a special election this summer, according to Politico.

The sentiments of this cycle could also fuel their strong-arm tactics, to Ryan’s dismay, experts say.

“They’ll sort of look at it and say, ‘It’s our least Republican members who lost, the ones who didn’t stand with us. Clearly that’s a signal that they had it wrong. We have it right,'” said Michael DiNiscia, the associate director of the John Brademas Center for the Study of Congress at New York University.

On one hand, you had a presidential primary where they major takeaway is that the Republican base is fed up with the establishment. On the other hand, the ideologues of the party will likely blame a Trump loss on not picking a nominee who was conservative enough.

“They’re going to want purity and purity does not include cutting deals with Democrats,” said Norman Ornstein, a congressional scholar at the American Enterprise Institute.

If Dems win the Senate, Ryan will have to bargain with them as well.

For the last two years, House Republicans have had the benefit of working with a GOP Senate, which has allowed them to, among other things, put Obamacare repeal on the President’s desk and present a unified front when playing hardball with Obama. That will no longer be the case if Democrats can eke out a flip of the Senate.

“If Democrats control the Senate, then it’s not just that Ryan has to figure out how to get through the caucus and the House Democrats. Anything that he could conceivably want to get done would require working with Democrats in the Senate and that is also a challenge,” Reynolds said.

If Clinton wins, she could be the first Democratic president since Glover Cleveland to enter the White House without her party controlling both chambers. But continued House chaos only strengthens her and Democrats’ bargaining position.

“She and her team will try to press her advantage as much as possible while the Republicans are a little off balanced” DiNiscia said.

The question of Ryan’s presidential ambitions will loom in the background of all these fights.

Ryan has positioned himself as the party’s reluctant savior, having only accepted the speakership with public hesitation, while emphasizing a leadership approach that decentralizes power. But every move he makes after this year’s election will be viewed through lens of a potential 2020 presidential run.

“He is not going to win a lot of friends and be in a good position within the party if he is not seen as being friendly to most parts of the party,” Fortier said. “Despite some instincts to really cut a deal, I think there’s really going to be some pressures that are really going to make it difficult for him, both as a speaker and as a potential candidate for president.”

Ryan may not ultimately opt to run for president, but at the very least, he has signaled he wants to play a role in pointing the party towards a more pro-active, substantive direction. That is a mark he can only make while he has a seat at the table.

“I don’t think everything Paul Ryan will do is governed by his own larger political ambitions,” Ornstein said. “But at the same time, whether you are thinking of settling for a significant period for Speaker of House or looking ahead — you’re a very young man — to 2020 or beyond that, how you operate in the context of a party undergoing an existential crisis, there’s just no easy answer to that.”