House Speaker Paul Ryan’s retirement announcement Wednesday morning set off a round of heartburn within the White House and among its allies over the damage that may be done in what’s left of his term.

And while publicly no lawmaker is yet discussing a push to get the speaker to leave early, certain quarters of the Republican Party and Capitol Hill want him out of Congress right away rather than in early 2019.

“He’s not going to stick around. That’s not an option,” said one prominent GOP donor. “As a donor, would I ever give him a dollar if he’s not going to stay around? Fuck no…. We need someone with skin in the game who actually cares about the majority.”

Sources inside the Trump White House tell The Daily Beast that their primary concern following Ryan’s announcement was that more fellow House Republicans would follow him to the exits, enhancing the likelihood that Democrats gain control of the chamber. That fear seemed at least partially realized when Rep. Dennis Ross (R-FL) said that he too would be retiring from Congress minutes after the Ryan news broke.

But the concerns extend beyond 1600 Pennsylvania Avenue and aren’t just about the possibility of GOP lawmaker exodus that may soon transpire. On the Hill, there is fear that a lame-duck speaker all but guarantees a legislative logjam, as no lawmaker will now feel compelled to take a tough vote at Ryan’s insistence. There is also an appetite within the Republican caucus to get new leadership in right away, with the front runners being Majority Leader Kevin McCarthy (R-CA) and Majority Whip Steve Scalise (R-LA).

“I don’t know what he was thinking. I think this was a huge miscalculation. I think this is the captain abandoning the sinking ship,” said former Rep. Tom Davis (R-VA), who told The Daily Beast he’d been talking to his ex-colleagues about the Ryan news. “Members are pissed. I think this is going to cause some angst.”

Ryan’s office did not return a request for comment.

Ryan pledged on Wednesday to “run through the tape, to finish the year.” And Rep. Mark Meadows (R-N.C.) told reporters that he knew of “no efforts to make him leave sooner than that."

But others don’t see that as politically tenable, with one GOP congressional aide noting that “Eric Cantor had committed to staying through the end in 2014, too,” only to leave earlier. “There’s a natural discomfort with the idea of a leadership member hanging on for several months as a lame duck,” the aide said.

Untethered from the political pressures, Ryan could feel emboldened to now push a legislative agenda contrary to the White House’s primary interests. That could include a piece of immigration legislation that is more generous to so-called DREAMers — undocumented minors brought to the country by their parents — than President Donald Trump is comfortable with or even a bill that would shield special counsel Robert Mueller from being fired by the president.

“There’s a good reason people in DC say if you want to make someone behave the way you want, keep them in a job where they have to play ball with you or answer to you for as long as possible,” said GOP strategist Liz Mair, who is close to Ryan. “Ryan is no longer in quite that position, and his view of what is in America’s best interest has never truly aligned with Trump’s— nor do his core values as a human being. That makes this a big risk for the White House."

But not everyone sees Ryan going down this route because, among other relevant factors, the speaker has shown little overt ideological affinity for some of these items. Despite critical words aimed at Trump and “bigotry” during the 2016 campaign, Ryan has been a reliable partner for the White House since January 2017. If anything, he’s been the most instrumental lawmaker in shepherding his agenda, from dismissing criticism over the president’s temperamental tweets to allowing the House Intelligence Committee investigation into Russia’s election meddling to become an exercise in partisan defensiveness.

Even if he sticks to his conservative roots, an emboldened Ryan could cause political headaches, including by doubling back on his long-standing desire to pursue overhauls of the nation’s social safety net—an objective that Trump has previously rejected and one that could put the rest of the Republican Party in a more precarious position ahead of the 2018 midterms.

“I think on the way out the door Paul Ryan is going to want to do something more conservative and smaller government,” said former Rep. Jack Kingston (R-GA), an ally and media surrogate of President Trump’s.

Kingston went on to predict that, ultimately, Ryan would remain relatively benign during his lame-duck period, neither pursuing wayward legislative ideas or becoming a vocal Trump critic. It’s a view shared with many senior staffers in the Trump White House, who noted that keeping his exalted status in the Republican Party is dependent on it.

“He has always been a team player,” Kingston said, of Ryan. “Remember, he was a congressional staffer. He likes the institution, whereas [Trump critic Sen. Jeff] Flake as a House member liked to attack Republican earmarks and never was a team player.”