Samuel Rayburn served as speaker of the House for over 17 years. Tip O’Neill held the position for 10. Both were exceptions to the rule. Most speakers don’t serve long terms and most go on to leave a legacy within Congress and over legislation.

Rarely, if ever, do they become personifications of a national moment. That’s because the speaker’s gavel comes with a ticking clock and the knowledge that, like buying a boat, it sounds fun but your happiest days with it will be your first and your last.

You are, at the heart, a caretaker of a body that will last beyond you—a political leader in the institutional, not personal, sense. And yet, despite that, and despite short terms being the norm, ambitious members of Congress have always sought the position. Today, the familiar refrain in Washington about being speaker is “who would want that job?” But the truth is, a lot of lawmakers do.

With Paul Ryan’s announced retirement on Wednesday, we are reminded once more that for all the power and allure, the speakership is now term-limited in spirit, with the pressures of the conference and national politics slowly eating away at any mandate you have to serve and making even two-year terms look like an eternal race for survival.

A speaker must have five things to succeed: 1) Policy expertise; 2) Process expertise; 3) Support of the conference (party); 4) Fundraising prowess; and 5) Support among national primary electorate.

The last one is new. The support of the conference used to be enough, but in today’s Congress intra-party support depends on your broader popular appeal. Ryan used to have that by the truckload. Until he didn’t.

Going back to when Mitt Romney chose him as his running mate, one must recall that Ryan was the symbol of the party’s future: youthful, wonkish, humble, and solidly conservative. The old joke (or insult depending on your audience) was that Ryan was created in a laboratory in The Heritage Foundation. As a national candidate, he was more than appealing, he was the face of serious governing.

Fast-forward to John Boehner’s departure. Ryan became the only logical successor. He had the policy and process expertise after serving as chairman of the Ways and Means and Budget Committees. He had also consistently been in the room with leadership when big decisions needed to be made, which made him both trusted and someone whose guidance was considered invaluable.

Ryan had the support of the conference. And when Boehner left, he was practically forced into the job.

But the support among the party faithful was waning as his name ID grew and inside agitators rose. He became the face of a supremely unpopular institution and the target of a group of members whose only purpose is to thwart leadership and score conservative media and grassroots adulation. The cliques within the conference became increasingly tribal and began serving little interest other than self-promotion and the lust over pointless scorecards.

Ryan’s support within the broader conference also waned, not because he had become a different figure from the one they’d previously loved. He was speaker now, and thus the place for them to lay all the blame of their broken promises and unmet expectations fell at his feet. Shouldering that burden is so much a part of the job that leaders often tell reluctant but vulnerable members to do so. Ryan knew this and played along gamefully.

Indeed, one of the most misunderstood parts about being speaker is who he or she actually represents. Yes, they represent their district. But inside the Capitol, they represent their conference: a group that grows more polarized by the day with safer districts, unrealistic demands, and a shallow depth of policy acumen.

When Ryan critics say he didn’t stand up to President Trump enough, it stems from this misunderstanding. Members of the conference expected him to represent their collective will, and the will to defy and castigate the president has never existed. That was to my great regret as the well as the regret of others. But Ryan chose to see it as an opportunity to pass an agenda largely in line with his own: tax cuts and other unheralded items but probably not as much as he would’ve liked. He will leave with a spending record that surely aggravates him.

There was still dysfunction, of course. But that is a result of the incremental loss of the leverages of leadership that preceded his tenure. The loss of earmarks not only abdicated fiscal decisions to the faceless and unaccountable bureaucracy, but it took members attention away from their districts. Without individual spending priorities, all of their attention is placed on the day-to-day grind of national politics. Budgets don’t get passed and compromise within the party and across the aisle becomes a daunting if not insurmountable task.

Ryan’s critics on the left (and among a number of conservatives too) will ask why he didn’t serve as more of a protector of the institution of Congress against a seemingly unhinged executive. Certainly, President Trump will loom large over Ryan’s legacy, much as it will the tenure of Ryan’s successor. The first question in the first interview Ryan received after his announcement was about a porn star, illicit payments, and Access Hollywood tapes.

Ryan’s critics on the right will question the timing of the announcement as the Republican Party enters, what is objectively, one of its most difficult electoral paths in recent memory. With his retirement, a Democratic wave seems more certain. A Democratic speaker seems closer to a reality.

But a change in party control won’t wipe away the factors that led to Paul Ryan’s departure.

Whether it’s Nancy Pelosi or Steny Hoyer, or a more youthful Democrat (perhaps someone under 75 years old) who becomes the next speaker, he or she will face many of the same problems: polarized members, tribal caucuses, and unrealistic expectations.

Maybe it’s better to just buy a boat.

Rory Cooper is a former communications director to Majority Leader Eric Cantor, former White House adviser to President George W. Bush, and currently a managing director at Purple Strategies.