It’s been about two weeks since House Speaker Paul Ryan announced that he would retire, and a narrative has already emerged about Ryan’s legacy and why he wanted out. In short: Ryan had no place in President Trump’s Republican Party.

Ryan is a standard establishment Republican, the narrative goes, uninterested in President Trump’s populist bromides and racist appeals to some white voters. Ryan is cool and conventional; Trump is bombastic. Ryan is preoccupied with policy and ideology; Trump with style. Ryan is a fiscal conservative; Trump is a cultural conservative.

There’s some truth to all of that, and it’s not unimportant. But it’s much more difficult to separate Ryan and Trump than their demeanors suggest. Indeed, separating the two misunderstands the modern Republican Party and how it came about. Research has found that the two main ideological strains of contemporary conservatism — cultural and fiscal — are closely intertwined; we can trace the political careers of both Ryan and Trump to these ideological linkages.

Presidents can reshape what their parties stand for in the long term, but presidential ideology is also what social scientists call a “lagging indicator.” That is, we tend to get presidents who represent new ideological strains only after activists have been pushing the party in that direction for some time. And if we think about Trump as a product of long-term forces in the GOP, then his relationship to Ryan looks somewhat different.

Let’s start with Ronald Reagan, the icon of the modern GOP and the man whose presidency generally marks the beginning of it. Reagan’s political style was not Trump’s, and it’s difficult to imagine him talking about “shithole” countries. But it’s equally wrong to pretend that race wasn’t part of the larger political agenda at that time. Reagan helped tie civil rights backlash into mainstream conservatism. He kicked off his 1980 campaign in the South with a comment about states’ rights. Reagan was elected less than 20 years after new laws expanded the role of the federal government in protecting civil rights and voting rights, and his administration downplayed civil rights enforcement in the Justice Department and took stances against many of the affirmative action practices that earlier administrations had pursued. These stances fit an ideology that embraced both “color-blindness” and the paring back of federal power.

Now, the racial dynamics of the 2016 election were more extreme and explicit. Research by John Sides, Lynn Vavreck and Michael Tesler found that new divisions between white Republicans and white Democrats on racial attitudes emerged during Barack Obama’s presidency, and those in turn affected voting patterns in 2016. In a separate analysis, Tesler found that racial attitudes were more important to vote choice in 2016 than in 2008.

But that’s mostly a matter of degrees.

Indeed, just as with Reagan, researchers have found that race (and racism) isn’t so easily separated from the policy ideas that Ryan has most championed: shrinking the welfare state. Numerous scholars have pointed out the highly racialized images that surround welfare discourse and the importance of racial attitudes in determining how Americans feel about government assistance to the poor. In an environment this thick with implicit linkages between small government and social factors, you can reject hateful ideas and still contribute to resentment.

Ryan has himself found how linked these issues are by stepping into messy territory a few times. Some Democrats took issue with Ryan’s statements that he and Mitt Romney lost the 2012 presidential election because of the higher turnout in “urban” areas, suggesting that the language was an attempt to blame minorities for the Republican ticket’s loss. He caused a bigger controversy in 2014 when he said that a “culture problem” of men not working was driving poverty in U.S. cities.

Ryan acknowledged that his remarks had landed poorly and indicated later that he had changed some of his positions. But his comments indicated a connection between cultural and economic conservatism that had long been present.

Immigration is another topic on which race and economics intersect and on which Trump has been depicted as at odds with mainstream conservatives like Ryan. But the comparison with Ryan is at least somewhat complex. The House speaker has updated his website to accommodate Trump’s positions and actions. He has a low score from the interest group NumbersUSA, which advocates for reducing immigration, owed in part to his general stance that more legal immigration is good for the economy and the country (a stance with which the group disagrees). But as early as 2012, Ryan began to move to the right on a variety of immigration questions, including whether to grant permanent legal status to undocumented farm workers. So while Ryan’s views are still distinct from Trump’s, he’s been moving toward a more restrictive view of immigration, a shift that started even before Trump was a major political figure in the GOP.

One last demonstration of that overlap: Ryan and Trump, supposedly so different, actually followed similar trajectories to national political prominence. At least, those paths share a few characteristics.

First, opposition to Obama was crucial in both cases. As Craig Gilbert of the Milwaukee Journal Sentinel noted in 2009, Ryan emerged as a leading purveyor of Republican vision after the Democrats won big in 2008. Similarly, Trump’s electoral appeal drew not so much on policy difference from Obama as on cultural backlash.

Both are also simultaneously of the tea party movement and separate from it. The electoral strength of that movement helped Ryan eventually land the chairmanship of the House Ways and Means Committee after Republicans won a House majority in 2010 and then the position of speaker after John Boehner resigned out of frustration with the House Freedom Caucus — a group that drew on some tea party membership and ideas. Trump’s connection to the tea party is both less obvious and more direct. The movement paved the way for insurgent campaigns and supplied some of Trump’s talking points about the party establishment and the media. This insurgent tendency goes back to Reagan as well; he once rode a tide of anti-establishment conservatism against figures like Gerald Ford, Bob Dole and George H.W. Bush. Newt Gingrich continued this tradition when he became House speaker in 1995.

All that insurgent thinking, of course, has contributed to what a difficult job being the speaker of the House has become. And it’s all the more difficult with a president who has to be worked around rather than worked with. But the stronger motivation for Ryan to leave may not be Trumpism, it might just be Trump.