Pfeiffer, far right, in the Map Room of the White House. Photo: Pete Souza/Courtesy of the White House

Dan Pfeiffer, who left his position as senior adviser at the White House last week after having worked with Barack Obama since his first presidential campaign, has been involved from the outset in navigating the central contradiction at the heart of Obama’s public persona: He ran as a figure who could overcome partisan polarization, yet he has instead presided over more of it despite accomplishing the majority of the substantive agenda he promised.

Obama and his spokespeople have spent most of their administration quietly at war with the conventional wisdom in Washington over the cause of this failure, and Pfeiffer has spent much of his time in the administration dealing with, or scolding, members of the media, mostly in off-the-record conversations. But in an interview last week, a few days before he resigned, he explained in unusually candid terms the administration’s thinking—and how the White House lost its illusions.

“I think [Obama] believes, and I certainly believe, that while we can always do better, this is a case where structural forces are the large actor here,” he told me. Pfeiffer cited three of them. The first is rising polarization—“the great sorting,” as he called it—which, over a period of decades, has driven white conservatives out of the Democratic Party and moderates out of the Republican Party, creating two ideologically homogeneous political organizations. The second is the disintegration of restrictions on campaign finance, which “gives people even more incentive to play to the far right or to a set of special-interests donors, so that one individual can basically, especially in these House races, do a $1 million expenditure and completely tip the balance.” And, finally, the news media has changed so that people select only sources that will confirm their preexisting beliefs.

All of this combined makes communication with Republicans mostly hopeless. “There’s very little we can do to change the Republicans’ political situation because they are worried about a cohort of voters who disagree with most of what the president says,” Pfeiffer said. “We don’t have the ability to communicate with them—we can’t even break into the tight communication circles to convince them that climate change is real. They are talking to people who agree with them, they are listening to news outlets that reinforce that point of view, and the president is probably the person with the least ability to break into that because of the partisan bias there.”

Pfeiffer’s reading of the red-blue impasse isn’t that it’s a permanent catastrophe. Demographic change will eventually force Republicans to compete with Democrats for some of the same voters, reopening a national political conversation that is accessible to both parties. And Democrats will find the millennial generation in play. “We’re going to have to work harder to get them registered to vote and involved, and that offers an opportunity, because while they are very progressive in some of their general leanings, they’re less tied to institutions and parties.” But that will have to happen after this administration has left the scene.

The original premise of Obama’s first presidential campaign was that he could reason with Republicans—or else, by staking out obviously reasonable stances, force them to moderate or be exposed as extreme and unyielding. It took years for the White House to conclude that this was false, and that, in Pfeiffer’s words, “what drives 90 percent of stuff is not the small tactical decisions or the personal relationships but the big, macro political incentives.”

If you had to pinpoint the moment this worldview began to crystallize, it would probably be around the first debt-ceiling showdown, in 2011, when Obama tried repeatedly and desperately to cut a budget deal with House Speaker John Boehner only to realize, eventually, that Boehner did not have the power to negotiate. The administration has now decided that in many cases, even adversarial bargaining fails because the Republican leadership is not capable of planning tactically. “You have to be careful not to presume a lot of strategy for this group,” Pfeiffer said. “I’ve always believed that the fundamental, driving strategic ethos of the Republican House leadership has been, What do we do to get through the next caucus or conference without getting yelled at? We should never assume they have a long game. We used to spend a lot of time thinking that maybe Boehner is saying this to get himself some more room. And it’s like, no, that’s not actually the case. Usually he’s just saying it because he just said it or it’s the easiest thing to solve his immediate problem.”

This analysis puts the administration at odds with the reading of American politics that still dominates much of Washington reporting. Many political journalists imagine that the basic tension for the White House lies between Obama’s liberal base and appealing to Americans at the center, who will be crucial for tipping elections.

Pfeiffer believes the dynamic is, in fact, the opposite: “The incentive structure moves from going after the diminishing middle to motivating the base.” Ever since Republicans took control of the House four years ago, attempts to court Republicans have mostly failed while simultaneously dividing Democratic voters. Obama’s most politically successful maneuvers, by contrast, have all been unilateral and liberal. “Whenever we contemplate bold progressive action,” Pfeiffer said, “whether that’s the president’s endorsement of marriage equality, or coming out strong on power-plant rules to reduce current pollution, on immigration, on net neutrality, you get a lot of hemming and hawing in advance about what this is going to mean: Is this going to alienate people? Is this going to hurt the president’s approval ratings? What will this mean in red states?” And yet this hesitation has always proved overblown: “There’s never been a time when we’ve taken progressive action and regretted it.”

This was deeply at odds with the lesson Bill Clinton and most of his aides (many of whom staffed Obama’s administration) had taken away from his presidency. But by the beginning of Obama’s second term, at least, the president seemed fully convinced. “As we were preparing for the potential that we would lose the midterms,” Pfeiffer told me, “a lot of the advice we got around town was, You have to show major contrition; heads have to roll; you have to give some sop to the Republicans. The president’s view was, No, we’re not going to do that. We’re going to go out and we’re going to be the opposite of contrite; we’re going to be aggressive in our policies and our politics. And that worked. It caused people to cheer. But that’s the exact opposite of the sort of advice you’d get in this town.”

Though the administration has wound up embracing a very different political strategy from the one it began with, one thing has remained consistent: Obama’s disdain for conventional wisdom. In his introduction to America as the keynote speaker at the 2004 Democratic National Convention, he criticized the pundits who “like to slice and dice our country into red states and blue states.” Now the pundits insist that Obama would bridge the partisan divide if only he spent more time golfing with John Boehner. Those whom Obama once dismissed as cynical he now dismisses as naïve.

Which isn’t to say that he sees his presidency as triumphant. I asked Pfeiffer about how his boss’s view of politics has changed. “He had hopes of being able to change the polarization, not just in the country, but in Washington,” Pfeiffer told me. “We learned very quickly that that was a lot harder than we thought. He will always say that his one biggest regret is that he’s been unable to deliver on that promise.”

*This article appears in the March 9, 2015 issue of New York Magazine.