Hundreds of pages of internal White House memos show Obama grappling with the unpleasant choices of government. Illustration by The Heads of State

On a frigid January evening in 2009, a week before his Inauguration, Barack Obama had dinner at the home of George Will, the Washington Post columnist, who had assembled a number of right-leaning journalists to meet the President-elect. Accepting such an invitation was a gesture on Obama’s part that signalled his desire to project an image of himself as a post-ideological politician, a Chicago Democrat eager to forge alliances with conservative Republicans on Capitol Hill. That week, Obama was still working on an Inaugural Address that would call for “an end to the petty grievances and false promises, the recriminations and worn-out dogmas that for far too long have strangled our politics.”

Obama sprang coatless from his limousine and headed up the steps of Will’s yellow clapboard house. He was greeted by Will, Michael Barone, David Brooks, Charles Krauthammer, William Kristol, Lawrence Kudlow, Rich Lowry, and Peggy Noonan. They were Reaganites all, yet some had paid tribute to Obama during the campaign. Lowry, who is the editor of the National Review, called Obama “the only presidential candidate from either party about whom there is a palpable excitement.” Krauthammer, an intellectual and ornery voice on Fox News and in the pages of the Washington Post, had written that Obama would be “a president with the political intelligence of a Bill Clinton harnessed to the steely self-discipline of a Vladimir Putin,” who would “bestride the political stage as largely as did Reagan.” And Kristol, the editor of the Weekly Standard and a former aide to Dan Quayle, wrote, “I look forward to Obama’s inauguration with a surprising degree of hope and good cheer.”

Over dinner, Obama searched for points of common ground. He noted that he and Kudlow agreed on a business-investment tax cut. “He loves to deal with both sides of the issue,” Kudlow later wrote. “He revels in the back and forth. And he wants to keep the dialogue going with conservatives.” Obama’s view, shared with many people at the time, was that professional pundits were wrong about American politics. It was a myth, he said, that the two political parties were impossibly divided on the big issues confronting America. The gap was surmountable. Compared with some other Western countries, where Communists and far-right parties sit in the same parliament, the gulf between Democrats and Republicans was narrow.

Obama’s homily about conciliation reflected an essential component of his temperament and his view of politics. In his mid-twenties, he won the presidency of the Harvard Law Review because he was the only candidate who was trusted by both the conservative and the liberal blocs on the editorial staff. As a state senator in Springfield, when Obama represented Hyde Park-Kenwood, one of the most liberal districts in Illinois, he kept his distance from the most left-wing senators from Chicago and socialized over games of poker and golf with moderate downstate Democrats and Republicans. In 1998, after helping to pass a campaign-finance bill in the Illinois Senate, he boasted in his community paper, the Hyde Park Herald, that “the process was truly bipartisan from the start.”

A few years later, Obama ran for the U.S. Senate and criticized “the pundits and the prognosticators” who like to divide the country into red states and blue states. He made a speech against the invasion of Iraq but alarmed some in the distinctly left-wing audience by pointing out that he was not a pacifist, and that he opposed only “dumb wars.” At the 2004 Democratic Convention, in Boston, Obama delivered a retooled version of the stump speech about ideological comity—“There is not a liberal America and a conservative America; there is the United States of America!”—and became a national political star.

In 2006, Obama published a mild polemic, “The Audacity of Hope,” which became a blueprint for his 2008 Presidential campaign. He described politics as a system seized by two extremes. “Depending on your tastes, our condition is the natural result of radical conservatism or perverse liberalism,” he wrote. “Tom DeLay or Nancy Pelosi, big oil or greedy trial lawyers, religious zealots or gay activists, Fox News or the New York Times.” He repeated the theme later, while describing the fights between Bill Clinton and the Newt Gingrich-led House, in the nineteen-nineties: “In the back-and-forth between Clinton and Gingrich, and in the elections of 2000 and 2004, I sometimes felt as if I were watching the psychodrama of the Baby Boom generation—a tale rooted in old grudges and revenge plots hatched on a handful of college campuses long ago—played out on the national stage.” Washington, as he saw it, was self-defeatingly partisan. He believed that “any attempt by Democrats to pursue a more sharply partisan and ideological strategy misapprehends the moment we’re in.”

If there was a single unifying argument that defined Obamaism from his earliest days in politics to his Presidential campaign, it was the idea of post-partisanship. He was proposing himself as a transformative figure, the man who would spring the lock. In an essay published in The Atlantic, Andrew Sullivan, a self-proclaimed conservative, reflected on Obama’s heady appeal: “Unlike any of the other candidates, he could take America—finally—past the debilitating, self-perpetuating family quarrel of the Baby Boom generation that has long engulfed all of us.”

Obama was not exaggerating the toxic battle that has poisoned the culture of Washington. In the past four decades, the two political parties have become more internally homogeneous and ideologically distant. In “The Audacity of Hope,” Obama wrote longingly about American politics in the mid-twentieth century, when both parties had liberal and conservative wings that allowed centrist coalitions to form. Today, almost all liberals are Democrats and almost all conservatives are Republicans. In Washington, the center has virtually vanished. According to the political scientists Keith T. Poole and Howard Rosenthal, who have devised a widely used system to measure the ideology of members of Congress, when Obama took office there was no ideological overlap between the two parties. In the House, the most conservative Democrat, Bobby Bright, of Alabama, was farther to the left than the most liberal Republican, Joseph Cao, of Louisiana. The same was true in the Senate, where the most conservative Democrat, Ben Nelson, of Nebraska, was farther to the left than the most liberal Republican, Olympia Snowe, of Maine. According to Poole and Rosenthal’s data, both the House and the Senate are more polarized today than at any time since the eighteen-nineties.

It would be hard for any President to reverse this decades-long political trend, which began when segregationist Democrats in the South—Dixiecrats like Strom Thurmond—left the Party and became Republicans. Congress is polarized largely because Americans live in communities of like-minded people who elect more ideological representatives. Obama’s rhetoric about a nation of common purpose and values no longer fits this country: there really is a red America and a blue America.

Polarization also has affected the two parties differently. The Republican Party has drifted much farther to the right than the Democratic Party has drifted to the left. Jacob Hacker, a professor at Yale, whose 2006 book, “Off Center,” documented this trend, told me, citing Poole and Rosenthal’s data on congressional voting records, that, since 1975, “Senate Republicans moved roughly twice as far to the right as Senate Democrats moved to the left” and “House Republicans moved roughly six times as far to the right as House Democrats moved to the left.” In other words, the story of the past few decades is asymmetric polarization.