It was early October 2007 in Iowa, the drying stalks still stood in cornfields grown more precious in the age of ethanol, and far away, in the moneyed precincts of both coasts, Barack Obama's top donors were anxious. Despite having raised heaps of cash, he was trailing Hillary Clinton by more than 20 points in the national polls with only three months to go before Iowa's first-in-the-nation caucus. For the worriers, it was getting late. But for Iowa voters, it was early enough that many barely knew what to make of this biracial rookie senator and still muffed his exotic name. In Waterloo, it had been "Hi, Senator Barack." Now, here at the Buchanan County fairgrounds in Independence, it was "Hi, Obama," from a nervous Geri Punteney as she rose in the front row to ask her question.

But it wasn't a question, really. "I have a brother who's dying of cancer," she said, and as soon as it was out she had broken into sobs, and then apologies for her sobs, and more sobs. Obama stepped forward, as if knowing what was expected of him and yet slightly embarrassed by it. He took her hand with some consoling mumbles. Punteney, 50, collected herself enough to tell the rest: her 48-year-old brother was working as a truck driver despite having cancer, so as not to lose his insurance. She herself had stopped working at a riverboat casino to care for her mother, and so was forgoing needed dental surgery. "I don't think it's fair that my brother has to work when he's dying of cancer just to keep his insurance," she said.

And here Obama parted from his two main rivals in the race, Clinton and John Edwards, and from the only Democrat elected president in the past 30 years, Bill Clinton, and from the whole tradition of politics that had prevailed in his party for the past few decades as it tried to maintain its historic bond with the common man. He did not get angry. He did not wrap Punteney in a hug. Instead, he stood still and said in a level voice, "First of all, we're all praying for you." He then told her that he had lost his mother to cancer in her early fifties. And only then looking up at the crowd, he segued into a policy conclusion that was startling for its cool rationalism. Punteney's and her brother's plight, he said, was a sign of the problem in tying health insurance to work without real alternatives. It was wrong that the US was the only developed nation not to guarantee health coverage for its citizens, even as it spent more per person on health care than anyone else. "We don't spend it wisely and we don't spend it fairly," he said.

Moments later, Obama decided to bring up two minor campaign squalls. The day before, an Iowa television reporter had asked him why he was not wearing a flag pin on his lapel, and Obama had answered that he had worn a pin after September 11 2001 but stopped when the pins "became a substitute for I think true patriotism, which is speaking out on issues that are of importance to our national security". Incipient outrage was already spreading online, and Obama wanted to revisit it - to drive the point further. "After a while, you start noticing people wearing a lapel pin, but not acting very patriotic," he said. "My attitude is, I'm [less] concerned about what you're wearing on your lapel [than] what's in your heart." He moved to another internet favourite, his suggestion to some Iowa farmers a few months earlier that they consider growing speciality vegetables, such as rocket (Americans know it as "arugula"), instead of only subsidised crops such as corn and soybeans. "Anybody gone into Whole Foods lately and see what they charge for arugula?" he had said, which had brought derisive reminders from the press that Iowa had no branches of that organic grocery chain. In Independence, Obama went back at it. He repeated his suggestion on speciality crops and mocked the response to his earlier comment. "They said, 'Oh, Obama's talking about arugula in Iowa. People in Iowa don't know what arugula is.'" He paused, and the scorn rose as he continued. "People in Iowa know what arugula is. They may not eat it but they know what it is."

I don’t know where he’s come from, but he speaks on a whole other level, and he transcends the politics of today . . . We’ve been waiting for a guy like this

A year to the 2008 election, on the first of my many tours covering Obama on the trail, it was all here. There was the inability, or reluctance, to capitalise fully on the woes of working-class Americans beaten down by deindustrialisation, globalisation and Republican governance – a characteristic that would become all the more consequential as the economy crumbled over the next year, providing a giant opening for the Democrats that Obama would not deign to exploit to its utmost until Wall Street’s collapse last month, when a surge of voter discontent swept him along

almost in spite of himself. And there was the conviction that flaps over trinkets and leafy vegetables had debased politics, and that voters would agree if he made the case – a conviction that would be challenged as those very two symbols took leading roles in the cultural resistance to him that would be capped by John McCain’s selection for his running mate of a conservative Alaska governor whose every feature seemed designed for contrast with Obama.

Together, these early clues pointed to an extraordinary fact about the 2008 election, nearly as extraordinary as the racial breakthrough it represented: Barack Obama was running not on a record of past achievement or on a concrete programme for the future, but instead on the simple promise of thoughtfulness - the notion that the leadership of the country should be entrusted not on the basis of résumé and platform, but on the prospect of applying to the nation's problems one man's singularly well-tempered intelligence.

Many of Obama's own supporters would miss this, would grow upset later as he moved from a perceived position on the left to the centre on several issues - voting for compromise legislation to permit federal wiretapping, for instance, or speaking out in favour of a Supreme Court decision that strongly endorsed the right to bear arms. Obama is betraying his politics, came the charge from the left. What does he really stand for? From the other side came: he's offering nothing new, it's all warmed-over liberalism. Overlooked was that his politics appeared a matter less of weighing left and right than of judging smart and less-smart, with nods to fairness and common sense tossed in the mix - a matter of hashing things out, hearing out the other side, seeing what worked. He found tiresome many of the usual ideological tussles, wanted to save the fight for the biggest issues - health care, taxes, global warming. His best, most invigorated moments on the trail would come when he was summoning his audience to step back and think, and showing he believed them capable of that, urging them to demand an end to the petty skirmishes distracting from the greater cause. And his true self-betrayals would come in concessions not to conservatism, but to point-scoring or glibness, in his truisms about energy independence, education reform and trade policy, or in his habit of misquoting rivals.

So, the candidate of hope was at bottom hoping that aptitude could trump experience, though his campaign would never put it that way. Instead, it gamely inflated his qualifications. His three years as a community organiser in Chicago were made out to have been a bounty of hard-won successes for the “families devastated when the steel mills closed down”, when his own memoir had defined that episode precisely by its instructive futility. The results of his eight years in the state legislature (concluded only four years prior!) were exaggerated to the point of casting the creation of a health-care task force as an important step on the way to universal coverage. His work in the US Senate on the noble but innocuous subject of nuclear non-proliferation was used as evidence of his ability to work across the aisle. And the campaign would make much of his policy plans, though on most counts, his proposals differed from Clinton’s only in the small print. This was something Obama himself once acknowledged with candour to a roomful of voters in Iowa City, saying that all the Democratic candidates were going to “set up [health-care] plans you can buy into it if you’re poor – if you can’t afford it we’re going to subsidise it, we’re going to emphasise prevention, blah blah blah” – the last three words amounting to as offhand an indictment of campaign plank-peddling as I had ever observed on the trail.

No, as much as Obama claimed that "this is not about me", of course it was. Not about what he had done or what his website said he would do, but about the man and the qualities he appeared to possess - eloquence, self-possession, charisma, decency, lucidity, sobriety, discipline, judiciousness, empathy, wit, verve, acumen, suppleness, equipoise, confidence, style. (Not to mention his grandfather's strong chin and an enviable cigarette-burnished baritone.) In a word, talent. Which meant that Obama was testing one of the deepest schisms in the country's character, the Jeffersonian conundrum. The same Founding Father who saw as America's unique strength its capacity to produce leaders of modest origins but profligate gifts had also set in motion the forces that, two centuries on, would work to undermine such men and women.

In 1813 Jefferson wrote to John Adams to declare that "there is a natural aristocracy among men. The grounds of this are virtue and talents . . . The natural aristocracy I consider as the most precious gift of nature for the instruction, the trusts, and government of society . . . May we not even say that that form of government is the best which provides the most effectually for a pure selection of these natural aristoi into the offices of government?" As if anticipating the Bushes and Clintons, Jefferson had set against his natural aristocrat the nepotism of the "pseudo aristoi", a "traditionary reverence for certain families, which has rendered the offices of the government nearly hereditary in those families".

But it was Jefferson, too, who in his debates with Alexander Hamilton - first treasury secretary, Anglophile and advocate of a strong central government - had exalted the yeoman farmer and the honest earth he tilled, setting his inherent goodness against the corrupted hordes of the metropolis, which Jefferson slapped with the same epithet wielded to effect today: European. "Our governments will remain vir tuous for many centuries, as long as they are chiefly agricultural," he wrote to James Madison in 1787. "When [Americans] get piled upon one another in large cities, as in Europe, they will become corrupt as in Europe."

For two centuries, this rural bias had manifested itself mainly in the woefully unrepresentative US Senate (where Wyoming has the same two votes as New York, despite having one-fortieth its population) and in campaign imagery (Lincoln's log cabin, Teddy Roosevelt's years in Dakota Territory). It would take Nixon and his heirs to distil Jeffersonian egalitarianism to its most potent form, a hardy sour mash of righteous resentment against "elites" that flourished to the point where a savant like Bill Clinton had to lard his brilliance with McDonald's photo ops, while George W Bush, a product of Andover and Yale, settled on a West Texas ranch and found little political cost in mangling his mother tongue like so much cleared brush. A country that demanded only the best - best digital camera, best heart surgeon, best college - had developed ambivalence about undisguised excellence in its highest office.

“I never heard of Obama. I don’t know where he came from. My boyfriend said he gave a speech at the last convention, but I don’t remember that”

This was the asterisk in the American meritocracy that Obama was attempting to erase. And if America was divided, this was the great cleavage confronting him as the self-anointed unifier. To those inclined to recognise potential in the qualities he represented even in the absence of hard evidence of its having been realised, here was a leader unlike any the country had seen for quite a while. These voters thought they grasped at some level what the world was coming to, that it demanded effervescence over solidity, knowledge over dogma, nuance over vehemence, and that it no longer distributed rewards according to years of attendance at the company holiday party. It was the world of the Google boys setting up shop in their Silicon Valley garage, high-school basketball stars going straight to pro, and wars begun by men of deep experience but rigid minds. Many saw this, and were either themselves prospering on these new terms or, if not, had accepted them as the new reality and had decided that the kid with the big skills and short résumé was the one for the moment. "It's a miracle that [Obama's] come aboard. I don't know where he's come from, but it's communication on a whole other level, and it transcends the politics of today," Doug Summers, a screenwriter in Santa Fe, said to me. "It's old versus new. We've been waiting for a guy like this. Like I told my kids: this guy's going to change your lives."

Many others looked at this same new world and were flummoxed by it. To them, it was a betrayal of what they held dear - that time served counted for something, as did work that involved something other than talking or writing or computing. If the winnings were now being given out on the basis of talent and not rank, then why was it that kids of a certain class were riding the escalator faster than ever? For these voters, Obama offered ultimate proof that this new so-called meritocracy of the number-crunchers and keyboard-tappers was bankrupt. That this man who had "done nothing" should be considered for the most powerful office in the world on the basis of certain personal qualities was a bad joke, an affront and, yes, possibly a conspiracy. "I never heard of Obama. I don't know where he came from," said Kathy Zarzycki, a nurse and Clinton supporter in Ohio's industrial Mahoning Valley, a Democratic stronghold where Obama has struggled. "My boyfriend said he gave a speech at the last convention, but I don't remember that."

On these terms would the lines be drawn. The divisions became quickly evident, so much so that one could have taken US census data and predicted most primary outcomes to within a few points. (Clinton's late rally was more than anything a matter of the schedule; the Rust Belt and Appalachian states where she was strong fell later in the order.) There were some exceptions. Geri Punteney, not an obvious demographic match for Obama, told me that day at the fairgrounds that his measured response to her plea had won her over. "You don't have to get outraged," she said. "He seems more down to earth, more mellow." The question was whether the millions of other Geri Punteneys, those who had not seen the thoughtfulness gambit up close, would feel the same way.

The basic biography is by now familiar to most. Barack Hussein Obama was born in 1961 in Hawaii to an 18-year-old white woman with middle-class Kansas roots and a 25-year-old Kenyan exchange student who married her but left two years later to attend Harvard. Moved with his mother, an idealistic anthropologist, to Indonesia between the ages of six and ten to live with her new husband. Returned to Hawaii to be raised primarily by his grandparents. College in Los Angeles and New York, then the organising stint in Chicago, law school at Harvard, and a return to Chicago, where he ran for state senate in 1996. Eight years in the legislature wrapped around a failed run for the US House, then the race for US Senate, blessed by luck when two different opponents were felled by tawdry divorce scandals, and capped by The Big Speech at John Kerry’s 2004 convention in Boston, which got all the speculation started.

Missing from the standard chronology is how, or when, it happened - how a late adolescent adrift in a mild pot habit, misguided fantasies of basketball greatness and steadily accreting racial confusion grew, within a decade, into the outstanding law student many of his classmates predicted would be the first black president. In his own narrative of self-discovery - written at the age of 33 - Obama pinpoints a few key moments: an anti-apartheid speech at his Los Angeles college, where he realised the power of his voice; a turn towards scholarly self-discipline in his next few years at Columbia; his exposure to urban deprivation in Chicago. Underlying it all, he postulates, was the desire to live up to the expectations of, and fill the void left by, the brilliant, flawed and vanished African father.

What matters most, though, is not so much when the light of ambition went on – Obama’s own

account is as reliable as any self-mythology – but that Obama realised that it had, and decided to follow it even at the risk of appearing highly presumptuous. Try as his campaign might to mask it, he knew full well the audaciousness of his bet on talent. It was why he was going for it this time around, despite the calls to wait his turn. If he was to run on potential, it would do no good to dull the package with more years on Senate subcommittees. In 2008, he could still run his race from a skyscraper in Chicago, that great heartland city he’d made his home; any later, he’d be running out of DC, where stardom, like the city’s building heights, was levelled by custom. The timing was ideal in other ways, too. If ever the country would be ready for a hyperarticulate candidate running on a platform of pragmatism and reconciliation, it was at the end of the Bush years. Even more opportune was the presence of Hillary Clinton. What better way for a candidate whose biggest weakness was his inexperience than to prove his worth in the act of campaigning – by slaying a giant?

What matters most is not so much when the light of ambition went on, but that Obama realised that it had, and decided to follow it

In August 2007, Obama visited with some Democrats in New Hampshire, the first primary state. Carol Moore, a former state representative, told him she was tired of losing her heart to cerebral, liberal insurgents like Bill Bradley, the former New Jersey senator and New York Knicks player who ran against Al Gore in 2000, only to see them crushed by the Democratic machine. She had decided never to fall for one again, but still she wanted to know from Obama why he would be any different. How could she know he would be able to see it home? "Once I win the primaries, no one's going to doubt my credentials or my toughness," he said, as she relayed it to me. "If I beat Hillary, no one's going to doubt my electability and my qualifications - or my strength."

But Clinton would be most valuable of all as a foil, so well did she offset Obama's "new politics" mantra. There was the war in Iraq, which Obama had opposed, not because it was wrong but because it was "dumb" and "rash". Obama's anti-Clinton critique extended further. His speeches were riddled with put-downs no less effective for leaving her name unsaid, as he called for a politics that "wasn't based on spin and PR" or "triangulating and poll-driven positions". He said he was not running "because of some long-held ambition", and turned every barb from Clinton into a teasing new take on the failures of the "old politics", infuriating her camp.

And Clinton played to form. She seemed to have it both ways in a debate answer on immigration. Her campaign dredged up writings from Obama's Indonesian kindergarten purporting to show that he, too, had nursed presidential dreams. And, one snowy December day at a law office in New Hampshire, one of her top advisers there, a lawyer married to the state's former governor, responded to my questions about Obama's rising poll numbers by expressing his deep concern that, if Obama were to win the nomination, Republicans would inevitably bring up his admitted cocaine use as a young man, and might ask whether he had ever been a drug dealer, because, after all, they would argue, who really knew the extent of it?

In Iowa, where Obama had spent by far the most time, the equation tipped his way. In New Hampshire, where the Clintons' roots ran deeper, she shocked us all, winning by three points just a week after her campaign was caught trying to make its crowds appear larger than they were. Here, Obama's bet on raising the level of discussion had failed. In a debate three days before the vote, he found himself in an awkward position. The moderator had asked Clinton about her low likeability ratings, and her charming response had included praise for Obama. He had to say something, and his rejoinder - "You're likeable enough, Hillary" - made for the most charged and grown-up moment between the two of them all campaign. Saying any more would have been phoney, given her recent attacks on him, yet for all that he still felt some affection for her, and expressed it in the dry tone of a 1940s leading man. She got this, and accepted it with an equally wry and genuine smile. But the adult irony did not translate on screen, the line became enshrined as smug condescension, and, combined with her welling up on camera two days later, it helped rally New Hampshire women to her in big numbers on primary day.

We would get much else wrong. In South Carolina, we would unfairly blame Hillary’s wipeout on Bill’s blundering, when in fact his one big stumble came after most of the votes were cast, when he compared Obama’s victory to the 1984 and 1988 South Carolina wins by Jesse Jackson, the demonstrative civil rights leader who shares little with Obama but the colour of his skin. Obama’s rout was instead the result of the shift of the African-American vote after Iowa – which offered proof “they” would vote for him – a shift that was sealed by his transformation on the South Carolina stump into a master of Southern black vernacular. We would also be slow to recognise that Obama had put himself in position to win the nomination by holding his own on Super Tuesday, the early February day on which 22 states voted. And as it dawned on us, we tried to explain it away, to pretend that this gathering upset was about things other than his talents and the electorate’s gradual reckoning with them. Doing so would have required acknowledging his abilities to an extent that the press felt uncomfortable doing, spooked as it was by Clinton’s (and later John McCain’s) charges of favouritism.

So we focused instead on Obama's record-breaking fundraising, his cohesive brain trust, his stellar ground organisation, his decision to rack up delegates in states with caucuses, where voters expressed their preference in public and where liberal Democrats dominated. All these things were true, but by emphasising them we skirted the obvious, that Obama had been in a position to win those caucuses, starting with Iowa, only because many Democrats had fallen for him in a way they never would for Hillary. Because he had it and she did not - the promise of a true break from the past and the ability to walk two or three times a day into an arena of thousands and pull them in and lift them up with barely a shout, so modulated and assured was his 45-minute, off-the-cuff stump speech. There was, it turned out, a difference between being able to inspire and to rally, which is why Clinton would fall short even as her pitch grew more insistent. The sheer relentlessness could do nothing for the hint of joylessness her campaign had laboured under from the start.

We tried our best to ignore this. In April, many of us gathered in Butte, Montana, a once-thriving silver and copper town that now stands, half-abandoned yet stately and surreal, on a bluff beside a huge abandoned mining pit. There, Obama and Clinton addressed Montana Democrats in a rare joint appearance. He, delivering the norm, and somewhat perfunctorily, held the hall rapt. Whereas she offered herself as merely a more famous variation on the local officials who also spoke - punches at the air, a charge of partisan particulars, the laundry list of correctives, all of it nearly as classic and time-worn as that city on the bluff. The crowd half-listened, the line for Moose Drool beer grew longer. The story of the campaign was laid bare in the contrast, but we, sitting back in the rafters, averted our eyes as best we could. Despatches from the event focused on this or that new charge or policy twist, as if the candidates had even been competing in the same decade.

Clinton herself was more honest on this score, and knew she could never close the gap, which was why she launched her critique of the oratory itself as "empty words". This resonated with those already inclined to distrust Obama, but had limits as an attack, because what was politics but the attempt to communicate with voters and other politicians and bring them to one's side? More than that, it ran up against the fact that Obama was not a political performer in the conventional sense. Most American stump showmen - Huey Long, Ronald Reagan, Bill Clinton - plunged into the moment, lost themselves to it and the narrative created for them. Obama did not. He relished the crowd, sure, but he did not entirely open himself up to it. Even in the biggest arenas, he retained a hint of the observer's remove, which was to be expected given that he was, after all, a writer. His memoir, Dreams From My Father, if clichéd at moments, had demonstrated a true writer's sensibility, and his political tract, The Audacity of Hope, had managed at points to transcend that form.

This aspect of the candidate was, to me, the biggest mystery of all. Most of us tend towards one pole or the other, observer or participant, critic or actor, consultant or candidate - one burdened by heightened self-awareness, the other liberated, if not always for the best, by a shortage of the same. Here was someone who possessed enough self-consciousness and detachment to have noted, in The Audacity of Hope, telling details of his first visit to the White House - of Bush's asking an aide for a squirt of hand sanitiser in a moment alone with Obama, of the Secret Service's reaction when Obama set his hand on Bush's shoulder. Yet here he was, day after day, also satisfying huge crowds who all expected a revelation. In the week leading up to Super Tuesday: 13 cities in 11 states. In one day alone: 14,000 in Boise, 18,000 in Minneapolis and 20,000 in St Louis. This hybrid character type confounded Clinton's attempt to cast him as a mere itinerant troubadour. But it also helped explain why not all the voters he met were signing on. At some level, perhaps, they could sense that even as Obama was shaking their hand or asking about the new machine on their shop floor, he might be filing something away for the next book.

We could see this unusual blend behind the scenes as well. On his ventures back to the press section of the plane, he bantered with the challenging bravado of the self-confident politician, leaning forward in his trademark white shirt into the scrum of mini-tape recorders and cameras. But even here there was a reserve, a not entirely concealed disdain for our clamouring questions and whatever dust-up of the day they happened to revolve around, as if he thought that presumably well-educated scribes like us ought to know better. There was also a slight reticence one on one. Assigned as the lone "pool" reporter to cover him on a detour from the campaign trail in Ohio to Illinois to attend a memorial service for five college students killed in a classroom shooting, I had several amiable chats with him on the chartered private jet.

We spoke about our kids, home towns, sports. But there was a limit to the engagement - unlike Bill Clinton, this was not a candidate who needed to win over every person presented to him, to convey as much of his brilliance as possible in each encounter. It was as if Obama was held in check by a constant sense of perspective. He enjoyed good conversation, but there were a lot of reporters to meet along the way, a lot of people, and only so many hours in the day. At some point on our return flight, it became clear he just wanted to read the papers, which he did, ploughing through the New York Times and the Wall Street Journal, front to back.

Even as he stayed on course to claim the nomination, he felt demoralised over the campaign’s detour into the cultural shadowlands

One April day on her way back from a lunch break in the withered steel town of Charleroi, Pennsylvania, Theresa Krall, a social worker and Democrat, said she liked Clinton the best. Between Obama and McCain, she wasn’t so sure. “Obama speaks wonderfully, and he’s very intelligent. But I’d have to research more about him against McCain.”

Such was the nature of Obama's bet on meritocracy that, at some point along the way, the primaries against Clinton seemed to blur into the looming general election against John McCain. The conservative, blue-collar Democrats who sided with Clinton in the Pennsylvania primary following the rantings of Reverend Jeremiah Wright, the long-time minister of Obama's Chicago church, were the same voters Obama would be fighting hardest for months later against McCain. McCain and Clinton echoed each other in savaging Obama for his poorly phrased musings, at a fundraiser in San Francisco, about "bitter" small-town Americans and their tendency to "cling" to guns and religion. It was a riff that showed Obama's analytical side in a less flattering light, drew attention to the very cultural gap he was seeking to close, and was only partly mitigated by having been intended as a defence of rural America against charges of bald racism. Even as he stayed on course to claim the nomination, Obama's demoralisation over the campaign's detour into the cultural and racial shadowlands was plain - the listlessness that some saw on his face at a Philadelphia debate where Clinton brought up his association with a 1960s domestic terrorist-turned-college professor was more like aggrieved disbelief that it should have come to this. Urged to counter during this difficult Rust Belt phase, Obama maintained his trademark equanimity but also stoked false fears about Clinton's health plan and misquoted her as praising the North American Free Trade Agreement. (Later, he would also take out of context McCain's comments about the need for a 100-year US presence in Iraq and about the economy.) Clinton hurt her own cause with embellished tales of landing under sniper fire in Tuzla, Bosnia, ammunition for those who did not trust her word. But Obama's true deliverance came, at last, when both Clinton and McCain tried to push their populist advantage too far, clubbing Obama for his refusal to go along with their call for a summer-long gas tax holiday. It was a classic gimmick (in reality, it would save consumers little) and one that seemed perilous to oppose, but it was just the foothold Obama needed to return to the high ground as he ridiculed the idea at every stop.

This was Obama in his element - quoting experts, appealing to higher reason: "The American people are smarter than Washington gives us credit for." And it worked, as he put the race away in North Carolina. Shortly before the final primaries, he embarked on a victory lap through Oregon, and here was Obama as he wanted to be seen. On the long flight west, he made a rare extended visit to the back of the plane for a word game with the press and aides. (Having to lead his team-mates to the word "revolution" without resorting to the words "American" or "French", he offered: "Thomas Jefferson called for it to happen every once in a while." When none of his team-mates knew the answer, he went more middlebrow: "A Beatles song.") In Eugene, he paid an unannounced visit to the University of Oregon's legendary track in the middle of a big annual athletics meeting; once the women's two-miler was done, the meet came to a halt and the candidate strolled around the track, bathed in sunset and waving up to the old grandstand as the crowd - the ultimate Obama demographic, track enthusiasts in the Pacific north-west - rose as one to applaud. An hour later he spoke to thousands on campus, aglow with self-reflective triumphalism.

"There've been times where you get whacked so many times that after a time, you feel you have to whack back. You've got to go negative. You don't want to look like a wimp," he said. "The times . . . I'm most proud of is when we resisted the impulse, and the times that I'm least proud of is when we succumbed to that impulse."

But the turn towards McCain would renew the original challenge in a different form. Against Clinton, his best card had been the one he wanted to play, the call for new politics. Against McCain, he was expected to wield a blunter instrument: the charge that McCain’s party had made a complete hash of everything. An obvious tack, but it complicated his call for post-partisan reconciliation. His stump speech had from the start been full of economic prescriptions – though these were often overlooked in the focus on his rhetorical crescendos – and he had given several serious addresses on Wall Street reform, demonstrating a deeper engagement on the subject than McCain. Yet the pitch lacked an overarching thrust, a lunge for the Republican jugular. It was as if Obama was held back by what he knew, that much of the country’s economic pain had as much to do with global upheavals as with the Bush tax cuts and trade policies. As if he knew his forays into populism in the primaries had edged into protectionist demagoguery, and that an African-American candidate could afford to sound only just so angry and intense. But it was also as if he thought that winning on the economy was somehow too easy and more conventional than what he had really set out to do in this campaign.

To those unnerved by his gifts, the African roots and Muslim name only ratcheted up the distrust. Where do they make people like this? Not around here

Chatting with me early in the primaries, David Axelrod, Obama's strategist, said not to expect the usual partisan fire from Obama: "He's never been an aficionado of the cheap applause line. Voters get sold short. They're smart and sophisticated. They realise that it's important to replace a Republican with a Democrat, but that it won't do enough", if "all we do is change parties without challenging our politics".

This sort of talk unnerved Democrats who did not understand why many of Obama's ads against McCain did not hammer the Republican brand more aggressively. They fretted about how Obama's discursive style would fare in the debates, even though he had shown improvement over the course of his countless encounters with Clinton. Obama's challenge on the economy also involved an awkward political reality. Even as the weak economy was making him the alleged favourite, he was in fact lagging in states and regions that had been hit the hardest under Bush. Conditions had only worsened across the Rust Belt since 2004, when Kerry won Michigan and Pennsylvania and narrowly lost Ohio, yet all summer Obama was barely holding on to the first two and lagging in the third. Meanwhile, the former Republican strongholds he was hoping to nab, Virginia and Colorado, were states that remained relatively prosperous.

The easy explanation for this paradox was racial prejudice, but something broader was at work. Over the past decade, Democrats had gained with college-educated professionals and in the more dynamic regions, in formerly Republican areas such as northern Virginia and the Philadelphia suburbs, while Republicans had claimed distressed states that had been Democratic, such as West Virginia, places that felt increasingly disconnected from the megalopolises that candidates of the Kerry and Gore kind most represented. Leading Democrats were worrying about Obama's struggle to carry conservative "Reagan Democrats" whom Clinton had won in the primaries in places like western Pennsylvania, yet many of them had already deserted the party for Bush, and Kerry and Gore had carried that state anyway. Sure, Clinton had a better shot at reclaiming them, and Obama could not lose too many of them, but they were no longer the base of the party, certainly not of his party.

Not that race didn’t play into it. For some, particularly older voters, Obama was the first affirmative-action nominee, period. Bob Norman, a retired awning maker, was blunt when I came to his door in the working-class Cleveland suburb of Parma. “The media’s letting him sled on down the hill because he’s coloured,” he said. Marlene Newlon, a landlord in Grafton, West Virginia, offered a more direct variant as we strolled down the street of that shrivelled railroad town: “I’m not voting for a black president. I’m sorry.”

On the flipside, who knew how many of the young voters who were shattering all assumptions about their endemic apathy were doing so to flaunt their tolerance, a benefit gained from a youth spent listening to Kanye West and imitating Kobe Bryant? And there was the basic fact that what Obama had done was essentially to carry the same highly educated liberals attracted to reformers past (Bradley, Howard Dean, Gary Hart) and then, thanks in part to his heritage, add to them black voters, breaking up the coalition that had helped past machine Democrats stave off the highbrow types.

The more interesting role of race, though, lay in the way Obama's ethnic particulars exacerbated the split perception of his talent. To those unnerved by his gifts, the African roots and Muslim name only further ratcheted the distrust: Where do they make people like this? Not around here . . . Meanwhile, among his admirers, much was made of Obama's ability, as a biracial American, to embrace and represent all. But was it not also true that for many his appeal lay in all the things he was not - that his confounding of the stereotypes on either side only made him more appealing for those inclined to admire such singularity? Consider a black politician of equivalent aptitude with typically African-American roots. Would not his uniqueness have been more obscured than Obama's by discussions of racial redemption and his embodiment of a group? And consider a young white Democrat with the same outsized skills. Would the ambition that drives Obama be writ more conspicuously, and unappealingly, on the face of a rising star who could not, like Obama, be said to be putting his talents towards the cause of national reconciliation? If this bright young white candidate gazed into the distance as is Obama's wont, chin upturned just so, might it appear to voters as if he were thinking only of himself, not of history?

Thus did the elite candidate who happened also to be the first serious black candidate move into the home stretch. McCain picked up where Clinton left off with his “Country First” theme, casting Obama as a self-aggrandising climber and going so far as to charge that “Obama would rather lose a war in order to win a political campaign”. There were the “celebrity” ads, which in a clever twist showed Obama in a dazzling light, smiling broadly before 200,000 Berliners – he is a star, you are not, and he knows it. But the most direct counter to the audacity of talent would come from Sarah Palin.

Here was a culture warrior from the front lines, her pro-life convictions evident in her decision to carry to term a baby with Down's syndrome and to encourage her pregnant 17-year-old to marry and start a family. In her convention speech, Palin delivered with gusto a speech that bristled with rural resentment, including a quotation from the notorious arch-conservative, anti-Semitic columnist Westbrook Pegler: "We grow good people in our small towns, with honesty and sincerity and dignity." This was followed by a string of bold appeals to the heartland, including her declaration that "it's time that normal Joe Six-Pack American is finally represented in the position of vice-presidency".

Challenges to her qualifications brought retorts about Obama's own swift rise, to which Obama partisans wanted to respond that it was less about résumés than about capacity and learning and engagement with the broader world. They wanted to say this, but with the spectre of Jefferson's yeoman looming, they mostly held their tongue and grimaced - until, to their delight, Palin's glaring struggles in her television interviews made their case for them.

Palin's stunning burst into the Outside, as Alaskans call the Lower 48, was soon enough trumped by cataclysm, the Wall Street meltdown. And within two weeks, the financial crisis had, among other things, utterly reshaped the presidential race, throwing a bevy of key swing states into clear leads for Obama. This was received as predictable - any evidence of economic plight worked to Obama's benefit, right? But this had not been the case all year. The economy had been in decline all campaign, and yet Obama had failed to open as wide a lead as many Democrats thought he should. Yet something was different now. At one level, the sheer scale of the devastation was prompting a partisan shift that brushed aside all else, a furious lashing out at the incumbent party, maybe even an ideological epiphany. Beyond that, however, it seemed possible that Obama was at last benefiting from the politics of thoughtfulness. Voters who had lost their jobs in Steeltown, USA a year ago had wanted to see more passion than Obama was able or willing to provide. But voters terrified and perplexed about this more abstract, far-reaching crisis wanted something else: to know that the people in charge were thinking hard and carefully before acting. They wanted to see serious deliberation among wise men. Many voters rightly saw the crisis as the fault of the coastal elite, the MBAs who had made their millions pushing paper in New York and the hedge-fund outpost of Greenwich, Connecticut. But voters also saw that a crisis this grave and convoluted required an elite response, measured, informed, competent. And as Obama reacted with his usual restraint - restraint bordering on detachment - and as McCain jerked into a more haphazard response, the needle moved. McCain, so long simply a default option for voters wary of Obama, had, between the Palin pick and response to the crisis, come to seem "unsafe" in his own right, polls showed.

Voters saw that a crisis this grave required an elite response, measured, informed, competent. McCain had come to seem “unsafe” in his own right

The numbers continued to move after their first two debates, where Obama met McCain’s visible contempt with yet more calm, to the frustration of some of his own supporters. Just like that, he was suddenly in command, finally riding the current of economic distress, and positioned to close the deal with the far superior ground organisation and money machine he had been building all year. McCain, meanwhile, made a final attempt to frame the out-of-nowhere talent he was up against as threatening. Palin started it off, with new warnings about the radical/professor whom Obama once knew in Chicago.

"Our opponent . . . is someone who sees America, it seems, as being so imperfect, imperfect enough that he's palling around with terrorists who would target their own country," she said on the trail in Colorado. "This is not a man who sees America as you see America and as I see America." Such remarks provoked a virulent sort of enthusiasm from members of the party base who now dominated McCain-Palin events. Among undecideds, however, Obama again appeared to be reaping the benefits of his demeanour - exotic as he was, his extreme moderation made it harder for voters to picture him in cahoots with bomb-throwers. His surrogates scolded McCain and Palin for inciting the crowd, and a somewhat abashed McCain tried to switch to a more temperate tone. And Obama plodded happily along, proposing a new $60bn economic relief package on 13 October before retiring to Toledo, Ohio - selected for its proximity to all those long-suffering post- industrial voters who seemed to be finally warming to him - to prepare for the final presidential debate.

Before Palin's debut, and before the markets fell apart, Obama had his final chance at a mass, unmediated presentation of the meritocrat's case, at the Denver convention. The McCain camp howled at Obama's choice of a 76,000-seat stadium and the classical backdrop constructed for the stage, and even many Democrats cringed, but they should have seen it coming. This campaign, after all, had been launched on aspiration and presumption. Not for him the cramped indoor podium where a hundred other pols had all, in their way, made the customary appeal to the average voter in the average state. Barack Obama was betting, again, that America was ready for a little elevation.

So much hung in the dry, thin air of Denver - not just the rest of this astonishing campaign, but also everything after that. What would come of all this if he made it, and finally had the chance to embark on the work he had spent 21 months talking about? How would a post-partisan candidate rein in a Democratic congressional majority out to run roughshod? How would the decisions be made at day's end when all due deliberation had run its course, and the thoughtful unifier was in the position of having to make one side or the other unhappy? (Most immediately: would he stand by his calls for middle-class tax cuts and big spending on health care, alter native energy and infrastructure, even as the deficit hawks claimed these things were now unaffordable? Would he raise taxes on the wealthy and on capital gains even in a deep recession?) How would his reform push fare in a Washington so calcified by lobbyists and corporate lucre, and how disappointed would his expectant base be by the inevitable lapses and compromises?

Most of all, how would this man, who had never run anything but a law review, a Senate office and his campaign staff, fare as national leader in a time of ultimate stress? From early in the campaign, he had opened his stump speeches by declaring he was running now because "we are at a defining moment in our history" and "cannot afford to wait". To some ears, it sounded grandiose. Now, it was all too true. Was this more than he had bargained for - or was this just the sort of moment for which the natural aristocrat was made?

But first, the speech. He laced it with enough partisan fire to satisfy the back-seat drivers worried that he had been insufficiently tough in the face of McCain's barrage. Yet near the end he managed to return to the idea that got it all going back in Iowa a year earlier, and in Boston three years before that, only now it came with the extra edge of having seen just how many were hoping he would fail.

"For part of what has been lost these past eight years can't just be measured by lost wages or bigger trade deficits. What has also been lost is our sense of common purpose - our sense of higher purpose. And that's what we have to restore," he said. "I know there are those who . . . dismiss such beliefs as happy talk. They claim that our insistence on something larger, something firmer and more honest in our public life is just a Trojan horse for higher taxes and the abandonment of traditional values. And that's to be expected. Because if you don't have any fresh ideas, then you use stale tactics to scare the voters. If you don't have a record to run on, then you paint your opponent as someone people should run from . . . You make a big election about small things. And you know what - it's worked before. Because it feeds into the cynicism we all have about government . . . But I stand before you tonight because all across America something is stirring."

With just a couple weeks to go, amid earth-shaping gusts even stronger than what he has been creating for nearly two years, it's looking like he might be right.

Alec MacGillis writes the "Letter from Washington" for the New Statesman