The following correction was printed in the Observer's For the record column, Sunday June 15 2008

The article below refers to failed impeachment proceedings against President Bill Clinton and attempts to impeach him. We should make it clear that Mr Clinton was successfully impeached by the House of Representatives but later cleared by the Senate.

From the beginning, it was Hillary who took charge. 'Look,' she told the future President of the United States in a wood-panelled reading room at the Yale Law School library, 'if you're going to keep staring at me, and I'm going to keep staring back, we should at least introduce ourselves. I'm Hillary Rodham.'

The year was 1970, and Bill Clinton and Hillary Rodham had just embarked on a lifetime of cinematic personal and public highs and lows. As she did that flirty afternoon in New Haven, Connecticut, Hillary played a dominant role, one greater than many Clinton watchers ever realised. With her discipline and determination and his almost supernatural charm, they made for a seemingly unconquerable couple - their marriage vows as resistant to scandal as their power was to political enemies.

But now, in the closing days of the 2008 Democratic presidential race, the Clintons' luck is running out. Blindsided by Obamamania, Hillary is almost certain to concede defeat in the weeks to come, abandoning her dream of a restoration, of returning to a White House the Clintons clearly assumed would be theirs again some day.

Indeed, the long-reigning royals of the Democratic Party must contemplate their greatest defeat with disbelief. Their lifetime in politics has often been an absurd high-wire act, a constant defiance of natural political gravity. Time and again they seemed beaten, broken, shamed - and each time they bounced back stronger than before. The question now, in the twilight of Hillary's campaign, is whether they can mount one last comeback, or whether the curtain has fallen at last on the Clinton era.

They were always an unlikely pair. He was a rural Arkansas boy raised in a turbulent household. She came from an upright middle-class Chicago suburb, raised by stern Methodist parents. But their shared brilliance and a passion for politics bonded them, not least against Bill's taste for a less cerebral brand of woman.

After law school, Hillary followed her love down to the alien terrain of Arkansas. Even then, Bill was a minor legend, a Rhodes Scholar with irresistible charm and endless potential. Though they were always equals in private, Hillary was content to subordinate herself publicly to his ambitions - although she may have simply been biding her time. Last year, Her Way, a Clinton biography by Jeff Gerth and Don Van Natta Jr, claimed the couple had a 'secret pact of ambition', a written 20-year plan to make each of them president. The Clintons deny it.

Elected Arkansas governor at just 32, Bill suffered an early disaster - many more awaited him - when voters rejected him after one term. Undaunted, he campaigned his way right back into the job in the next election. Safely re-ensconced, Bill made his name with a rare combination of personal appeal and policy wonkery. Hillary, who worked as a lawyer and focused on children's advocacy, was a more awkward fit. To accommodate Southern social mores, she adjusted her north-eastern feminist style of dress, hairdos, and even her name (she would grudgingly trade in Rodham for Clinton) - the first of countless image adjustments she would resent even as she perfected them. Behind the scenes, however, she brought discipline and order to his unruly habits.

And as he came of political age in the Eighties, Bill's timing was perfect. The Democratic party was suffering a crisis of leadership. Though now derided by some Democrats as a relic, even a reactionary, back then Bill was - much like Barack Obama today - a bold reformer. The humiliating routs of Walter Mondale in 1984 and Michael Dukakis in 1988 convinced Democratic elders that the party had to shake its image of northern liberalism, which was driving working-class white voters into the arms of the Republicans. With his Southern drawl and moderate 'New Democrat' philosophy Bill Clinton offered an ideal new face. 'The New Democrats saw themselves as reformers who ran against the entrenched interests of the Washington Democratic party and thought it was out of step with the rest of America,' says Kenneth Baer, editor of the policy journal Democracy and author of a book about the New Democrats.

Thus in 1992 Bill Clinton was strongly pro-trade and argued that Democrats must embrace big business (in contrast to the gritty anti-trade, anti-Wall Street populism of Hillary today). But his signature issue was welfare reform that would require the poor to work for their benefits - an idea with intellectual merit that also signalled to white voters that Clinton would no longer blindly throw their tax dollars at the inner city. (Clinton also defied black political leaders. In 1992, he enraged Jesse Jackson by denouncing the rapper Sister Souljah as racist, at a conference hosted by Jackson.) While Clinton did have a populist bent of his own - he supported middle-class tax cuts and universal health care, and his slogan called for 'Putting People First' - his goal was to recast Democrats in a more moderate image.

In a preview of the Clintons' future, Bill's 1992 presidential campaign was nearly snuffed out by personal scandal. Longtime whispers of Clinton's womanising grew louder when a sultry blonde named Gennifer Flowers went public, claiming to have had a long-term affair with him. That forced Bill and Hillary to sit for a famous national television interview in which Bill pleaded contritely to 'causing pain in my marriage'. Then came the punishing revelation that Clinton had evaded the Vietnam War as a young man. But after plummeting in the polls, he rebounded to finish a strong second in the critical New Hampshire primary, a result celebrated as tantamount to victory, and which propelled him to the nomination. He was famously branded 'The Comeback Kid', a taste of things to come.

It was the start of America's long and wearying acquaintance with Clintonian drama - as well as a growing obsession with Hillary's role in their marriage. During the Gennifer Flowers scandal, Hillary argued that, while she understood Bill's flaws, she wasn't just 'sitting here as some little woman "standing by my man" like Tammy Wynette'. The truth is that Hillary seemed to have committed herself to both a marriage and a political partnership with Bill, with all the flaws that entailed. But the line sounded like a sneer at the millions of 'little women' apparently less enlightened than Hillary. Soon after, Hillary would defend her role as a working mother (she was a white-shoe Arkansas lawyer) by saying, 'I suppose I could have stayed home and baked cookies and had teas.' Hillary had thus introduced herself as a snobby careerist with seeming disdain for middle American women. It is an image she has spent 16 years working to shed, perhaps finally succeeding only in these waning days of her own presidential campaign.

In many ways, Barack Obama's campaign has eerily echoed Bill Clinton's 1992 candidacy. Bill, too, campaigned on the themes of change and hope. (He was even 'the Man from Hope'.) Clinton promised to cure a national malaise (in 1992 a recession was under way - prompting the famous Clinton slogan, 'It's the economy, stupid'). And like Obama, Bill contrasted his youth and freshness against an entrenched Washington insider: George HW Bush, father of the current president. Clinton was elected by a comfortable margin. On his inauguration day, he was just 46 years old - a year younger than Obama is today.

The Clinton era began, fittingly, in a state of chaos. New to Washington, the Clintons were disorganised and error-prone, thanks in part to an inexperienced staff and Bill's own lack of discipline. They mishandled cabinet appointments. US forays into Somalia and Haiti became fiascos. Clinton dithered as war raged through the Balkans. And a clumsily-executed bid to allow gays and lesbians to serve openly in the military enraged generals and even some Democrats. (Bill did push through a landmark economic plan, raising taxes on the wealthy to start reducing a huge budget deficit.)

And the scandals kept coming. A conservative magazine reported new tales of Bill's sexual adventures in Arkansas, including the claim by a former state employee named Paula Jones that Bill had invited her to a hotel room in 1991 and exposed himself, asking for sex. The New York Times probed the Clintons' investment in a failed real estate venture known as Whitewater, and whether Bill had improperly intervened with regulators. (Years later the two stories would conjoin and explode, with the unwitting aid of a young White House intern.) The Clintons' lawyer and longtime friend, Vincent Foster, buckled under the strain and shot himself. It was indicative of the passions the Clintons stirred that conspiracy-mongers claimed Hillary was having an affair with Foster, and perhaps had him killed. (One right-wing congressman even re-enacted Foster's death by shooting a melon in his back yard.)

Things were complicated by Hillary's remarkable influence. White House aides understood that she was effectively a co-president, particularly after Bill granted her an office in the West Wing. The media was sceptical of this unprecedented role for a First Lady. It didn't help when - in a fit of spirituality she has long since kept under wraps - she issued a grandiose call for a new 'politics of meaning' to waken America from 'a sleeping sickness of the soul' and help people find 'some core level meaning in our individual lives and meaning collectively'. (One disdainful profile in the New York Times dubbed her 'Saint Hillary'.)

But nothing compared with the debacle of Hillary's attempt to reform the US health care system. At times Hillary showed herself a deft politician as she charmed congressional barons at committee hearings and in back rooms. But ultimately her effort was defined by alleged secrecy, arrogance, and micro-management. A fierce and demagogic campaign by business groups and partisan Republicans wrecked the grand project when it reached Congress in mid-1994. Hillary hatred reached new heights. 'Clinton entered the political scene as the first First Lady to offer herself up as an equal partner to the presidency,' The New Republic's Jonathan Chait recently noted in the Los Angeles Times. 'Tens of millions of Americans, mostly men, developed an irrational hatred for her... She simply represented the strong woman to them, and they hated her for it.'

Led by the firebrand Georgia Congressman Newt Gingrich, conservatives stoked white working-class voters into a rage at a White House that seemed clueless, liberal, and run by a woman, for God's sake! In the 1994 midterm elections Republicans seized control of both Houses of Congress for the first time in four decades, with plans to severely trim the size and scope of the government. At a press conference soon afterward, Bill Clinton was pathetically reduced to declaring, 'The president is still relevant here.' Few in Washington believed it. Indeed, for much of Clinton's first term, he appeared destined for a one-term presidency that history would record as a grand failure.

It was an act of terror that kickstarted the next Clinton comeback. On 19 April 1995, a truck bomb devastated a federal government building in Oklahoma City, killing 168 people. The bombing, perpetrated by domestic right-wing radicals, was Bill Clinton's first chance to play the great protector-healer during a moment of national trauma. 'The president grew a foot taller as he began to fulfil his new role on the American stage,' biographer Nigel Hamilton has written.

Clinton also benefited from the overreach of his legitimate political rivals - the egomaniacal Gingrich, speaker of the new Republican House of Representatives, and Gingrich's ineffectual Senate counterpart, Bob Dole, who was to challenge Clinton's re-election in 1996. Gingrich and Dole pressed for large budget cuts that Clinton spun as an assault on health, education and environmental spending. When Republicans forced a brief budget shutdown over the dispute in late 1995, Clinton cast them as scary radicals - and the public agreed.

Meanwhile, Clinton asserted himself abroad, finally using air power to quieten the Balkans and brokering new Middle East peace talks (though he failed to prevent genocide in Rwanda). He also enlisted the Machiavellian pollster Dick Morris to help him chart a centrist course between the Gingrich-Dole Republicans and his party's left wing. In the summer of 1996, Clinton signed a Republican welfare reform plan that he felt was too harsh on the poor after Morris warned him that rejecting the bill would cost Clinton his re-election that autumn. On the back of a humming economy and peaceful world, Clinton coasted past the 73-year-old Republican candidate Bob Dole, who seemed a slightly doddering relic from another era, by a nine-point margin.

'Do you know who I am?' Clinton asked Gingrich during one of their contentious budget negotiation meetings, as it was dawning on the Republican that the president was still relevant after all. 'I'm the big rubber clown doll you had as a kid. And every time you hit it, it bounces back. That's me - the harder you hit me, the faster I come back up.' Boing, boing, boing.

In the wake of the health care debacle, Hillary had suffered an extreme crisis of self-confidence. She turned to self-help gurus, including the spiritual adviser Jean Houston, who encouraged Hillary to conduct imaginary conversations (some branded them séances) with the late Eleanor Roosevelt. She lamented a media culture that glorified scandal over policy, for which she had a passion, publicly deriding those who 'believe that being negative is clever, being cynical is fashionable'. Yet she also understood political reality. Her public foray away from the ceremonial role of First Lady had been a disaster, and the smartest response was simply to lay low. By the time Bill's second term began in 1997, Hillary had taken on a soft-focus role, writing a book about children's welfare (It Takes a Village) and travelling abroad to promote women's rights. Behind the scenes, however, she was still her husband's most important adviser. She quietly urged him to sign the welfare bill, and pressed him to choose her close friend Madeleine Albright as secretary of state, which in turn enhanced her influence over foreign policy.

At the start of Bill Clinton's second term he seemed to be operating almost on autopilot. The stock market was booming and the world was calm. Republicans were humbled by their 1996 electoral defeat. Bill had grown into the presidency. Was it possible that the Clintonian drama had plateaued at last?

No. Just before dawn on Wednesday, 21 January 1998, Bill Clinton shook his wife awake. 'You're not going to believe this, but... There's something in today's papers.' That 'something' was the news that, from 1995 to 1996, Clinton had carried on an affair with a former White House intern named Monica Lewinsky.

Like stars colliding to form a horrendous black hole, the Clinton scandals had finally converged. Lewinsky had been discovered - thanks to her treacherous 'friend' Linda Tripp - by lawyers for Paula Jones, who was pressing a sexual harassment lawsuit against the president (which he had foolishly refused to settle). Soon, word reached Kenneth Starr, a conservative judge leading an official investigation into Whitewater. Starr began examining whether Clinton had obstructed justice by telling Lewinsky to lie to Jones's lawyers. His resulting 'Starr Report', in September 1998, chronicled the affair in pornographic detail, right down to the president's erotic use of a cigar.

Bill almost didn't make it. Many pundits declared that if his initial denials - 'I did not have sexual relations with that woman' - proved false, he would be finished. Clinton's lie (repeated to Starr under oath) was exposed by the semen-stained dress Lewinsky had once worn on one of the many days she 'serviced' the president, including while he was on the telephone to congressmen and world leaders. With Clinton exposed by DNA evidence and Republicans preparing impeachment proceedings, top Democrats privately discussed forcing out the president, so Al Gore could claim the office in time to erase the Lewinsky stain, as it were, for his 2000 presidential bid.

Once again, the Clintons were blessed by useful enemies. Starr and the leaders of the Republican impeachment proceedings overplayed their hands, and came across as puritanical zealots. The economy continued to thrive: incomes were rising, poverty was falling, and the Internet boom was minting huge amounts of new stock market wealth. Americans disapproved of Clinton's character, but his job approval ratings rarely dipped below a commanding 60 per cent.

And perhaps most critically, Hillary Clinton again chose to stand by her man. Though disgusted with her husband's libido, she fought harder than ever for his political fate. She lobbied Democratic congressmen not to abandon Bill, telling one, 'I'm the field general of this operation.' And she fumed at a media establishment the Clintons felt had always hated them. The real story of impeachment, Hillary claimed, was 'this vast, right-wing conspiracy that has been conspiring against my husband since the day he announced for president'.

Impeachment was a failure. Clinton was acquitted by the Senate. And in the 1998 midterm elections, Republicans lost several seats in Congress. Boing.

Impeachment unexpectedly drove Hillary Clinton's popularity to unprecedented levels, and while many Americans still despised her, she clearly had the support base for an independent political career. In early 1999 she began plotting a run for Senate from New York. Although Hillary had no ties to the state she was popular there, and, according to Her Way, she felt that New Yorkers 'were savvy enough to reject the media-fed stereotype of her and instead accept her on her own terms'. She announced her candidacy a year later, taking centre stage at last - and the first step towards an eventual run for president.

Hillary's 2000 Senate campaign would foreshadow her 2008 presidential bid. After years of suffering Bill's chaos, she managed her own operation in highly methodical fashion. She travelled the state endlessly, hoping to re-introduce herself to the public as a genial and moderate woman, not the imperious leftist she had seemed before. She fanatically controlled the media, keeping reporters at a distance and perfecting the art of saying nothing interesting. Bill aided her by staying out of the way. His final months as president were defined by foreign affairs, including the Kosovo bombing campaign, brinkmanship with Saddam Hussein, and a last-ditch Middle East peace push. The Clintons did leave office in January 2001 on a disgraceful note, after a last-minute rush of presidential pardons that appeared tied to money and favouritism. But by that time Hillary had comfortably defeated her Republican rival and was an elected official in her own right.

In the Senate, Hillary continued her reinvention. Long gone was the 'politics of meaning' and grand ambitions like universal health care. Now Hillary was an incrementalist, a humble workhorse. She attended obscure committee hearings and never grandstanded. Battling her lingering image as a fierce partisan, she worked with conservative Republicans, even those who had tried to impeach Bill.

Everything seemed to be going according to plan. Conventional wisdom settled around the idea - which several years earlier would have seemed ludicrous - that Hillary was a president-in-waiting. But something unpredictable would prove her undoing: the war in Iraq.

As president, Bill Clinton had promoted an assertive foreign policy, in the belief that the world's lone superpower had important interests and obligations abroad. Hillary shared that philosophy, supporting her husband's bombing campaigns in the Balkans and his repeated missile strikes against Saddam Hussein. In a little-noted speech during her Senate campaign she cautioned against thinking 'that we should intervene with force only when we face splendid little wars that we surely can win, preferably by overwhelming force in a relatively short period of time. To those who believe we should become involved only if it is easy to do, I think we have to say that America has never and should not ever shy away from the hard task if it is the right one.'

Iraq in particular had bedevilled Bill Clinton, who was convinced Saddam did possess WMD stockpiles. But many Democratic strategists, including the Clintons' new favourite pollster, Mark Penn, argued that only by demonstrating their 'toughness' on national security could Democrats win elections in the post-9/11 world. Moreover, as a woman hoping to be commander in chief, Hillary was no doubt reluctant to appear 'weak'. And so, in October 2002, she voted to authorise war in Iraq.

When John Kerry succumbed to George Bush in 2004, the 2008 Democratic nomination seemed to be Hillary's for the taking. The Clintons had developed a massive political, intellectual and fundraising network. Hillary was loved by many Democrats; there was sympathy not least for the humiliations Bill had heaped upon her, combined with real respect for her formidable political acumen. At the same time, Bill had disinfected his own reputation, running a foundation which addressed global challenges like Aids and climate change, and achieving a kind of supra-political statesman status. Their personal drama also seemed to be receding, as Bill and Hillary developed independent lives - she spent most of her time in Washington, while he travelled or stayed at home in Chappaqua, in New York state.

Though some questioned her broader electability, Hillary initially seemed a good bet for her party's nomination. Kerry would not run again (to the relief of many Democrats), nor would the newly-rehabilitated Al Gore. Democratic contenders like John Edwards, Joe Biden and Bill Richardson all seemed slight in comparison. As the Iraq war descended into a catastrophe, Hillary was castigated by liberals for her war vote. Yet her subsequent critiques of the war seemed to insulate her. As late as 2006, after all, she faced no significant Democrat who had openly opposed the war from the start.

By that time, an electric buzz had grown around the young Senator from Illinois, Barack Obama. Obama had dazzled the Democratic Party with a stirring speech at the party's summer 2004 national convention. Obama was especially appealing to Democratic activists because he had opposed the Iraq war. Still, the notion that a first-term Senator could simply plunge into a presidential race seemed ludicrous - or so the Clintons felt. In hindsight, it seems clear that Hillary was too complacent about the challenge Obama presented. She dismissed him as a lightweight, and imagined that the hype around his speeches would inevitably fade. What's more, one suspects that Hillary had developed a sense of entitlement. She and Bill had become Democratic royalty, and felt that in some sense the nomination belonged to her. Certainly this newcomer, who had paid no dues, shed no blood and tears, couldn't possibly snatch it away.

But like so many entrenched politicians, Hillary made the mistake of misjudging the public mood. She spent too long harping on her 'experience', underscoring her roots in Washington at a time when furious voters wanted to see the Capitol flushed out. (In one recent national poll, 85 per cent of respondents said they were dissatisfied with the direction of the country.) She also argued repeatedly that she had been 'tested' in the fires of American politics. 'I've been taking the incoming fire from Republicans for about 16 years now. And I'm still here,' she told one interviewer. Clinton aides made it known they could fight as ruthlessly as any Republican, and would never, ever allow Hillary to be caricatured and maligned as John Kerry and Al Gore had been.

But Obama turned these points against her. It was time to 'turn the page', he argued, and move on from the 'divisive' battles of the past. Experience 'wasn't the bar that mattered,' says one senior party strategist. 'What [Obama] offers matters more to voters.' And every point Obama made about change and a new direction was buttressed by the fact that, around the time Hillary was supporting the Iraq war, Obama had spoken out publicly against it.

It didn't help that, after a few years of dormancy, the bad old Bill returned. As Obama ascended, Bill Clinton seemed unable to stand the indignity. (He may also have been madly jealous that Obama was supplanting him as the most beloved Democrat in the land.) Bill began to stray off-message, and to lash out at enemies real and imagined. He churlishly sought to diminish Obama's opposition to the Iraq war as 'a fairly tale', and clumsily downplayed his own supportive words for the invasion. He repeatedly scolded reporters about their coverage in made-for-TV fits of pique. And most devastating, he sealed the stampede of black support to Obama when, before the important South Carolina primary on 26 January, Bill sought to diminish Obama's impending win there by reminding people that Jesse Jackson had carried the state as well. To many Democrats, it appeared perilously close to race-baiting, an effort to marginalise Obama as a 'black candidate' with limited appeal.

Meanwhile, the presidential trail exposed the limitations of Hillary's political skill. She was a mechanical speaker, all the more so compared with her husband. Role reversal had put both Clintons in an uncomfortable spot. 'Bill had great difficulty playing the role of supporter and advocate for her, which she always played for him quite effectively,' says Sally Bedell Smith, author of For Love of Politics, a study of the Clintons. 'And she struggled because she's not the natural politician that he is.'

Many other things went wrong for Hillary. Her strategists failed her by not crafting a plan for an extended nomination fight, which allowed Obama to scoop up critical delegates in states she never really contested. And she refused to apologise for her Iraq vote, defending her rationale when voters probably just would have preferred an admission of error.

As her campaign sank, Hillary found an unexpected new image. Shedding her longtime image of a cultural elitist, she styled herself as a populist. Suddenly Obama was the snooty elitist, and Hillary the tribune of the working-class Americans who had long been suspicious of her. Where she was once the ultimate policy wonk, a paragon of educated expertise, she was now deriding economists who mocked her gimmicky and demagogic proposal for a gas tax holiday as out of touch with working America. And a woman who felt constant resentment at the national obsession with her gender began talking constantly about her gender, claiming that she had been the victim of 'sexism' in the campaign. And although her campaign had long been predicated on a carefully constructed image of strength, when she got in trouble Hillary increasingly played a damsel-in-distress card, as when she complained that the men onstage at debates were ganging up on her. Or, most famously, when she welled up at a public event on the eve of the New Hampshire primary, apparently sending droves of sympathetic women to the polls to rescue her.

It was not enough to save her. Although, at the time of going to press, Hillary has not officially conceded, her chances of overtaking Obama are close to nil. This campaign, then, will be the most resounding defeat the Clintons have faced. Their reputations and influence will have been diminished. But few who follow them closely believe this is the end.

'Politics is the rocket fuel that runs the two of them,' says Bedell Smith. 'And I just can't imagine that they're not already thinking about the next campaign.' That could mean a vice presidential bid on Obama's ticket, though it's difficult to see that as anything but a deeply awkward and fraught marriage for all involved. Or maybe a Hillary bid for governor of New York, or perhaps leader of a Democratic Senate.

Or maybe something more. Those close to the Clintons say they truly believe that Obama - due to his inexperience, his race, his infamous Chicago pastor - cannot defeat John McCain this autumn. And should that happen, the Clintons will waste no time in proclaiming that they told us so. Reeling from a horrid defeat, Democrats may even listen. 'They think he's going to get destroyed,' says a Democratic insider, 'and the party's theirs again.'

No one who has witnessed the fantastical saga of Bill and Hillary Clinton could call that impossible.

· Michael Crowley is a senior editor at The New Republic magazine