In this exclusive extract from The End of the Party, a richly detailed history of Labour in government since 2001, the Observer's chief political commentator, Andrew Rawnsley , discloses how Tony Blair became so demoralised in the spring of 2004 as Iraq descended into chaos that he almost quit Downing Street. Meanwhile, a seething Gordon Brown waited in the wings

By the autumn of 2003, Tony Blair looked more vulnerable than at any previous time in his premiership. Polling suggested that half the public wanted him to resign. Sixty per cent of his own party members said he was wrong to go to war in Iraq and approaching half of them wanted him to quit immediately or at the next election. Tensions with his Chancellor, Gordon Brown, were growing. John Prescott organised a dinner party for the two men to discuss their differences.

That November evening, the three men met at the Deputy Prime Minister's grace-and-favour apartment in Admiralty House. John Prescott typically liked to serve his guests with steak and kidney pudding.

Brown arrived in a foul temper. When they sat down for dinner, the Chancellor complained that his seat wasn't high enough. Prescott went off to find another. "Do you want a different chair as well, Tony?" he asked. "No, it's all right," responded Blair sardonically. "Gordon has always looked down on me."

On Brown's subsequent account to his camp, Blair admitted that he was in a deep hole. "I won't turn it around before the election," he said. If Brown was co-operative and helped to "get me through the next six months", Blair pledged he would hand over the premiership in the summer of the following year. "Naive as always about Tony, Gordon believed him," says one of Brown's closest confidants. He left the dinner more certain than before that he had a promise of a handover.

The morning after the Prescott dinner, Brown called four key aides, Spencer Livermore and Sue Nye, and the two Eds, Balls and Miliband, together for a meeting at the Treasury. "Tony has said he is going to go," he told them excitedly. "We should start preparing."

"Are you sure?" asked Nye. "We've been here before," remarked Balls, unconvinced. Livermore and Miliband also expressed scepticism.

"It's going to happen," Brown assured them. "He said it in terms. Prescott was there. Prescott won't let him break the promise this time."

Blair gave a rather different account of the dinner to his friends, suggesting that he'd implored Brown to be more co-operative by saying: "I'm happy to give you your place in the sun, but you've got to accept that I am Prime Minister." He suggested he'd done a half-deal, making a handover conditional on Brown's good behaviour.

Spring 2004

The wall of the staircase which sweeps up from the ground floor of Number 10 to the first floor is lined with portraits and pictures of all its previous occupants, the still famous and the long-forgotten men and one woman who have ruled Britain from Downing Street. They are in chronological order. At the bottom of the stair is Sir Robert Walpole, the first and longest-serving Prime Minister. At the top, a hanging space waited for Tony Blair. When he was in a mordant mood, he would draw the attention of a visitor to the spot. He would say: "That's where they put you when you're done."

By the spring of 2004, he felt done. The amazing run that began with his election to the leadership in 1994 and swept him through two landslide victories was definitively over. His morale was collapsing, his health was deteriorating, his unpopularity was spiralling and many of the ambitions of a badly wounded leader seemed to have crumbled to dust. He had hit the rock bottom of his premiership.

Consummate actor that he was, Blair was skilful at concealing the severity of the descent from the public and the media. He was also adept at masking it from the great majority of his colleagues and officials. "He managed to disguise it from most people," says his Cabinet Secretary, Andrew Turnbull. "It wasn't visible to me. I only believed in The Wobble when it became clear afterwards that there had been one."

Only those closest to him could see the interior collapse of the Prime Minister. There had been few days since 9/11 when he had not been living on his nerves. He found it difficult to sleep. When it eventually came, rest often did not last long. He would wake with a start in the middle of the night to find sweat trickling down the back of his neck.

His hair was dramatically thinner and what remained of it was much greyer than it had been in May 1997. There was a yellow tone to his skin. "You look young. Why do you look so much younger than me?" he remarked to a junior minister of a similar age. The other man responded: "Because I'm not Prime Minister."

The make-up that was slapped on him for public appearances did not entirely camouflage the stress and exhaustion etched into his face. Those who saw him when he was not wearing pancake were often shocked by how he looked.

Nights were also broken by Leo, now aged nearly four. Leo would be disturbed by the ring of the phone in the flat, or just wake up anyway, and then refuse to go back to sleep. Blair had "a day from hell" when he came back from a European Council late one night to find that Leo was with Cherie in their bed. The Prime Minister ended up trying, and failing, to get his own rest in Leo's little bed in the nursery. During a short break in Bermuda at Easter, another holidaymaker thought Leo was the Prime Minister's grandson. That commentary on how old he was looking made Blair sigh: "I obviously need to get to the gym."

He had tried to deal with the stress by taking up a fitness regime about which he had become quite fanatical. He would work out at Number 10 and use the running machine in the gym in the police guardhouse at Chequers. The result was to make him look thinner and more haggard. He would complain of exhaustion to close friends, groaning: "I'm so tired."

His heart condition was worrying both him and Cherie. In October 2003, he had a scare while spending the weekend at Chequers. His chest was gripped with pain and on Sunday evening he was rushed to Hammersmith hospital in London to be given emergency treatment and placed under supervision for five hours. An irregular heartbeat was diagnosed. He and his aides were frightened that this intimation of his mortality would weaken his political authority. That Sunday night, David Hill, Blair's Director of Communications, arranged to rush the Government's chief medical officer, Sir Liam Donaldson, to an interview for Radio 4's The Westminster Hour in order to deliver reassurances that the condition was neither life-threatening nor incapacitating. That did not entirely succeed in smothering speculation about the Prime Minister's health. There was more reason to be anxious than the media knew. Blair cut down on coffee, but perversely refused to take the pills that were prescribed for his condition. His heart would suddenly and scarily start to race, most alarmingly when he was performing at news conferences and in the Commons. "I had the feeling that he was only operating at 60 to 70% or so of his capacity," thought one of his intimates. He confided to one of his most trusted aides that he even "spaced out" several times in the middle of Prime Minister's Questions.

There was a further toll on his family. Blair was "utterly aware of the fickle nature of people's adulation and people's hatred," says his friend Charlie Falconer. He tried to insulate himself and his children by "preserving an ordinary family life". Leo was too young to be aware of what was happening to his father. Not so Euan, Nicky and Kathryn. The Blairs' children had to make a difficult adjustment. Their dad had been a hugely popular leader in his early years and they had largely enjoyed the celebrity that went with that. Now the children had a father who was widely loathed for the Iraq war, not least by their own age group.

An emotional trauma with one of the children came as a terrible shock to both the Blairs. As one of his aides says: "He's a decent human being. He's a very good dad. It shook him very deeply."

Blair was increasingly doubtful that he could achieve more with the premiership. The Northern Ireland peace process, to which he had devoted commendably vast amounts of time and energy, was at an impasse. Iraq was so dire that Tessa Jowell, one of his closest Cabinet allies, publicly called it "a shroud over the Government". Public service reform was still proving frustratingly intractable. A senior politician who saw a lot of him observed: "He's not very happy. I'm not convinced he gets to the end of many weeks and thinks he has really achieved something."

A threatening band of Labour MPs appeared to be in permanent revolt. "No Prime Minister can survive long-term with a deadweight of 60 or 70 rebels out to get him by any means possible," noted a Cabinet member. 'If 30 more have a genuine concern about an issue, that's a hundred against you from the start."

Then there were the endless guerrilla attacks orchestrated by the impatient Gordon Brown. "It's just constant psychological warfare from Gordon," one of Blair's most senior aides told me at this time. "He will not give up until he has got Tony out." Cherie was livid about the "constant attrition" from Brown "rattling the keys above his head".

On 28 April graphic pictures of American soldiers abusing Iraqi prisoners began to surface in the US media. They were broadcast on network television the next day and then carried around the world. In one of the most shocking photos, a female American soldier, Lynndie England, was shown with a cigarette dangling from her mouth giving a thumbs-up sign while pointing at the genitals of a naked and hooded young Iraqi who has been ordered to masturbate. The abuse was committed at the Abu Ghraib prison, one of Saddam's torture chambers near Baghdad, which added to the ghastly symbolism. The iconic image of Iraq was no longer the toppling of the tyrant's statue. It was a female American soldier holding an Iraqi detainee on a leash.

After many months of ignoring warnings from Amnesty International and others who had gathered allegations about torture and killings, Tony Blair was forced to respond. He called himself "appalled" and declared: "Nobody underestimates how wrong this is or how wrong this will seem to be."

The full extent of the barbarity was not yet publicly revealed in 2004. But the pictures from Abu Ghraib were appalling enough to add to the crisis of Tony Blair's premiership that spring. Already facing relentless accusations that he was mendacious about the WMD, these revelations ate into the moral case for the war. The head of the Foreign Office, Michael Jay, regarded it as hugely damaging.

"You have to conduct foreign policy in accordance with the values you espouse. If you don't do that, you lose an enormous amount of moral authority."

"He got down because of the aftermath of Iraq," says Peter Mandelson. "There was a temporary lapse of morale, spirit, heart. He was prepared at that moment to walk away from it all." Philip Gould, Blair's political consultant, agrees that it was "Iraq – the enormity of it weighed him down". Tessa Jowell, a Cabinet minister very close to Blair, says: "He was very low, he was very lonely and he was very tired." "It wasn't a spasm," believes another ally, Stephen Byers. "He was wobbling for a while."

David Blunkett felt Blair "was really down and needed lifting… when things are going badly you sometimes go into a black hole". Peter Hain agrees that it "was a period of tremendousdarkness for him". Alan Milburn reckons: "He'd lost confidence, the Government had lost direction, he looked very vulnerable."

A senior member of the Cabinet who had known Blair for years says: "This was a very bleak part of Tony's premiership. The war had changed the whole atmosphere of British politics. The north London liberal middle class where he came from was turning viciously against him over Iraq. He was utterly miserable and the neighbour was saying: 'When the fuck are you going?'"

Cherie was consumed by anxiety that they would be out on the street if her husband suddenly quit. With his agreement, she secretly arranged the purchase of a £3.6 million house in Connaught Square. "A mortgage the size of Mount Snowdon" was guaranteed against his future earnings in retirement.

"It was all coming in on him at once," comments David Hill. In the words of Sally Morgan, one of Blair's closest aides: "Iraq was a quicksand swallowing him up. The atrocities. Those terrible photos. And he started losing people who had supported him throughout. He was stuck in this long, dark tunnel and could see no way out of it."

The Cabinet Secretary saw it eating away at the Prime Minister: "The justification for the war didn't stand up. In terms of making Iraq a more decent place to live, was it? No, it was in a worse place."

The Tories were advancing on Labour in the polls and Blair's personal ratings dropped to the lowest of his premiership. Those of Brown sparkled in comparison. The Chancellor consolidated his position with a 2004 Budget that increased spending on health and promised a further £8.5 billion over four years for education. "Gordon was at his peak," comments Philip Gould.

Brown also found more money for pensioners. This was what the average Labour MP had come into politics to do rather than fight a disastrous war in the Middle East alongside a very right-wing President. A typical poll had more than a quarter of Labour supporters saying they might switch their vote because of the war. Polling indicated that Labour would have a much bigger majority at the next election if Blair was replaced with Brown.

Cabinet colleagues had rarely seen Brown so cheerful. "Gordon has got such a spring in his step, he's so whistle-while-you-work," noted one minister. "Something about the succession must have been said." John Prescott knew that something had been said. In the early spring, Blair rang up his deputy and confirmed: "I'm going in June."

In Number 10, Sally Morgan was hatching a quiet conspiracy to stop the Prime Minister from resigning. She would check his appointments diary. If she saw that he was due to meet a friendly face, she would ring the visitor beforehand to encourage him to pump oxygen back into the morale of the Prime Minister. She also invited allies in the Cabinet to drop by to cheer him up, call him at weekends and in the evening and have him to lunches and dinners so that he felt less isolated.

In late April, Tessa Jowell came to Blair's study to offer her shoulder to him. "You're going to get through this," she told him. "I'm fine, darling. Don't worry about me. I'm fine," he responded, unconvincingly. Jowell told him: "Never think you're alone. We're here for you."

Philip Gould observes: "He goes to women a lot at these moments: he finds it easier." It was "women who were best at reassuring and bolstering him", agrees Jonathan Powell, his Chief of Staff, because Blair felt he could be more emotionally open with them.

Other callers who worked to persuade him to stay were Hilary Armstrong, David Blunkett, Stephen Byers, Charles Clarke, Charlie Falconer, Alan Milburn and John Reid, who were all recruited to the campaign to make Blair feel loved. "Our job is to sustain him until the safety of summer," Blunkett told colleagues. Patricia Hewitt, not so personally close to Blair, but no enthusiast for a Brown premiership, wrote a note urging him to stay. Peter Mandelson was an influential voice. "Don't be so daft," Mandelson told Blair when they discussed resignation. "Come on. Buck up. Buck up. Think of what you have to do. Think of what you've got to achieve. You're the best politician in this country by a mile. So just get on top of this."

Staff at Number 10 noted that Mandelson and Morgan suddenly started to involve themselves intensely in a plan to speed up "Iraqisation", the handing over of control to Iraqis. "They were trying to show Tony there was a way out."

Cherie was the most crucial actor in the campaign to stop her husband resigning. She had her moments of doubt about whether they could endure the pressures of power and she worried about his health and the children. But Cherie enjoyed being the chatelaine of Number 10 and didn't want her husband to quit while he was behind. She detested the thought of surrendering the keys to Gordon Brown.

On the evening of Tuesday 11 May, the Blairs had dinner with Michael Levy, old friend, fundraiser and Middle East envoy, and Levy's wife, Gilda. Jonathan Powell rang Levy beforehand with a warning that the Prime Minister was near resignation. "This is very important, Michael," said Powell. "He really needs a lift." The foursome sat down in a small private dining room at Wiltons in Jermyn Street. With paintings of hunting scenes on the walls, the restaurant was a traditional haunt of old-school Tories. Levy came to the dinner with both a warning and an encouragement. The warning was that donors to the party were picking up rumours that Blair might not be around for much longer. That was making it hard for Levy to prise open their cheque books. "You have to make a decision, Tony," Levy told him.

The encouragement came in the form of praise for all Blair had achieved and all he could yet achieve as Prime Minister. Levy, ever the salesman, laid on his best patter to sell Blair to Blair. "Now is not the time to give up," said Levy, flattering Blair with the argument that he was the only person who could win Labour a third term.

It seemed to have the desired effect. The next morning, Cherie waited until her husband was in the shower and out of earshot. Then she rang Levy. He had Cherie's thanks: "Tony came home much happier last night." A little later, she sent the peer some flowers.

Cherie was "by miles" the most significant influence in convincing her husband not to quit, according to Charlie Falconer. David Blunkett agrees she was "really crucial in persuading Tony not to step down". She argued with him that to go now "would be read by history as a tacit admission of failure", as indeed it would have been. "For her, it was beyond the pale to surrender to the next-door neighbour," says Levy.

June 2004

Tony Blair was still out of the country when the country returned its verdict on him. Blair had travelled on to Washington for the funeral of Ronald Reagan as Britons went to the polls for the local and Euro-elections on 10 June. Labour lost more than 450 council seats, slumping into third place behind the Liberal Democrats. The Euro-elections were even worse. Labour's share crashed to a terrible 23%. Blair was not exactly cheerful when he got the news, but neither did he react despondently. "They're bad, but they're not that bad," he told those travelling with him.

Michael Howard's Conservatives fell short of the psychologically crucial 40% threshold in the locals and scored a paltry 27% in the elections for the European Parliament, a dismal result for the principal Opposition party at a time when the Government was so unpopular.

In the wake of the results, Cabinet loyalists flooded the airwaves to play up the disappointment for the Tories and play down Labour's pummelling at the hands of the voters.

"They were crap elections, but there was a brilliant operation afterwards. We had our people on every media outlet," says Sally Morgan. The exercise successfully smothered attempts by some of the Chancellor's supporters to stir up discontent against the Prime Minister.

The agitation was anyway half-hearted because Gordon Brown was working on the assumption that he would soon be moving into Number 10. "Gordon trusted him to hand over," says one of Brown's closest allies. "Against all previous experience, he still trusted him."

To help save the elections from being a total catastrophe for Labour, Brown and his team had put a lot of effort into the campaign. They were disconsolate and mutually recriminatory when the outcome weakened Howard and therefore strengthened Blair. To Brown's face, Ed Balls said: "You've been a mug."

Blair was now almost completely resolved not to leave Number 10, his morale further buoyed by advice from Philip Gould that Labour could win another three-figure majority at the next general election. What Blair had yet to summon up was the courage to tell Brown that his promise of a handover was not worth the paper it was not written on.

"All conversation stopped," says an aide at the centre of Brown's circle. "It all went suspiciously silent. Tony couldn't bring himself to tell Gordon directly. He couldn't explain what he was doing."

Brown came round to Number 10 to try to get an answer. Sally Morgan says: "Gordon was just losing it. He was behaving like a belligerent teenager. Just standing in the office shouting: 'When are you going to fucking go?"'

Members of the Chancellor's entourage tried to take things into their own hands. Ed Miliband was always regarded as the least thuggish of the Chancellor's crew, but the iron had now entered his soul. He stormed in to see Sally Morgan. "Why are you still sitting here? Why haven't you packed up to go?" demanded Miliband. "There's a deal and he's got to go. There's a deal. Prescott was the witness to it." Morgan claimed never to have heard of any such deal: "I don't accept what you're saying is true." She went into the den to tell Blair: "You're not going to believe this. I've had Ed Miliband round telling me to pack up." Blair contacted Prescott, who "went mad" because he didn't want to be dragged into it. Miliband phoned Morgan soon afterwards. "How dare you tell people?" he shouted down the phone. "That was supposed to be a private conversation."

According to David Hill : "It happened quite regularly. You'd have numbers of Brown people coming round to Number 10 saying: 'You shouldn't be here any longer'.''

Brown's camp were becoming demented in anticipation of what they saw as an incipient betrayal. No one was more maddened than the Chancellor, who had been readier to believe the promises of a handover than the more sceptical Ed Balls and the rest of his entourage. Blair could not bring himself to tell Brown directly. So the media had the conversation for them.

18 July was the beginning of the tenth anniversary week of Blair's leadership of the Labour party. On that day, the Observer splashed: "Blair: no deal with Brown on No 10". My story and the commentary inside were based on extensive conversations at the highest levels within Number 10, where I had been given the emphatic impression that Blair had totally recovered from the psychological pit of the spring and was now fixed on fighting another election and serving a full third term.

It was widely conjectured among lobby correspondents that the principal source for this exclusive was Tony Blair himself. The Treasury took that as read. Brown vented his fury with his confidants. "Newspapers were hurled around the office and trampled on," says one senior Brown aide. A boiling Brown then demanded an explanation from Blair. "I was asked a question," replied Blair, mock innocently. "I answered it." Brown shouted back: "Are you fucking going or not?" He did not get a straight answer from the other man.

John Prescott got the two of them together for one of his marriage counselling dinners at Admiralty House. "Give me a date," demanded Brown. Blair finally admitted to his change of mind. He couldn't go now, he contended, because it would look as if he had been defeated by Iraq. "I need more time," he told Brown. "I can't be bounced." The dinner ended badly.

In August 2006, Tony Blair leaves for his summer holiday amid growing demands within the Labour party for him to announce a date for his departure.

Before Tony Blair's departure for the Caribbean, his closest allies had counselled him that he would have to come back from holiday with a strategy to manage the rising clamour from Labour MPs for clarity about how long he intended to go on as Prime Minister. Blair remained hugely reluctant to say any more about this in public on the grounds that talking about it would just lead to "another Hiroshima of speculation".

His silence only made his critics more voluble. There was a major debate with his aides at Chequers in April and another in July. Jonathan Powell was the leader of the diehards. The Chief of Staff believed that Blair should still be planning to remain at Number 10 at least until 2008. Phil Collins, Blair's chief speech writer, took his side. So did David Hill, mainly on the grounds that the media would try to drag him forward from whatever deadline he set. That chimed with Blair's own feelings. "Whatever date I give, my enemies will come back and demand a date six months earlier," he told his aides. Collins observes: "He was always reluctant to give the date. That was the only thing he had left."

Matthew Taylor, Blair's senior policy adviser, was the most persistent advocate within Number 10 for a precise timetable. He argued that Blair would squash the endless speculation, silence his enemies and snuff out Brownite plots if he was publicly clear that he would leave in the summer of 2007.

Ruth Turner, the Director of Government Relations, tended to agree because she was "getting it in the neck" from Labour MPs and the Cabinet all the time – "he's got to tell us, he's got to give a date". There was a strong dimension of Blair, the side of him that was anxious to avoid the fate of Margaret Thatcher, that wanted to leave Number 10 with dignity. That was in contention with the other side of Blair, who had meant it when he said he wanted to serve a full third term and hated the idea of handing the crown to Brown under duress.

He would talk "in almost mystical terms" about how "he had made a contract with the British people to serve a full term". Whatever view they took, there was near universal agreement among his senior staff that Blair had to be more precise about his intentions. Even the ultras like Ben Wegg-Prosser could now sense that "it was going to be difficult to get beyond 2007". The current vagueness offered no fixed point for his allies to rally around while providing ammunition for the Brownites and other Labour MPs who were saying that the uncertainty damaged the Government.

As the pressure mounted, some of Blair's closest confidants were increasingly worried that it would all end badly for their friend in Number 10. One of them mournfully remarked to me that summer: "Prime Ministers never get their departures right, do they?"

What no one else knew was that John Prescott had presented Blair with a stark ultimatum. The two of them met alone in July shortly before Blair went abroad. Prescott was a very weakened figure after the humiliating exposure of his adultery with a junior civil servant. But he still retained one weapon, the threat of revelation, that he could use on Blair. The deputy had long been telling Blair he had to announce his departure date. Prescott also believed that it was only fair to give Brown two years as Prime Minister to establish himself before a general election.

When they met that July, Prescott told Blair that he must make a public declaration that he would leave by the summer of 2007. He said Blair had to make that announcement in his speech on the Tuesday of that autumn's party conference. If he didn't deliver, Prescott threatened, he would announce his own resignation in his speech on the Thursday and reveal that he was quitting because Blair had broken the promises to Brown witnessed by Prescott. "I will make it clear that you are to blame," Prescott menaced the Prime Minister.

Blair protested that this was unnecessary. He had already given assurances to both Brown and Prescott that he was secretly planning to leave in 2007 anyway. But this time his deputy was not willing to be smoothed into submission. Prescott responded that private promises like that weren't good enough any more. "Gordon doesn't believe you. And I don't fucking believe you."

© Andrew Rawnsley