The relationship between Tony Blair and Gordon Brown became so hostile that Blair described his chancellor as "mad, bad, dangerous and beyond redemption" and likened Brown's behaviour to that of a "mafioso" in his dealings with him, Lord Mandelson has revealed.

The breakdown in the two men's relationship is detailed in the third instalment of Mandelson's memoirs, The Third Man, in which he describes how he and other No 10 aides drew up plans – codenamed Operation Teddy Bear – to split the Treasury into two in an attempt to weaken Brown's challenge to Blair after the prime minister agreed in 2003 not to fight the next general election, only to renege on his promise.

But plans to remove Brown from the Treasury and put him in the Foreign Office were twice considered and rejected by Blair because he feared Brown would resign and become an even greater threat.

Blair's broken promise led to an almost complete breakdown in relations between the two men, and while Mandelson's memoirs lack genuine revelation in many areas they confirm some intriguing details.

Mandelson said at a meeting with John Prescott, then deputy prime minister, Blair agreed to the deal in which he would step aside in favour of Brown. Blair is quoted as saying that even Prescott was "scared" by Brown. "He knows there's something wrong with him," Blair is reported as saying. Mandelson said Blair described Brown as "flawed, lacking perspective and having a paranoia about him".

It becomes clear from Mandelson's account that Blair never believed the deal was binding because it was dependent on the former chancellor helping him achieve his domestic agenda, which he believed Brown could not and would not do.

In early 2005 when Blair finally went back on his promise not to lead Labour into the general election, Mandelson said Brown came to him to demand a final exit date. Blair is quoted as saying: "He's like something out of the mafiosi. He's aggressive, brutal ... there is no one to match Gordon for someone who articulates high principles while practising the lowest skulduggery."

Mandelson says Blair "variously believed, and told me, that Gordon was mad, bad, dangerous and beyond hope of redemption". The bitterness continued after Labour's 2005 election victory, with Mandelson relating how Blair described one meeting he had with Brown to discuss pension reform as "the ugliest I've ever had".

In the extracts, serialised in the Times, Mandelson says that John Birt, the former BBC chief who became a No 10 adviser, and Jonathan Powell, Blair's chief of staff were responsible for drawing up Operation Teddy Bear. Under the plan, a US-style Office of Budget and Delivery would be carved out of the Treasury, leaving Brown in charge of a finance ministry to handle macroeconomics. Crucially Brown would have lost control of departmental purse strings.

But when Blair put the plan to Brown, the then chancellor said "no". Mandelson says: "Tony decided his position was just too weak for him to impose it. It was a fateful moment."

Mandelson also sets out how Brown had targeted him. Before the 2001 election, after Mandelson was cleared of any wrongdoing in the Hinduja passport affair, Blair told him: "Gordon wants you buried." He added of Brown: "I have no illusions any more. He could come for me – and he would do probably if he got the chance."

Blair added later: "He's thinking of only one thing. Only of removing me, but I am not going to be pushed out."

That Blair reneged on a pact is well known but Mandelson's recollection adds a fresh layer of understanding about the relationship between Brown and Blair.

Mandelson reports on a dinner hosted by Prescott, who invited the two men to a reconciliation at Admiralty House.

Brown's bottom line that night, Mandelson recounts, was that Blair must leave before the next election. Reports emerged that a deal had been done, but Blair denied to Mandelson that he had promised to go before the election.

In today's extract, Mandelson writes: "Much later he admitted to me that a deal had been done, and that he had in fact agreed to Gordon's demand that he do just that. So there was a deal between Tony and Gordon but it was in 2003, not in 1994."

Mandelson was referring to the "Granita pact" in 1994. Brownites claim Blair agreed over a dinner at the former Islington restaurant that he would go for the leadership but would hand over to Brown – a pact Blairites deny.

The Admiralty House deal required Brown to help Blair to deliver his domestic policy agenda. When, the next day, Prescott is supposed to have asked for Mandelson's help in getting Blair to implement the deal and when the peer asked Blair was what going on he was told to calm down. "Just play along," he said. "Just keep talking to them."

When pressed he said: "Well, I don't think he'll help me. So the situation won't arise. It won't happen. But I've got to do this — so play along."

In the spring of 2004, Blair told Mandelson he was not keeping to the deal with Brown "but it was the only way of managing Gordon, he said, of getting through until he [Mr Blair] was stronger".

Shortly afterwards, when it appeared that Brown would not give Blair explicit support in the tuition fees revolt, Lord Mandelson said to him: "Don't kid yourself. Tony will not go if he loses. You're not going to get his job by watching him being carried out on a stretcher."