Alistair Darling has acknowledged that it might have been right to try to depose Gordon Brown when he was prime minister, but said he had been held back by "residual loyalty".

Darling, who was Brown's chancellor, described a "chaotic" time in government, with the cabinet riven by bitter disagreements at "the very top" which he said prevented the Labour party from being able to steer the UK through the financial crisis and hampered the party's chances at the 2010 election. There was, he writes in his memoirs, a "permanent air of chaos and crisis" at the heart of government.

In the book, serialised in the Sunday Times, Darling reveals he held a meeting with David Miliband in 2009 shortly after the destabilising cabinet resignation of James Purnell to discuss "getting rid" of Brown. At the time, Darling was blocking Brown's desire to move him from his post and replace him with Ed Balls. In the meeting, Miliband and Darling decided a coup was inconceivable, while the departure of Purnell protected him from sacking.

In an interview on BBC1's Sunday AM programme Darling admitted that "perhaps" senior Labour politicians should have "done something" about Brown's leadership, but said he was held back by a longstanding friendship.

"If you want to criticise us collectively perhaps we should have done something but … I'm afraid, for me, despite everything and if Gordon's listening to this he may find it difficult to believe, but I had a residual loyalty which I found it difficult to overcome. We go back a long, long way. This whole thing was very unpleasant."

Brown appointed Darling chancellor when he became prime minister in 2007. In his memoir, Darling describes Brown's government as a "fairly brutal regime, and many of us fell foul of it", whose behaviour was "sometimes appalling".

Darling gave an interview to the Guardian in 2008 detailing his views on the severity of the economic crisis, and at the time he described the ensuing anonymous briefing against him by Brownites as the "forces of hell". On Sunday he said they had "left a mark" on him and that it was "deeply unpleasant". "What is so debilitating is when your own lot are doing it to you. It's not exactly new in politics but it left a mark on me that you really can't erase."

Darling said that he and Brown were so divided over economic policy that it damaged the party going into the 2010 election. He went on to say that the Labour government of which he was part could have "charted a political way through" the problems that were thrown up by the banking meltdown and subsequent recession. "We could have come through this. We didn't because there was a disagreement at the very top."

They disagreed over the depth of the crisis, with the prime minister insistent that the economy would recover after six months, while Darling says he believed it would take longer. Darling's aides let it be known at the time that this was what he thought. Now Darling has revealed Brown thought his position wrong and based on misleading advice.

"You need to be united at the top but you also need a credible economic policy," Darling said. "If you don't have a credible economic policy you are simply not at the races, and our problem was it was so blindingly obvious to the outside world that the two of us – Gordon and myself – were at odds that it really hampered us when it came to the election in 2010."

In his book, he writes: "No one wanted to acknowledge that we were heading for an extremely serious downturn … I was condemned for having said no more than was true." He claims he was forced to present a budget that "simply lacked credibility", and reveals that the 2009 budget was unwritten 48 hours before its presentation. In particular he singles out Brownite allies Balls and Yvette Cooper for resisting spending cuts, while Brown refused to increase VAT.

The Tories have seized on his remarks, saying they showed that Labour's current economic plan – Ed Miliband's leadership is still signed up to Darling's plan of halving the deficit over the course of this parliament – was not credible.

Darling intends the book, Back from the Brink, to be a document of his role grappling with the financial crisis in 2008. He writes that he personally ordered the rescue plan for the banks in 2008, for which Brown has since taken the credit. At the time, Darling was concerned Brown's allies would derail it.

Darling writes that No 10 nearly jeopardised the whole bailout of the banks by opening their own negotiations: "Some of his advisers had opened up a separate channel to the banks. It was clear some of the senior bankers had a direct line to No 10. Any attempt to talk to Gordon about this parallel operation was met with brusque dismissal: it wasn't happening.

"In the aftermath of this crisis there have been many who have claimed authorship of what proved to be a highly successful plan … it really doesn't matter who thought of the scheme first. What matters is that it worked. What I know is that the Treasury, the Bank and the FSA [Financial Services Authority] started this work … under my instruction."

On the reluctance of the governor of the Bank of England to save the banks for fear of moral hazard, Darling writes: "I was so desperate that I asked the Treasury to advise me as to whether we could order the bank to take action."

He reveals that Brown told him his appointment as chancellor was "only for a year or so". In 2009 Brown told him he would be moved. Darling said he would leave the government if that happened. Purnell's sudden resignation ensured it was impossible to move Darling at all.

Darling also revealed that his caricature as a "bearded Trot" by the former Labour leader Neil Kinnock was a case of mistaken identity – Kinnock had confused him with the bearded Labour councillor John Mulvey. "I've become less leftwing than I was, but I was never a Trot," Darling told the Sunday Times.

Gordon and me – by Alistair



Darling on his relationship with the Brownites

"Damian McBride was no fan of mine – he clearly disapproved of Gordon's decision to appoint me as chancellor. He used to look at me like the butler who resented the fact that his master had married someone he didn't approve of. I'm not sure that he ever spoke to me. He would give me a curt nod, nothing more."

Darling on Brown

"Richard Crossman in his diaries observes that political friendships should be cool and detached. I am afraid he is probably right. By now, ours was lacking in both qualities."

How to give a speech – Brown-style

"I vividly remember a speech Gordon made while we were in opposition, the famous 'neoclassical endogenous growth theory' speech. He started speaking as the second half was still being written. Halfway through, a hand appeared from behind a curtain and handed him the rest of the speech. This exemplifies Gordon's approach to working on such matters."