Gordon Brown's abusive behaviour and volcanic eruptions of foul temper left Downing Street staff so frightened that he received an unprecedented reprimand from the head of the civil service, an explosive new book by the Observer's Andrew Rawnsley reveals today.

Sir Gus O'Donnell, the cabinet secretary, became so alarmed by the prime minister's behaviour that he launched his own investigations when he received reports of Brown's bullying of staff. O'Donnell then gave the prime minister a stern "pep talk" and ordered him to change his behaviour. "This is no way to get things done," he told Brown.

The revelation that the prime minister's behaviour was so extreme that it triggered a warning from Whitehall's most powerful official will shock the political world and is bound to lead to claims from his opponents that he is not fit for another five years in office as a general election draws near.

Rawnsley's book also reveals that after the debacle of the cancelled election in 2007 an increasingly unpopular Brown became more and more paranoid. When briefed that November about the loss of confidential data discs, containing the personal details of more than 20 million people, he leapt across the room and grabbed Gavin Kelly, his deputy chief of staff, by the lapels of his jacket. Brown snarled into Kelly's face: "They're out to get me!"

These incidents, and others, are revealed in the vivid and extraordinary account by Rawnsley of how Brown treated employees at all levels – from top aides to duty clerks and secretaries. He paints a picture of an often lonely and desperate figure who took out his frustrations on those around him as he struggled to cope with the pressures of running the country in his early months in No 10.

According to Rawnsley, O'Donnell was so disturbed by the effect on those in Downing Street that he took it upon himself to try "to calm down frightened duty clerks, badly treated phone operators and other bruised staff by telling them, 'Don't take it personally'".

During one rage, while in his official car, Brown clenched his fist in fury after being told some unwelcome news and then thumped the back of the passenger seat with such force that a protection officer sitting in the front flinched with shock. The aide sitting next to Brown, who had just told him the information that provoked the outburst, cowered because he feared "that the prime minister was about to hit him in the face".

Rawnsley writes that "the cream upholstery of the seat-back in front of Brown was flecked with black marks. When having a meltdown the prime minister would habitually stab it with his black marker pen."

Rawnsley's book, The End of the Party, charts Labour's second and third terms in power and is based on hundreds of interviews with witnesses to the key events in the government's life, including cabinet ministers, No 10 officials and senior civil servants.

Rawnsley reveals that another victim of the prime minister's wrath was Bob Shrum, a respected American political consultant and speechwriter, who had worked for Brown for years. When Brown was accused of plagiarising phrases used by Al Gore and Bill Clinton in his 2007 conference speech, the prime minister screamed at a shaking Shrum: "How could you do this to me, Bob? How could you fucking do this to me?"

A senior adviser on foreign affairs, Stewart Wood, found himself on the end of a similarly shocking volley of verbal abuse when he attempted to brief the prime minister about a Downing Street reception for European ambassadors. "Why have I got to meet these fucking people?" he yelled at Wood. "Why are you making me meet these fucking people?" Brown then roughly shoved aside the stunned adviser.

Rawnsley also recounts how an angrily impatient Brown became so furious with a No 10 typist that he turfed her out of her seat and took over the keyboard.

However, the book does show the softer side of the prime minister, recounting how he is capable of being incredibly solicitous towards colleagues at times of family emergency and bereavement.

The book, serialised in the Observer today and next Sunday, contains the most detailed and authoritative account of Labour's decline into infighting and bitterness under Brown. It also exposes the extent of the tense, mistrustful and often acrimonious relationship between Brown and his chancellor, Alistair Darling, which at times has come close to breaking point.

After Darling had issued a warning, in an interview in 2008, that the economic crisis would be the worst for 60 years, Brown flew into a rage – believing such pessimism would be politically damaging – and told the chancellor in a furious telephone call that the financial turmoil "will be over in six months". In fact, the country was entering a recession that would last 18 months and the chancellor was later entirely vindicated.

Rawnsley tells how by the summer of 2008, when memories of his initial honeymoon as prime minister had long faded, those closest to Brown saw him gripped by dark despair. "That summer Brown descended into a terrible place, politically, psychologically and physically," he writes. A senior politician, who saw the prime minister shortly before parliament broke for the summer holidays, said: "He looked absolutely terrible. The shoulders were hunched. The flesh was literally dripping off his face. I wanted to give him a hug."

His wife Sarah told friends she was very worried about her husband. "I used to believe Gordon when he said he wasn't a workaholic," Sarah Brown told one friend over lunch. "I don't now."

Last night an official spokesman for the prime minister refuted the claims made in Rawnsley's book: "These malicious allegations are totally without foundation."

Earlier Brown had admitted he was "not perfect", but urged voters to take a "second look" at Labour in a speech designed to appeal to former supporters who might have deserted the party. He told a rally at Warwick University, in Coventry, that David Cameron's policies were the "same old Conservative economics of the 1980s", and claimed Labour was the party of change. He also revealed the party would fight the election under the slogan "a future fair for all". In his speech, Brown praised his colleagues in the cabinet, including Darling, Harriet Harman and Peter Mandelson.

A YouGov poll published today by the Sunday Times, meanwhile, reveals that the gap between Labour and the Conservatives has shrunk to just six points – the closest position in more than a year.