Britain's top police officer, Sir Paul Stephenson, announced his shock resignation as he was brought down by his failure to tell senior figures, including the prime minister, that Scotland Yard had hired a former News of the World executive as an adviser while refusing to reopen inquiries into phone hacking.

Stephenson had been angered over the questioning of his integrity and said he had to go in order to stop the controversy distracting the Metropolitan police from the challenges it faced, including the 2012 London Olympics.

The crisis over hacking engulfing News Corporation began to turn toxic for Stephenson on Thursday after the arrest of Neil Wallis, who was the News of the World's deputy editor under former No 10 aide Andy Coulson and during the period when it is alleged phone hacking was widespread at the paper. Hours after Wallis was arrested, it emerged that he had worked for the Met.

In a stunning statement , Stephenson stressed his integrity and dismissed weekend claims that it was compromised by accepting a free stay at a luxury health spa where Wallis had been hired as a PR consultant.

Stephenson said: "I have taken this decision as a consequence of the ongoing speculation and accusations relating to the Met's links with News International at a senior level and in particular in relation to Mr Neil Wallis who as you know was arrested in connection with Operation Weeting last week ...

"I have heard suggestions that we must have suspected the alleged involvement of Mr Wallis in phone hacking. Let me say unequivocally that I did not and had no reason to have done so. I do not occupy a position in the world of journalism; I had no knowledge of the extent of this disgraceful practice and the repugnant nature of the selection of victims that is now emerging; nor of its apparent reach into senior levels."

In potentially toxic remarks for David Cameron, Stephenson said he had not mentioned Wallis's appointment to protect Cameron: "I did not want to compromise the prime minister in any way by revealing or discussing a potential suspect who clearly had a close relationship with Mr Coulson.

"I am aware of the many political exchanges in relation to Mr Coulson's previous employment – I believe it would have been extraordinarily clumsy of me to have exposed the prime minister, or by association the home secretary, to any accusation, however unfair, as a consequence of them being in possession of operational information in this regard."

The shadow home secretary, Yvette Cooper said: "It is a very serious concern that the Met commissioner felt unable to tell the prime minister and the home secretary about this operational issue with Neil Wallis because of the prime minister's relationship with Andy Coulson. It appears that their compromised relationship with Andy Coulson has put the commissioner in a very difficult position and made it even harder for the Met to maintain confidence around this difficult issue. Both David Cameron and Theresa May must take their share of the responsibility for this situation."

The Guardian has learned that Scotland Yard chiefs invited Wallis to apply for a senior communications post with the force in 2009, a decision Stephenson was aware of. Wallis was approached to apply for the two day a month contract by the Met, following discussions involving the forces's most senior figures.

A source with close knowledge of the Yard's thinking at the time said part of Wallis's attraction was his connection to former News of the World editor Coulson, who was a leading aide to Cameron, then in opposition and expected to become prime minister.

Part of the Met's thinking was that Wallis's connections would help the force's relationship with Cameron: "One [Wallis] is a lot cheaper and gives you direct access into No 10," the source added.

Stephenson was facing the prospect of a difficult Commons statement by Theresa May, the home secretary, and anxiety expressed by the deputy prime minister, Nick Clegg, about confidence in the Met because of the failure to tackle the scandal.

Stephenson also faced a rough ride tomorrow from MPs on the home affairs committee angry that in his evidence just weeks ago he had failed to tell them about the hiring of Wallis. On top of that he faced anger within his watchdog, the Metropolitan Police Authority, about his failure to inform them about Wallis's employment when he spoke to them last Thursday. On Thursday, Boris Johnson, had carpeted Stephenson over his actions.

The Met denied that Stephenson's five-week free stay at a luxury spa was in any way connected to the fact Wallis was a PR consultant to the company. The commissioner was recovering from a fractured leg caused by an operation to remove a pre-cancerous tumour, and only recently had returned to work.

One of his attractions to the people who appointed him, Johnson and the then Labour home secretary, Jacqui Smith, was his reputation as a safe pair of hands which was earned while he served as deputy to Sir Ian Blair during Blair's tumultuous commissionership. But his handling of the Wallis affair had tarnished that, as did admissions he had dined with News of the World executives including Wallis. On one occasion they met as the Met carried out its first phone hacking inquiry in 2006.

A Home Office spokesman said May told Stephenson she was "sorry" that he was resigning when he called to inform her of his decision. "The home secretary took a call from Sir Paul Stephenson tonight in which he informed her of his decision to resign as commissioner of the Metropolitan police service," the spokesman said.

Stephenson will stay in post until a replacement is chosen.

Johnson said: "Sir Paul believes, however, that the phone hacking saga now threatens to become a serious distraction during the run-up to the Olympic Games. He has persuaded me that someone else should now be allowed to take his work forward so that the focus can return to policing and bringing down crime."

The early favourite to replace Stephenson is Sara Thornton, chief constable of Thames Valley. Stephenson's deputy, Tim Godwin, set up the latest phone-hacking inquiry while the commissioner was off work through illness and has let it be known he has not accepted News International hospitality.

Lord Prescott, the former deputy prime minister who had called for Stephenson to resign, wrote on Twitter: "I always thought the Met and News International were too close and now we see how close they were. Another green bottle has fallen – more to come."

Peter Smyth, chair of the Metropolitan Police Federation, said: "I think it is a sad day for Paul and a sad day for the Met. He is a very private man, I have never had any reason to question his integrity." He has come to a decision based on what he knows about himself."