The Metropolitan police has dropped its attempt to order the Guardian to reveal confidential sources for stories relating to the phone-hacking scandal.

The Met had been hoping to force Guardian reporters to reveal confidential sources for articles disclosing that the murdered teenager Milly Dowler's phone was hacked on behalf of the News of the World. But after an intervention by the Crown Prosecution Service and widespread outrage, Scotland Yard was forced into an abrupt climbdown.

The Met claimed that one of the paper's reporters, Amelia Hill, could have incited a source to break the Official Secrets Act and broken the act herself.

At an Old Bailey hearing scheduled for this Friday, the Met had been due to apply for a production order to obtain all the material that the Guardian holds that would disclose sources for the newspaper's coverage of the phone-hacking inquiry this year.

The statement put out by the Met announcing its retreat left open the possibility that the production order could be applied for again, but the Guardian's lawyers have been told that the police have dropped the application. A senior Yard source said: "It's off the agenda."

The police application was formally being made under the Police and Criminal Evidence Act, but with an assertion that Hill had committed an offence under the Official Secrets Act by inciting an officer from Operation Weeting – the Met's investigation into phone hacking – to reveal information.

The Yard source said: "There will be some hard reflection. This was a decision made in good faith, but with no appreciation for the wider consequences. Obviously, the last thing we want to do is to get into a big fight with the media. We do not want to interfere with journalists. In hindsight the view is that certain things that should have been done were not done, and that is regrettable."

The Guardian's editor-in-chief, Alan Rusbridger, said: "We greatly welcome the Met's decision to withdraw this ill-judged order. Threatening reporters with the Official Secrets Act was a sinister new device to get round the protection of journalists' confidential sources. We would have fought this assault on public interest journalism all the way. We're happy that good sense has prevailed."

Many lawyers had expressed astonishment at the police resorting to the Official Secret Act. Their surprise was reinforced on Monday when the director of public prosecutions, Keir Starmer QC, revealed that the Crown Prosecution Service had not been contacted by officers before the application was made.

Neil O'May, the Guardian's solicitor, said: "This was always a misconceived application for source material. Journalists' sources are protected in law. For the Metropolitan police to turn on the very newspaper which exposed the failings of the previous police inquiries and reported on hacking by the News of the World was always doomed to failure. The Metropolitan police need to control the officers who are involved in these sensitive areas."

In a statement , the CPS said: "[On] Monday the Metropolitan police asked the CPS for advice in relation to seeking a production order against Guardian Newspapers.

"The CPS has asked that more information be provided to its lawyers and has said that more time will be needed fully to consider the matter. As a result, the scheduled court hearing will not go ahead on Friday. [The Metropolitan Police] will consider what application, if any, it will make in due course, once it has received advice from the CPS."

The Met said in a statement: "The Metropolitan police's directorate of professional standards consulted the Crown Prosecution Service about the alleged leaking of information by a police officer from Operation Weeting.

"The CPS has today asked that more information be provided to its lawyers and for appropriate time to consider the matter. In addition the MPS has taken further legal advice this afternoon and as a result has decided not to pursue, at this time, the application for production orders scheduled for hearing on Friday 23 September. We have agreed with the CPS that we will work jointly with them in considering the next steps.

"This decision does not mean that the investigation has been concluded. This investigation, led by the DPS, not Operation Weeting, has always been about establishing whether a police officer has leaked information, and gathering any evidence that proves or disproves that. Despite recent media reports, there was no intention to target journalists or disregard journalists' obligations to protect their sources.

"It is not acceptable for police officers to leak information about any investigation, let alone one as sensitive and high profile as Operation Weeting.

"Notwithstanding the decision made this afternoon it should be noted that the application for production orders was made under the Police and Criminal Evidence Act (PACE), NOT the Official Secrets Act (OSA).

"The Official Secrets Act was only mentioned in the application in relation to possible offences in connection with the officer from Operation Weeting, who was arrested on August 18 2011 on suspicion of misconduct in a public office relating to unauthorised disclosure of information. He remains on bail and is suspended.

"Separately, the MPS remains committed to the phone hacking investigation under Operation Weeting."The picture painted by Met insiders is that a relatively junior officer took the decision to take on the Guardian without consulting his superiors, setting off a calamitous chain of events that saw the Met condemned for an attempted assault on press freedom.

Police sources said the senior investigating officer who was inquiring into whether a member of the Weeting team had leaked information had taken the decision to seek the production order on his own.

The senior source said that not even Deputy Assistant Commissioner Mark Simmons had been told about the decision in advance. Simmons is the head of professionalism issues at Scotland Yard and is seen as a rising star within the force.

The senior source said: "There was not a lot of happy people at our place over the weekend because it was a decision made by the SIO. There was no referral upwards, and you would have thought on something as sensitive as this there would have been." Simmons and the incoming commissioner, Bernard Hogan-Howe, did discuss the issue, as the criticism grew, but the source said the commissioner had left it to Simmons to take the decision, and that there was no instruction or directive. The Met stressed that Hogan-Howe, despite being in charge of professional standards as deputy commissioner, was not involved in the original decision to seek a production order. Simmons took the decision to review the application by the SIO.

Geoffrey Robertson QC said: "This is a victory for common sense and freedom of speech. Had the police continued with this ill-considered action, journalists might have been forced to disobey a court order so as to protect their source.

"Putting journalists into that dilemma and possibly in jail would only bring discredit on police and the law. It should now be accepted that journalists are entitled to protect their sources of information, otherwise that information will dry up and there will be less public interest information, such as the hacking of Milly Dowler's phone." The Met's move had been condemned by all Britain's major newspapers, including the Times and Sunday Times, and the Daily Mail's columnist Richard Littlejohn.Isabella Sankey, director of policy for Liberty, said: "

"It would have been perverse in the extreme for early prosecutions in the phone-hacking scandal to be against those who blew it open rather than those who covered it up.

"We hope that editors and journalists never forget how important human rights are to a free press."

Michelle Stanistreet, general secretary of the National Union of Journalists, said: "We are delighted that common sense has prevailed and the Met has woken up to the fact that they cannot get away with such flagrant abuse of the Official Secrets Act.This was an outrageous attack on a central tenet of journalism – the protection of our sources. This is a victory for journalism, democracy and press freedom."

The Yard pursued its action against the Guardian without consulting the CPS, until Monday, or the attorney-general Dominic Grieve.