The aborted attempt by the Metropolitan police to force the Guardian to disclose confidential sources and other journalistic material raises important issues about press freedom and the way Operation Weeting is being conducted.

But it was the fact that the Met's application was based, in part, on powers in the Official Secrets Act 1989 which was undoubtedly the most troubling aspect. The application was made under the Police and Criminal Evidence Act 1984 (Pace). But the 1989 act is sucked into Pace because a judge granting an order for production of confidential journalistic material must first be satisfied that it would otherwise be possible to obtain a search warrant for that material under some statute other than the 1984 act.

The rationale behind this convoluted mechanism is to give additional protection to journalistic material. The Met chose to argue that they would have been able to obtain a search warrant under the Official Secrets Act 1989 by contending that material it suspected had been passed by a police officer to the Guardian was covered by sections four and five of the 1989 act – which (controversially) prohibits onward disclosure of material whose publication impedes the detection or prosecution of suspected crimes.

It is clear that ultimately the judge's discretion as to whether to grant a production order would have to be exercised compatibly with the right to freedom of expression under article 10 of the Human Rights Act – that (with other factors like the protection of journalistic sources in the Contempt of Court Act) would certainly have provided the Guardian with a robust public interest defence. But overall the operation of the public interest in the context of official secrets remains completely unsatisfactory.

Of course, unless you are an information absolutist, there needs to be deterrence against disclosures which genuinely harm national security. But, as the Guardian application highlights, the official secrets acts are not confined to such territory, are not deployed consistently and do not operate fairly.

The Official Secrets Act 1989 was enacted during the height of cold war paranoia. Spycatcher was making its inglorious way through the courts and Clive Ponting had been spectacularly acquitted of an offence under the Official Secrets Act 1911 for disclosures relating to the Falklands war. The 1989 act was brought in primarily to ensure there was no general public interest defence to disclosures deemed to qualify as official secrets. This draconian intention was confirmed by the House of Lords ruling in the Shayler case, which held that the blanket ban (subject only to the safety valve of prior official authorisation) on disclosures by security services personnel did not violate free speech rights under the European convention.

So if Cathy Massiter, the MI5 officer who was motivated by her conscience to leak details of spying against trade unionists and civil libertarians, had been prosecuted under the 1989 act she would have had no defence at all. The only comfort from the Shayler case was that the earlier linked application to obtain confidential journalistic material from the Observer and the journalist Martin Bright was largely unsuccessful – on self-incrimination and freedom of speech grounds.

So are there really no defences at all to the 1989 act? Well, there are still paths through the thorns. Katharine Gun, a GCHQ employee, was famously acquitted in 2004 of leaking a request by the US to the UK to bug members of the UN in the run-up to the Iraq war. Her defence – a form of necessity – was a desire to prevent an illegal war. The prosecution could not rebut that and swiftly dropped the case at the door of court.

Although there is no general public interest defence, public interest is relevant in assessing whether any prohibited disclosure is damaging (a requirement for most of the act's offences). This is particularly relevant when the government claims that a disclosure is so diplomatically embarrassing that it might seriously undermine international relations – something well short of endangering state secrets or national security. So if, à la David Hare's recent Page Eight TV drama, our prime minister confidentially acquiesced in rendition to torture, disclosure of that might be an offence under the 1989 act but the public interest would be relevant in assessing whether such a disclosure would be damaging.

In 2008, the prosecution of the Foreign Office official Derek Pasquill, who was alleged to have leaked details of UK involvement in rendition and interaction with Muslim extremist groups, was swiftly abandoned – it was not possible to prove that the disclosures were damaging to international relations, which raised the troubling possibility that the act was being used as a governmental embarrassment shield.

In 2007, the court of appeal ruled in the case of Keogh and O'Connor (a Cabinet Office employee and an MP's researcher who were each convicted of disclosing a memo between Tony Blair and George W Bush) that because the act was so broadly drawn, the burden remained on the prosecution to rebut the limited defences available in order to ensure compliance with the Human Rights Act.

Remarkably, prior publication is not a complete defence either. Despite the outcome of the Spycatcher litigation, it is theoretically possible to bring a prosecution for disclosure of something which was already largely in the public domain. When the 1989 act was enacted, the worrying example was given of the prosecution of a government official for confirming a newspaper story.

Some parts of the predecessor official secrets acts (dating from 1911 and 1920) still survive. In 2010 Daniel James, an interpreter for Nato forces in Afghanistan, was convicted under the 1911 act for passing documents to the enemy and for spying on behalf of Iran and sentenced to 10 years. But none of the acts are deployed consistently. Geoffrey Robertson QC's book notes that when the Daily Telegraph published details of the government's plans to kidnap Mahatma Gandhi and its editors revealed their source was the home secretary, unsurprisingly, there was no prosecution.

Similarly a minister's disclosures concerning the Westland affair were not punished nor were Cecil Parkinson's alleged revelations of cabinet proceedings to his lover. And parliamentary privilege protects any disclosures made by MPs, however damaging to national security.

After Katharine Gun's case collapsed, the government promised a full review of the 1989 act. This is now long overdue and the Guardian's recent experiences demonstrate that it is an issue of enormous importance. The act needs completely overhauling and should include a statutory public interest defence. Despite the Shayler ruling, public interest remains relevant to all offences requiring proof of damaging disclosure and public interest may also be material in deciding whether it was unreasonable for official authorisation to have been withheld. But why should public interest be sidelined in this way? Whistleblowers already have a statutory public interest defence in employment law, for example.

So-called "protected disclosures" are those assessed to be in the public interest. There are guiding factors in determining this: was the disclosure made in good faith and not for personal gain? How serious is the wrongdoing being exposed? Was it reasonable to tell the world at large? Specific examples are provided: disclosures of criminal offences, breaches of legal obligations and miscarriages of justice.

A similarly codified public interest defence would be a major improvement to the outmoded 1989 act, which is a relic of the cold war and in its current form ought to be confined to the plotline of a remake of a good John le Carré book.

Alex Bailin QC is a barrister at Matrix Chambers. He writes the official secrets acts chapter in Blackstone's Criminal Practice, regularly advises in cases involving the official secrets acts and defended Katharine Gun