It's the readers I feel sorry for. How, one wonders, are those who follow Britain's noisiest newspapers of the right to make sense of what they have been told? For nearly a year the Telegraph, Times, Sun and Daily Mail have warned that the hard-fought freedom of the press is in danger, that soon there could exist in this land a menace that has not existed in three centuries: state control of the written word, thanks to last year's Leveson report and the new regime of interference it mandated.

In this grim new world a newspaper editor could face the threat of jail simply for doing what journalists are meant to do, probing into those corners of public life the powerful would prefer stayed hidden. The readers' only comfort has been the knowledge that at least the right-leaning press is ready to stand firm in the defence of free expression.

What a shock, then, to open those papers this week. "Guardian treason helping terrorists," thundered Rod Liddle in the Sun. "Guardian has handed a gift to terrorists," announced the front page of Wednesday's Daily Mail, quoting the head of MI5, Andrew Parker, who had condemned this newspaper for revealing that pretty well anyone who uses the internet is monitored by a mass surveillance programme conducted by the NSA and GCHQ. Helpfully, the Mail found a professor at Buckingham University to call for the Guardian to be prosecuted under the Official Secrets Act.

Its editorial was clear. The Guardian had acted with "lethal irresponsibility". If the head of MI5 says something should not be published, then it should not be published. When it comes to reporting on such matters, an editor cannot possibly be allowed to decide for himself what to print. After all, as the Mail put it, "He's a journalist, not an expert on security." Put another way, in an ideal world a newspaper editor could face the threat of jail simply for doing what journalists are meant to do, probing into those corners of public life the powerful would prefer stayed hidden. Readers will surely be forgiven their confusion. One minute the papers are fighting to their last breath the threat of state control, the next they are cheering the secret state as it seeks to gag a newspaper. Those who study the media closely believe the motive is obvious. The Mail and others loathe the Guardian, they say, blaming it both for the entire Leveson process through its revelation of the phone-hacking scandal, and for subsequently insisting that any regulation be genuinely independent. The current attack is payback. In other words, so desperate are the rightwing papers to avoid state interference in the press, they'll demand more state interference in the press. They are bombing the village to save it.

In this thicket of confusion, there are two questions that should be answered. Was the Guardian right to publish the NSA revelations? The head of MI5 says no, but the editors of the most prestigious news organisations in the world disagree, describing such reporting – uncovering a pattern of global surveillance, rather than unmasking individual agents – as nothing less than the duty of journalism in a democracy, allowing voters to know what the state is doing with their money and in their name.

Perhaps that's what journalists would say (though not all journalists, we now discover). Perhaps it's equally predictable that Lib Dem Vince Cable would say, as he did on Friday that the Guardian had performed a "considerable public service". But more unexpected are the words of James Clapper, the director of US national intelligence, who has said of the NSA revelations: "I think it's clear that … some of the debate, actually needed to happen." That's rather hard to square with MI5's claim that the Guardian is guilty of dangerous treachery.

The second question relates to the link between the NSA story and press regulation. For if the threat of state control of the press is real, what kind of story do we think the state would want to control? Would it punish a red-top editor for rifling through, say, Cheryl Cole's dustbins; or would it hound Alan Rusbridger, the Guardian editor, for revealing that all of us are watched around the clock?

This should give everyone pause, especially those who, after seeing the slime Leveson found under various stones, became the loudest enthusiasts for regulation. There has been much focus on ensuring any new regulator is truly independent of the newspapers, on a genuine break from the Press Complaints Commission that was the wholly owned creature of Fleet Street. The Guardian has been adamant on this point.

But a new regulator must be just as independent of the state and, on this point, all the papers, including the most hawkish, may have made a fateful error. In their determination to keep politicians' hands off the press, they insisted MPs stay well away, passing no statute that would establish the new regulation system. In its place came a wonderfully Ruritanian ruse, the use of a royal charter. Politicians and press alike have embraced this medieval device, believing that a body magicked from the air by the Queen neatly dodges the threat of state control.

But they're wrong – and this week has proved why. For the body that oversees a royal charter, and can unpick its terms on a whim, is the Privy Council – an entity packed by ministers drawn from the government of the day, and which is deployed to do the state's most secret business, under the extensive, unchecked powers of the royal prerogative. It is the very last bit of government any believer in free speech would want anywhere near the press. Yet the newspapers' own proposal, rejected by ministers on Tuesday, called for just such a royal charter.

After this week, we don't have to imagine how such a system would work. The head of MI5 would no longer be confined to speechifying against the Guardian. It would need only a word in the right ear and, with the privy council and the charter as its weapons, the state could decide the Guardian had crossed the line and had to be silenced, leaving the public where it was before: in the dark.

There is not much time. Late on Friday the three main Westminster parties announced they had agreed a new regulatory set-up, centred once again on a royal charter, albeit one that cannot be altered by secret ministerial whim, but would require two-thirds majorities in both houses of parliament. That provides little reassurance: the requirement itself could be overturned by a simple Commons majority.

Ministers hope to have their new charter "sealed" by 30 October. Between now and then editors need to agree on an alternative. They might look for a new overseer, perhaps located in the judiciary rather than parliament. Or they could construct a new regulator whose members are truly independent but which is overseen by no part of the state, even if that means giving up the legal protections and reduced court costs Leveson envisaged. Such a move could be combined with a tougher law against the kind of violations of privacy that sparked the current fury, as well as reformed libel rules and new limits on media ownership, to ensure greater plurality.

Whatever the solution, it must not involve a royal charter and the privy council. Otherwise it will hand a gag to the most secretive elements of the British state. And, as we saw this week, they are itching to use it.

Twitter: @Freedland