The government's proposed press regulatory system backed by a royal charter "would be totally unconstitutional in the US and other countries", the head of the Society of Editors has warned.

Bob Satchwell, the executive director of the society, which draws its members from across the press and TV industry, said the charter had "key problems" that put press freedom at risk.

"You can't have a new system of regulation which is drawn up by and imposed by politicians," he told the BBC on Saturday.

"The things which are being proposed at the moment would be totally unconstitutional in the US and other countries.

"People in other countries, not just journalists, are looking at what's going on here at the moment with horror."

Jonathan Freedland, a columnist with the Guardian, agreed that the proposed system could be used to clamp down on press freedom.

He warned that the proposed regulatory system could result in the state security services being able to stop the publication of stories such as the leaked details of the operations of GCHQ and the National Security Agency (NSA).

Speaking on the Today programme on BBC Radio 4, Freedland highlighted a speech by the head of MI5 this week which criticised the Guardian for publishing the leaks.

"What worries me is the idea that under a system where there is a royal charter overseen by the privy council – which we all know is the body used for the state's most secret business – there would be a mechanism for the head of MI5 not just to complain in a speech but actually to prevent that kind of publication," he said.

The government has given the newspaper industry a warning that it risks being subject to full statutory regulation if it refuses to accept the royal charter.

The Labour deputy leader, Harriet Harman, who represented Labour at the cross-party talks, said the press should accept the verdict of the politicians and not try to challenge it through judicial review.

"I think it would be very regrettable because I think that it would be much better for them just to join in to the framework that has already been set forth. It's no danger to them at all," she said.

"For them to boycott it and say 'We are going to set up our own system' … I think leaves us exactly where we were before, which is the press are marking their own homework."

The representatives of the three main political parties agreed a series of changes to the proposed charter intended to make it more palatable to the industry and encourage newspapers to sign up.

But a newspaper industry steering group has made clear that the amendments failed to address the press's fundamental objections, saying the charter was neither voluntary nor independent.

"This remains a charter written by politicians, imposed by politicians and controlled by politicians. It has not been approved by any of the newspapers or magazines it seeks to regulate," the group said.

Paul Dacre, the editor of the Daily Mail, said that recent controversies surrounding an article about the father of the Labour leader, Ed Miliband, and the Guardian's publication of revelations by Edward Snowden illustrated that politicians should not "sit in judgment on the press".

"Some have argued that last week's brouhaha [surrounding Ralph Miliband] shows the need for statutory press regulation. I would argue the opposite. The febrile heat, hatred, irrationality and prejudice provoked by last week's row reveals why politicians must not be allowed anywhere near press regulation," he wrote in an article published in both the Daily Mail and the Guardian.

Maria Miller, the culture secretary, is understood to have told the industry that she cannot stop Labour and the Liberal Democrats joining forces to agree amendments to future legislation to place regulation on a statutory footing if the press refuse to abide by the new system.

There have been reports that there could be a legal challenge to the decision this week by the privy council to throw out an alternative charter put forward by the industry.

The latest text – amending the charter agreed last March in late-night talks with the parties and the Hacked Off campaign group – will go forward for final approval by the privy council on 30 October.

The newspaper steering group statement increases the prospect that much of the press will simply go ahead with their own system of self-regulation, despite the threat of exemplary damages in any legal action that newspapers are involved in if they remain outside the official system.

Miller has told the industry that she is open to suggestions over the next 10 days but made it clear that she has little room for manoeuvre because Labour believes it has made concessions.