A shellshocked newspaper industry was struggling to come to terms with a sudden all-party agreement to create a powerful new press regulator designed to prevent a repeat of the phone-hacking scandal.

The independent regulator will have powers to impose fines and demand prominent corrections, and courts will be allowed to impose exemplary damages on newspapers that fail to join the body.

All three party leaders hailed the "historic" deal, sealed in extraordinary late-night talks on Sunday in the office of the Labour leader Ed Miliband after months of wrangling, but many of the country's leading newspaper publishers were ominously wary.

Some Conservative MPs accused David Cameron of running up the white flag and the former Tory cabinet minister Peter Lilley urged newspapers to boycott the new system – an option being actively considered by some media groups.

The newspapers are furious that Cameron's policy adviser, the Cabinet Office minister Oliver Letwin, sealed the deal at 2.30am on Monday morning in Miliband's office, accompanied by Liberal Democrat leader Nick Clegg and four members of the victims' group Hacked Off.

No 10 was forced to say that Cameron had not been asleep in the early hours and that the critical aspects of the deal had been settled the previous afternoon in face-to-face talks between Clegg and Cameron.

Under the deal, the newspaper industry has lost its power to veto appointments to the body that will replace the Press Complaints Commission, the previous regulator discredited by its failure to investigate phone hacking by leading newspapers.

David Cameron urged the newspaper industry to sign up quickly to the agreement by setting up the new regulator. "It is a neat solution. It is not a panacea," he said.

Quoting the Labour MP Gerald Kaufman, the prime minister said: "It is closing time in the last chance saloon. This replaces a failed regulatory system with one that will work because it has some real independence at its heart and is going to be properly overseen without allowing parliament to endlessly interfere." But he stressed that the new royal charter only sets up the body to recognise the regulator, and it remains a voluntary choice for the industry to decide whether to set up the system of independent regulation.

If newspapers refuse to co-operate with the regulator, or set up a body that is not accepted by the new recognition panel, they will be more liable to exemplary damages in the event that they recklessly publish inaccurate stories.

In a statement, Associated Newspapers, News International, the Telegraph Media Group and the Express's publishers, Northern & Shell, said they would be taking "high-level legal advice" before deciding if they could join the new watchdog. The deal, they said, raised several deeply contentious issues.

"No representative of the newspaper and magazine industry had any involvement in, or indeed any knowledge of, the cross-party talks on press regulation that took place on Sunday night," they said. "We have only late this afternoon seen the royal charter that the political parties have agreed between themselves and, more pertinently, the recognition criteria, early drafts of which contained several deeply contentious issues which have not yet been resolved with the industry."

Downing Street sought to reassure small-scale web-based news providers and blogs that they would not be required to co-operate with the new regulatory system. No 10 said bloggers, tweeters, news aggregators and social networking sites such as Facebook or Twitter, as well as special interest titles, would be excluded, but there was concern that a workable definition of these would be difficult to come up with.

Cameron switched from a stance of defiance last Thursday to apparent capitulation on a range of points over the weekend.

Major parts of the newspaper industry have issues with the statute that would be laid down to guarantee that no change could be made to the royal charter, along with worries over the proposals to empower the regulator to force apologies. They are also strongly opposed to the notion that, for the first time, people from outside the industry would be involved in drawing up the newspaper code of practice, a code that has been widely praised and has been adapted by regulators in other countries.

"This is a political deal between the three parties and Hacked Off. It is not a deal with the newspapers," said one senior executive in one newspaper group.

Associated Newspapers, News International and the Telegraph Media Group had been exploring the possibility of boycotting the government-sanctioned regulator and setting up their own body if they believed it could threaten freedom of the press.

This is still "very much a live discussion", said one source. "Nobody is threatening it, or saying we will do it, but we won't be making a decision before we have had high-level legal advice," said the insider, who claimed that the existence of a "no change" statute to guarantee that the royal charter could not be amended by the privy council still opened the door to political interference.

Alan Rusbridger, the Guardian's editor-in-chief, gave a cautious welcome to the deal. He said: "We welcome the fact that there has been cross-party agreement. The regulatory settlement is by and large a fair one, with compromises on all sides. We retain grave reservations about the proposed legislation on exemplary damages. The agreed terms are not ideal but after two years of inquiry and debate we finally have the prospect of what the public wants - a robust regulator that is independent of both press and politics. It's a big improvement on what went before."

A second senior executive said the industry had already been advised that the proposal that the regulatory body could force newspapers into making apologies it did not agree would be illegal, as it would be contrary to article 10 of the European convention of human rights which protects freedom of speech. The editor of the Independent, Chris Blackhurst, said the proposed regulator "isn't perfect, but neither is it terrible", and said he did not think it would threaten journalism at his paper.

Views hardened within larger newspaper groups as the implications of a deal began to sink in, with Tim Jotischky, the deputy editor of the Telegraph, tweeting late on Monday: "We can never lecture a Mugabe or a Putin on freedom of expression again. Quite an achievement for Hacked Off et al."

The Newspaper Society condemned the deal for placing what it said was a "crippling burden" on the UK's 1,100 local newspapers. Adrian Jeakings, president of the society and chief executive of Archant, one of the biggest local newspaper publishers, said the political deal "completely ignores the Leveson recommendations on the local press" and opened an arbitration system that could affect the viability of local papers.

The Organisation for Security and Co-operation in Europe (OSCE) – representing 57 member states – warned that the new regulator could threaten freedom of expression in the UK. Dunja Mijatovic, the OSCE's representative on freedom of the media,urged Britain not to sacrifice its tradition of press self-regulation, which was regarded around the world as best practice.

"A government-established regulatory body, regardless of how independent it is intended to be, could pose a threat to media freedom," she said, adding that the phone-hacking scandal was a "criminal issue" which was being dealt with through prosecutions in the criminal courts. "This should not be used as an excuse to rein in all print media," she said.

Clegg's office rounded on Cameron, saying: "If it looks chaotic that Letwin was meeting members of Hacked Off in Ed Miliband's office at 2am to discuss press regulation, that is because it was chaotic. It is chaotic because the prime minister walked out of the talks unilaterally on Thursday rather than sitting down and having sensible discussions. We have ended up where we hoped, and expected."

Harriet Harman, the shadow culture secretary and Labour's lead negotiator, hailed the concessions extracted from Cameron since last week. She said: "What changed after Thursday is we got free arbitration, with a small narrowing of the route into arbitration, to reassure the regional press, we got the direction of apologies and corrections, we got abandonment of the veto on the appointment to the board of the regulator, and there is a role for working journalists in the writing of the code. We also got them to accept the amendment tabled by Lord Wilf Stevenson entrenching the Royal Charter for the press. We have been storming along."

She also defended Labour's insistence on staying close to campaigners saying: "Hacked Off gave the victims of press intrusion a support network of protection to the victims – people that have been previously isolated and frightened. Hacked Off allowed them no longer to be victims, but agents for change. We have had hour after hour after hour of meetings with them to make sure we are all moving together. When you are working with people from outside Westminster you have to do politics in a completely different way."