The British news media has never been so restricted, beset by the laws of an "authoritarian" government, greedy lawyers and dwindling editorial budgets, according to one of the industry's most important representative bodies.

The Society of Editors has submitted a dossier of evidence to Jack Straw, the Justice Secretary, claiming that "meritorious" articles by local newspapers are increasingly being suppressed because of the danger that legal action would bring ruinous costs. The dossier also contains examples of published stories exposing the behaviour of MPs, local authority leaders, owners of professional football and rugby clubs, business leaders and television personalities, which have been settled out of court for financial and not legal reasons, in order to avoid the danger of being bankrupted by the fees of the claimant's solicitors.

Speaking to The Independent on the Society's tenth anniversary, Robin Esser, executive managing editor of the Daily Mail, calls for a concerted effort to ensure that the concept of freedom of expression comes to be regarded as a constitutional right in Britain, as it is in America. He says the current threats to a free press are comparable to a many-headed serpent. "It's a bit like the hydra, every time you cut off one head two more appear ... we've got enormous issues at the moment."

Download the new Independent Premium app Sharing the full story, not just the headlines

Esser, a former editor of The Sunday Express who is chairman of the society's parliamentary and legal committee, criticises the police for abusing the Protection against Harassment Act 1997 (originally introduced to deter stalkers) and anti-terrorist legislation, in order to hamper the work of journalists. "Situations occur on a fairly regular basis of photographers being prevented from taking pictures in public places, being arrested and having their film taken away or inspected." He complains of "the increasing authoritarianism of the present government which sometimes without realising it, and sometimes realising it, introduces measures which impinge further on the media's rights and duty to ensure open democracy and open justice".

Last week the prominent barrister Ken Macdonald, QC, suggested that the recent growth of privacy legislation was strangling the freedom of the press. "We would pay a very high price indeed for underscoring the marketability of film stars and footballers," he said.

According to Esser, who has worked in Fleet Street for more than 40 years and reported on the first landings on the moon, "I think we are more restricted now than we've ever been before". The Society of Editors, which represents both newspapers and broadcasters, is concerned that the media, already weakened by the impact on editorial resources of falling advertising revenues, is being further cowed by the danger of incurring huge costs from libel actions brought by solicitors under "no win, no fee" conditional fee agreements (CFAs), which allow successful lawyers to double their charges. The issue is being investigated under a Litigation Costs Review being conducted for Mr Straw by Lord Justice Jackson.

The society has submitted to both the judge and Mr Straw a dossier which indicates that the local and regional media, in particular, has become terrified of the prospect of finding itself in court. Cases cited include a light-hearted piece on a local MP's views on expenses, which he complained about, leading to an out-of-court payment of £10,000 plus a bill for £26,000 in costs. Another local paper piece containing comments from a contestant on the BBC show Dragons' Den, who complained he had not been given the financial support promised by the dragons on screen, led to a settlement of £13,000 plus a further £7,000 paid in costs, despite the editor being advised that the piece was probably protected by the libel defence of "fair comment". A daily paper in the North of England paid out £10,000 (plus £7,000 costs) and made a full apology to a professional rugby league player for the local team who it had criticised after he was suspended for foul play by the league disciplinary committee. The settlement was in spite of legal advice that a defence of fair comment had "a high prospect of success".

A Welsh paper which featured a report by a conservation trust criticising plans to develop a building and land once occupied by the poet Dylan Thomas, provoked a writ from the property developer. On being advised of the potential costs under a CFA – where London solicitors can charge £390 an hour and then double the sum with a 100 per cent uplift or "success fee" if they win – the paper caved in, paying £10,000 in damages to the property developer and still incurring £16,000 in costs. The threat of a case brought under a CFA is often enough to ensure a story is withheld altogether. The dossier cites an investigation by a regional Sunday newspaper into a money-making cult, which was not published after the editor was warned that an action would be brought using CFA.

"In the light of this threat, the editor felt obliged to give the undertaking demanded, not because of any concerns about the quality of the journalism, the accuracy of the story, or the public interest, but purely because he feared that an adverse costs order may cause the newspaper to cease publishing," said the dossier. The report also cites the suppression of a story about an offshore company bidding to take control of the local football club. The paper had sought to expose the fact that the company had a subsidiary which had been criticised by business regulators in several EU countries but the article was dropped following a threat of an injunction and libel action.

Tony Jaffa, of Exeter-based solicitors Foot Anstey, who compiled the dossier, says large London law firms have become adept at exploiting the use of CFAs, which were originally introduced to increase public access to justice. He confirms: "The issue of CFAs has become an important tactical weapon in the litigation process."

Esser is adamant there has been a "chilling effect" on local papers and their willingness to investigate. "They will settle or they won't print a story which exposes hypocrisy or double dealing or disgraceful behaviour on the part of public servants, and that is a terrible loss."

The society continues to lobby ministers and their shadow counterparts to remove what it sees as unnecessary shackles on the media. It was exasperated by a backlash against new plans to allow journalists into Family Courts, with judges and officials seeking to maintain a ban on reporting. "It's madness," says Esser. "What newspaper has got people to send along to sit in court and not be able to write a story?"

He is hopeful, though, that progress is finally being made in introducing television cameras into courtrooms. "We feel that will help open justice and bring instant pictures to the public, just as the people in the public gallery can see what goes on. The judges fear some sort of OJ Simpson effect but that really isn't going to happen."

Though proud of the society's achievements over the difficult past decade, Esser knows that the hydra that threatens freedom of the press is far from slain. "There will be two more heads tomorrow," he says. "We have to remain vigilant."