The UK intelligence agency GCHQ has repeatedly warned it fears a "damaging public debate" on the scale of its activities because it could lead to legal challenges against its mass-surveillance programmes, classified internal documents reveal.

Memos contained in the cache disclosed by the US whistleblower Edward Snowden detail the agency's long fight against making intercept evidence admissible as evidence in criminal trials – a policy supported by all three major political parties, but ultimately defeated by the UK's intelligence community.

Foremost among the reasons was a desire to minimise the potential for challenges against the agency's large-scale interception programmes, rather than any intrinsic threat to security, the documents show.

The papers also reveal that:

• GCHQ lobbied furiously to keep secret the fact that telecoms firms had gone "well beyond" what they were legally required to do to help intelligence agencies' mass interception of communications, both in the UK and overseas.

• GCHQ feared a legal challenge under the right to privacy in the Human Rights Act if evidence of its surveillance methods became admissible in court.

• GCHQ assisted the Home Office in lining up sympathetic people to help with "press handling", including the Liberal Democrat peer and former intelligence services commissioner Lord Carlile, who this week criticised the Guardian for its coverage of mass surveillance by GCHQ and America's National Security Agency.

The most recent attempt to make intelligence gathered from intercepts admissible in court, proposed by the last Labour government, was finally stymied by GCHQ, MI5 and MI6 in 2009.

A briefing memo prepared for the board of GCHQ shortly before the decision was made public revealed that one reason the agency was keen to quash the proposals was the fear that even passing references to its wide-reaching surveillance powers could start a "damaging" public debate.

Referring to the decision to publish the report on intercept as evidence without classification, it noted: "Our main concern is that references to agency practices (ie the scale of interception and deletion) could lead to damaging public debate which might lead to legal challenges against the current regime." A later update, from May 2012, set out further perceived "risks" of making intercepts admissible, including "the damage to partner relationships if sensitive information were accidentally released in open court". It also noted that the "scale of interception and retention required would be fairly likely to be challenged on Article 8 (Right to Privacy) grounds".

The GCHQ briefings showed the agency provided the Home Office with support in winning the PR battle on the proposed reforms by lining up people to talk to the media – including Lord Carlile, who on Wednesday gave a public lecture condemning the Guardian's decision to publish stories based on the leaked material from Snowden.

Referring to the public debate on intercept evidence, the document notes: "Sir Ken McDonald [sic] (former DPP [director of public prosecutions]), Lord Goldsmith (former AG [attorney general]) and David Davis (former Shadow HSec [home secretary) [have been] reiterating their previous calls for IaE [intercept as evidence].

"We are working closely with HO [Home Office] on their plans for press handling when the final report is published, e.g. lining up talking heads (such as Lord Carlisle [sic], Lord Stevens, Sir Stephen Lander, Sir Swinton Thomas)."

Carlile was the independent reviewer of terrorism legislation in 2001-11, and was awarded a CBE in 2012 for his services to national security.

Another top GCHQ priority in resisting the admission of intercepts as evidence was keeping secret the extent of the agency's co-operative relationships with telephone companies – including being granted access to communications networks overseas.

In June, the Guardian disclosed the existence of GCHQ's Tempora internet surveillance programme. It uses intercepts on the fibre-optic cables that make up the backbone of the internet to gain access to vast swaths of internet users' personal data. The intercepts are placed in the UK and overseas, with the knowledge of companies owning either the cables or landing stations.

The revelations of voluntary co-operation with some telecoms companies appear to contrast markedly with statements made by large telecoms firms in the wake of the first Tempora stories. They stressed that they were simply complying with the law of the countries in which they operated.

In reality, numerous telecoms companies were doing much more than that, as disclosed in a secret document prepared in 2009 by a joint working group of GCHQ, MI5 and MI6.

Their report contended that allowing intercepts as evidence could damage relationships with "Communications Service Providers" (CSPs).

In an extended excerpt of "the classified version" of a review prepared for the Privy Council, a formal body of advisers made up of current and former cabinet ministers, the document sets out the real nature of the relationship between telecoms firms and the UK government.

"Under RIPA [the Regulation of Investigatory Powers Act 2000], CSPs in the UK may be required to provide, at public expense, an adequate interception capability on their networks," it states. "In practice all significant providers do provide such a capability. But in many cases their assistance – while in conformity with the law – goes well beyond what it requires."

GCHQ's internet surveillance programme is the subject of a challenge in the European court of human rights, mounted by three privacy advocacy groups. The Open Rights Group, English PEN and Big Brother Watch argue the "unchecked surveillance" of Tempora is a challenge to the right to privacy, as set out in the European convention on human rights.

That the Tempora programme appears to rely at least in part on voluntary co-operation of telecoms firms could become a major factor in that ongoing case. The revelation could also reignite the long-running debate over allowing intercept evidence in court.

GCHQ's submission goes on to set out why its relationships with telecoms companies go further than what can be legally compelled under current law. It says that in the internet era, companies wishing to avoid being legally mandated to assist UK intelligence agencies would often be able to do so "at little cost or risk to their operations" by moving "some or all" of their communications services overseas.

As a result, "it has been necessary to enter into agreements with both UK-based and offshore providers for them to afford the UK agencies access, with appropriate legal authorisation, to the communications they carry outside the UK".

The submission to ministers does not set out which overseas firms have entered into voluntary relationships with the UK, or even in which countries they operate, though documents detailing the Tempora programme made it clear the UK's interception capabilities relied on taps located both on UK soil and overseas.

There is no indication as to whether the governments of the countries in which deals with companies have been struck would be aware of the GCHQ cable taps.

Evidence that telecoms firms and GCHQ are engaging in mass interception overseas could stoke an ongoing diplomatic row over surveillance ignited this week after the German chancellor, Angela Merkel, accused the NSA of monitoring her phone calls, and the subsequent revelation that the agency monitored communications of at least 35 other world leaders.

On Friday, Merkel and the French president, François Hollande, agreed to spearhead efforts to make the NSA sign a new code of conduct on how it carried out intelligence operations within the European Union, after EU leaders warned that the international fight against terrorism was being jeopardised by the perception that mass US surveillance was out of control.

Fear of diplomatic repercussions were one of the prime reasons given for GCHQ's insistence that its relationships with telecoms firms must be kept private .

Telecoms companies "feared damage to their brands internationally, if the extent of their co-operation with HMG [Her Majesty's government] became apparent", the GCHQ document warned. It added that if intercepts became admissible as evidence in UK courts "many CSPs asserted that they would withdraw their voluntary support".

The report stressed that while companies are going beyond what they are required to do under UK law, they are not being asked to violate it.

Shami Chakrabarti, Director of Liberty and Anthony Romero Executive Director of the American Civil Liberties Union issued a joint statement stating:

"The Guardian's publication of information from Edward Snowden has uncovered a breach of trust by the US and UK Governments on the grandest scale. The newspaper's principled and selective revelations demonstrate our rulers' contempt for personal rights, freedoms and the rule of law.

"Across the globe, these disclosures continue to raise fundamental questions about the lack of effective legal protection against the interception of all our communications.

"Yet in Britain, that conversation is in danger of being lost beneath self-serving spin and scaremongering, with journalists who dare to question the secret state accused of aiding the enemy.

"A balance must of course be struck between security and transparency, but that cannot be achieved whilst the intelligence services and their political masters seek to avoid any scrutiny of, or debate about, their actions.

"The Guardian's decision to expose the extent to which our privacy is being violated should be applauded and not condemned."