William Hague has hailed GCHQ's 'democratic accountability', but legislation drafted before a huge expansion of internet traffic appears to offer flexibility

The legal loopholes that allow GCHQ to spy on the world

William Hague was adamant when he addressed MPs on Monday last week. In an emergency statement (video) forced by the Guardian's disclosures about GCHQ involvement with the Prism programme, the foreign secretary insisted the agency operated within a "strong framework of democratic accountability and oversight".

The laws governing the intelligence agencies provide "the strongest systems of checks and balances for secret intelligence anywhere in the world", he said.

Leaked documents seen by the Guardian give the impression some high-ranking officials at GCHQ have a different view.

In confidential briefings, one of Cheltenham's senior legal advisers, whom the Guardian will not name, made a note to tell his guests: "We have a light oversight regime compared with the US".

The parliamentary intelligence and security committee, which scrutinises the work of the agencies, was sympathetic to the agencies' difficulties, he suggested.

"They have always been exceptionally good at understanding the need to keep our work secret," the legal adviser said.

Complaints against the agencies, undertaken by the interception commissioner, are conducted under "the veil of secrecy". And the investigatory powers tribunal, which assesses complaints against the agencies, has "so far always found in our favour".

The briefings offer important glimpses into the GCHQ's view of itself, the legal framework in which it works, and, it would seem, the necessity for reassuring the UK's most important intelligence partner, the United States, that sensitive information can be shared without raising anxiety in Washington.

None of the documents advocates law-breaking – quite the opposite. But critics will say they highlight the limitations of the three pieces of legislation that underpin the activities of GCHQ, MI5 and MI6 – which were repeatedly mentioned by Hague as pillars of the regulatory and oversight regime during his statement to the Commons.

The foreign secretary said GCHQ "complied fully" with the Regulation of Investigatory Powers Act (Ripa), the Human Rights Act (HRA) and the Intelligence Services Act (Isa).

Privacy campaigners argue the laws have one important thing in common: they were drafted in the last century, and nobody involved in writing them, or passing them, could possibly have envisaged the exponential growth of traffic from telecoms and internet service providers over the past decade.

Nor could they have imagined that GCHQ could have found a way of storing and analysing so much of that information as part of its overarching Mastering the Internet project.

The Tempora programme appears to have given Britain's spymasters that resource, with documents seen by the Guardian showing Britain can retain for up to 30 days an astronomical amount of unfiltered data garnered from cables carrying internet traffic.

This raises a number of questions about the way GCHQ officials and ministers have legitimised the programme.

The briefings, which are entitled UK Operational Legalities, stress that GCHQ "is an organisation with a highly responsible approach to compliance with the law".

GCHQ also has a well staffed legal team, known as OPP-LEG, to help staff navigate their way through the complexities of the law.

But there appears to be some nervousness about Tempora. In a paper written for National Security Agency (NSA) analysts entitled A Guide to Using Internet Buffers at GCHQ, the author notes: "[Tempora] represents an exciting opportunity to get direct access to enormous amounts of GCHQ's special source data.

"As large-scale buffering of metadata and content represent a new concept for GCHQ's exploitation of the internet, GCHQ's legal and policy officers are understandably taking a careful approach to their access and use."

So how did GCHQ secure the legal authority for setting up Tempora, and what safeguards are in place for sharing the intelligence with the Americans? According to the documents, the British government used Ripa to get taps on to the fibre-optic cables.

These cables carry internet traffic in and out of the country and contain details of millions of emails and web searches. The information from these cables went straight into the Tempora storage programme.

In one presentation, which appeared to be for US analysts from the NSA, GCHQ explained: "Direct access to large volumes of unselected SSE data [is] collected under a Ripa warrant."

The precise arrangement between the firms is unclear, as are the legal justifications put before ministers. Isa gives GCHQ some powers for the "passive collection" of data, including from computer networks.

But it appears GCHQ has relied on paragraph four of section 8 of Ripa to gain "external warrants" for its programmes.

They allow the agency to intercept external communications where, for instance, one of the people being targeted is outside Britain.

In most Ripa cases, a minister has to be told the name of an individual or company being targeted before a warrant is granted.

But section 8 permits GCHQ to perform more sweeping and indiscriminate trawls of external data if a minister issues a "certificate" along with the warrant.

According to the documents, the certificate authorises GCHQ to search for material under a number of themes, including: intelligence on the political intentions of foreign governments; military postures of foreign countries; terrorism, international drug trafficking and fraud.

The briefing document says such sweeping certificates, which have to be signed off by a minister, "cover the entire range of GCHQ's intelligence production".

"The certificate is issued with the warrant and signed by the secretary of state and sets out [the] class of work we can do under it … cannot list numbers or individuals as this would be an infinite list which we couldn't manage."

Lawyers at GCHQ speak of having 10 basic certificates, including a "global" one that covers the agency's support station at Bude in Cornwall, Menwith Hill in North Yorkshire, and Cyprus.

Other certificates have been used for "special source accesses" – a reference, perhaps, to the cables carrying web traffic. All certificates have to be renewed by the foreign secretary every six months.

A source with knowledge of intelligence confirmed: "Overall exercise of collection and analysis [is] done under a broad, overall legal authority which has to be renewed at intervals, and is signed off at a senior political level."

The source said the interception commissioner was able to "conclude that [the process] was not appropriate", and that the companies involved were not giving up the information voluntarily.

"We have overriding authority to compel [them] to do this," the source said. "There's an overarching condition of the licensing of the companies that they have to co-operate in this.

"Should they decline, we can compel them to do so. They have no choice. They can't talk about the warrant, they can't reveal the existence of it."

GCHQ says it can also seek a sensitive targeting authority (STA), which allows it snoop on any Briton "anywhere in the world" or any foreign national located in the UK.

It is unclear how the STA system works, and who has authority over it.

The intelligence agencies also have to take note of the HRA, which demands any interception is "necessary and proportionate".

But the documents show GCHQ believes these terms are open to interpretation – which "creates flexibility". When Tempora became fully functional in around 2011, GCHQ gave the NSA access to the programme on a three-month trial – and the NSA was keen to impress.

The US agency sent a briefing to some of its analysts urging them to show they could behave responsibly with the data. Under a heading – "The need to be successful!" – the author wrote: "As the first NSA users to receive operational access [to Tempora], we're depending on you to provide the business case required to justify expanded access. Most importantly we need to prove that NSA users can utilise the internet buffers in ways that are consistent with GCHQ's legal and policy rules.

"In addition, we need to prove that NSA's access … is necessary to prosecute our mission and will greatly enhance the production of the intelligence … success of this three-month trial will determine expanded NSA access to internet buffers in the future."

The NSA appears to have made a successful case. In May last year, an internal GCHQ memo said it had 300 analysts working on intelligence from Tempora, and the NSA had 250. The teams were supporting "the target discovery mission".

But the safeguards for the sharing of this information are unclear.

Though GCHQ says it only keeps the content of messages for three working days, and the metadata for up to 30 days, privacy campaigners here and in the US will want to know if the NSA is adhering to the same self-imposed rules. One concern for privacy campaigners is that GCHQ and the NSA could conduct intercepts for each other, and then offer to share the information – a manoeuvre that could bypass the domestic rules they have to abide by.

This was raised by MPs during last week's statement, with the former Labour home secretary David Blunkett calling for clarification on this potential loophole.

Last week, the Guardian sent a series of questions to the Foreign Office about this issue, but the department said it would not be drawn on it.

"It is a longstanding policy not to comment on intelligence matters; this includes our intelligence co-operation with the United States.

"The intelligence and security committee is looking into this, which is the proper channel for such matters."