David Cameron has moved to close a hole in the oversight of Britain’s intelligence agencies after it was revealed for the first time that they were creating “bulk personal datasets” containing millions of items of personal information, some of it gathered covertly without any statutory accountability.

Some of the data appears to have been gathered from other government departments as well as commercial organisations.

The disclosure came in a long-awaited 149-page report prepared by parliament’s intelligence and security committee (ISC) examining the oversight and capabilities of the UK intelligence agencies in the wake of the revelations of Edward Snowden, the former US National Security Agency (NSA) contractor.

The inquiry found the laws governing the agencies’ activities – including mass surveillance – require a total overhaul to make them more transparent, comprehensible and capable of restoring trust in their work.

The report said the legal framework is unnecessarily complicated and – crucially – almost impenetrable. The current laws could be construed as providing the agencies with a “blank cheque to carry out whatever actives they deem necessary”, it said.

“The legal framework has developed piecemeal and is unnecessarily complicated,” the report concluded. “We have serious concerns about the resulting lack of transparency, which is not in the public interest.”

In its key recommendation the committee proposed that “all the current legislation governing the intrusive capabilities of the security and intelligence agencies be replaced by a new, single act of parliament”.

Although all sides welcomed the proposed clarification and consolidation of the complex laws, the precise content of that legislation will be hotly contested in the next parliament as advocates of security and privacy argue over how the new legal lines must be drawn.

In a sign of the scope of the existing loopholes David Cameron, the prime minister, rushed out a statement in the wake of the report saying the intelligence services commissioner, Sir Mark Waller, would be given “statutory powers of oversight of use of bulk personal datasets”.

In a heavily censored section of the report, the committee said the datasets contain personal information about a wide range of people and vary in size from hundreds to millions of records.



It added that there is no legal constraint on storage, restraint, retention, sharing and destruction. Surveillance agencies do not require ministerial authorisation in any way to access the information. Committee members said the information gathered in the bulk personal datasets is not necessarily gathered by the agencies, implying it may have been harvested by either commercial organisations or other government agencies for other purposes, and then handed over.

The datasets vary from hundreds to millions of records and are acquired through overt and covert channels, the committee disclosed and are not derived from any specific legal power.

Hazel Blears, the leading Labour committee member, was reluctant to disclose the nature of the information collected, but likened the datasets to a telephone directory and added they applied only to people in a certain category of interest to the agencies.

The committee state they were “concerned that until publication of its report there had been no public or Parliamentary consideration of the related privacy consideration and safeguards”. It appears that Waller raised the issue of the bulk personal datasets, telling the committee “it is a risk that some individuals will misuse the powers of access to private data which must be carefully guarded against”.

More broadly, the committee found that existing laws are not being broken by the agencies and insisted the bulk collection of data by the government does not amount to mass surveillance or a threat to individual privacy.

Blears, said: “What we’ve found is that the way in which the agencies use the capabilities they have is authorised, lawful, necessary and proportionate.

“But what we’ve also found is there is a degree of confusion and lack of transparency about the way in which this is authorised in our legal system. It is that lack of transparency that leads to people reaching the conclusion that there is blanket surveillance, indiscriminate surveillance.”

The report confirmed that GCHQ does have the capability for bulk interceptions but denied that represents a blanket or indiscriminate surveillance, saying the security services neither have the resources nor motive to look at more than a small fraction of the material available to it.

It said GCHQ could theoretically access communications traffic from a small percentage of the 100,000 servers which make up the core infrastructure of the internet, chosen on the basis of the possible intelligence value of the traffic they carry.

But in a controversial move, the committee redacted the percentage of items sent across the internet in a single day that are ever selected to be read by a GCHQ analysts, emphasising: “They will have gone through several stages of targeting filtering and searching so they are believed to be the ones of the very highest intelligence value.”

The report added: “The current legal framework of external and internal communications has led to much confusion. However, we have established that bulk interception cannot be used to target the communications of an individual in the UK without a specific authorisation naming the individual signed by a secretary of state.”

Despite the tone of reassurance over the conduct of the agencies, the committee detailed a succession of reforms designed to constrain the security services. It said: “We consider the communications of UK nationals abroad should receive the same level of protection under the law irrespective of where the person is located. The interception and communication of data should be authorised through an individual warrant signed by a secretary of state.”

It also recommended that misuse of GCHQ’s interception capabilities should become a criminal offence. It called for commissions responsible for overseeing the activities of the agencies to be put on a statutory footing since the current non-statutory framework is “unsatisfactory and inappropriate”.

It also found it unacceptable that MI6 undertakes intrusive operations abroad but is under no requirement to keep comprehensive and accurate records of when it uses these powers.

In a significant reform the committee recommended new privacy constraints on the communications data that goes beyond the narrow definition of “what, when, where of communications”, such as web domains visited or location-tracking information in a smartphone.

Shami Chakrabarti, director of Liberty, said: “The ISC has repeatedly shown itself as a simple mouthpiece for the spooks – so clueless and ineffective that it’s only thanks to Edward Snowden that it had the slightest clue of the agencies’ antics.

“The committee calls this report a landmark for ‘openness and transparency’ – but how do we trust agencies who have acted unlawfully, hacked the world’s largest sim card manufacturer and developed technologies capable of collecting our login details and passwords, manipulating our mobile devices and hacking our computers and webcams?

“No doubt it would be simpler if we went along with the spies’ motto of ‘no scrutiny for us, no privacy for you’ – but what an appalling deal for the British public.”