Agencies privately concede that ‘intrusive’ practices can invade privacy and that data is gathered on people ‘unlikely to be of interest’

Britain’s intelligence agencies have been secretly collecting bulk personal data since the late 1990s and privately admit they have gathered information on people who are “unlikely to be of intelligence or security interest”.

Disclosure of internal MI5, MI6 and GCHQ documents reveals the agencies’ growing reliance on amassing data as a prime source of intelligence even as they concede that such “intrusive” practices can invade the privacy of individuals.

A cache of more than 100 memorandums, forms and policy papers, obtained by Privacy International during a legal challenge over the lawfulness of surveillance, demonstrates that collection of bulk data has been going on for longer than previously disclosed while public knowledge of the process was suppressed for more than 15 years.

The files show that GCHQ, the government’s electronic eavesdropping centre based in Cheltenham, was collecting and developing bulk data sets as early as 1998 under powers granted by section 94 of the 1984 Telecommunications Act.



The documents offer a unique insight into the way MI5, MI6, and GCHQ go about collecting and storing bulk data on individuals, as well as authorising discovery of journalists’ sources.

Bulk personal data includes information extracted from passports, travel records, financial data, telephone calls, emails and many other open or covert sources. Often they are “fused” together to help pinpoint suspects.

The frequency of warnings to intelligence agency staff about the dangers of trespassing on private records is at odds with ministers’ repeated public reassurances that only terrorists and serious criminals are having their personal details compromised.

For example, a newsletter circulated in September 2011 by the Secret Intelligence Agency (SIS), better known as MI6, cautioned against staff misuse. “We’ve seen a few instances recently of individuals crossing the line with their database use … looking up addresses in order to send birthday cards, checking passport details to organise personal travel, checking details of family members for personal convenience,” it says.

“Another area of concern is the use of the database as a ‘convenient way’ to check the personal details of colleagues when filling out service forms on their behalf. Please remember that every search has the potential to invade the privacy of individuals, including individuals who are not the main subject of your search, so please make sure you always have a business need to conduct that search and that the search is proportionate to the level of intrusion involved.” Better where possible to use “less intrusive” means, it adds.

Theresa May unveils UK surveillance measures in wake of Snowden claims Read more

There has been disciplinary action. Between 2014 and 2016, two MI5 and three MI6 officers were disciplined for mishandling bulk personal data. Last year, it was reported that a member of GCHQ’s staff had been sacked for making unauthorised searches.



The papers show that data handling errors remain a problem. Government lawyers have admitted in responses to Privacy International that between 1 June 2014 and 9 February this year, “47 instances of non-compliance either with the MI5 closed section 94 handling arrangements or internal guidance or the communications data code of practice were detected.” Four errors involved “necessity and proportionality” issues; 43 related to mistransposed digits, material that did not relate to the subject of investigation or duplicated requests.

Another MI5 file notes that datasets “contain personal data about individuals, the majority of whom are unlikely to be of intelligence or security interest”.

The documents have been disclosed before a trial due later this summer at the investigatory powers tribunal, which hears complaints about state-authorised surveillance and the intelligence agencies. IPT sessions hear secret evidence behind closed doors.

Release of these internal records follows admissions by David Cameron and by parliament’s intelligence and security committee (ISC) last year in the wake of revelations by the US whistleblower Edward Snowden.



The most recent documents refer to a “more onerous authorisation process” after the prime minister’s avowal of the “use of bulk personal data”. They provide fresh detail of what is happening in the intelligence agencies.



Web and phone companies are required to retain data for official access for 12 months, but the intelligence agency documents make clear that acquired bulk data sets can be held far longer.

An MI5 memorandum says retention of “low intrusion” material needs to be reviewed only every two years. Some key words are missing from the memo, but it adds: “In MI5, a maximum retention period [redaction] is applied to [bulk personal data]. This can be increased in exceptional circumstances via a policy waiver. This waiver must be authorised by a senior MI5 official and agreed by the BPDRP [bulk data retention review panel] but shall be subject to a detailed review.”

Bulk personal data is exchanged with “foreign agencies”, presumably mainly those from other countries in the UK’s traditional “Five Eyes” alliance – the USA, Canada, Australia and New Zealand.

European court to consider legality of UK surveillance laws Read more

The documents do not specify every type of information exploited but give examples and broad categories: population data and passports, travel records, financial data and communications information. “Some of this data is publicly available, some of it is purchased and some of it is acquired covertly in accordance with SIS statutory functions,” according to an MI6 note.

Monetary information is held. “The fact that [MI5] holds bulk financial, albeit anonymised data is assessed to be a high corporate risk since there is no public expectation that the service will hold or have access to this data in bulk. Were it to become widely known that the service held this data, the media response would most likely be unfavourable and probably inaccurate.

“In some cases, it may be necessary for the relevant team to approach the data provider to examine whether any unnecessary/extraneous parts of the dataset can be removed prior to acquisition. Such extraneous data might include large numbers of minors, details of earnings or medical information.”



Death provides no escape. “Policy and processes in relation to bulk personal data is the same for both the living and the dead,” a combined agencies memo records.

Each intelligence service has its own database, it appears from the documents. For MI5, storage of bulk data is at their London HQ, Thames House. “In order to ensure the security and integrity of the datasets that the service relies upon for its enhanced analytical capabilities and to reassure data providers that their data will be handled securely, it is essential that the necessary physical controls are in place to mitigate unauthorised access to, or loss of, this information during transportation to and subsequent storage in Thames House.”

The justification for assembling such sophisticated databases, according to an MI5 document, is that it speeds up the process of detecting suspects. “By integrating bulk data [redaction] with information about individual subjects of interest from other sources of intelligence (liaison relationships, agent reporting, intercept, eavesdropping, surveillance) and from ‘fusing’ different data-sets in order to identify common links, we can better understand target networks, locations and behaviours, enabling a greater depth and breadth of target coverage.

“The fragmentary nature of many intelligence leads and the magnitude of the threat all mean that there is currently no effective method of resolving identities in a timely fashion without using bulk data.”

The standard MI5 form for acquisition of bulk data requires agency staff to a tick box if it holds sensitive personal data such as “biometric, financial, medical, racial or ethnic origin, religious, journalistic, political, legal, sexual or criminal activity” and membership of a trade union. MI5 officers also need to explain why acquisition is “necessary and proportionate”.

The documents show how alert the agencies are to their legal obligations. They refer to the agencies’ “ethics team”, the need for “proportionality” and “necessity”. One note stresses that GCHQ employees’ conditions of employment state that “unauthorised entry to computer records may constitute gross misconduct”.

But the papers also reveal how much latitude the law – notably Ripa, the Telecommunications Act, and the Data Protection Act – in practice gives them.



Investigatory powers bill: the key points Read more

The documents include for the first time certificates under section 28 of the Data Protection Act – signed by David Blunkett and Jack Straw in 2001 when they were home and foreign secretary respectively – which provided secrecy about authorised bulk data interceptions under section 94 of the Telecommunications Act. The existence of such directions were not disclosed until last year.

The quantity of information the agencies have been forced to release suggests their long-established position of “neither confirming nor denying” any operational details may be crumbling at the edges.

In parliamentary debate over the investigatory powers bill, the government has argued that the security services only conduct targeted searches of data under legal warrants in pursuit of terrorist or criminal activity and that bulk interception is necessary as a first step in that process.

Millie Graham Wood, a legal officer at Privacy International, said: “The information revealed by this disclosure shows the staggering extent to which the intelligence agencies hoover up our data.

“This highly sensitive information about us is vulnerable to attack from hackers, foreign governments and criminals. The agencies have been doing this for 15 years in secret and are now quietly trying to put these powers on the statute book for the first time in the investigatory powers bill, which is currently being debated in parliament. These documents reveal a lack of openness and transparency with the public about these staggering powers and a failure to subject them to effective parliamentary scrutiny.”

A Home Office spokesman said: “Bulk powers have been essential to the security and intelligence agencies over the last decade and will be increasingly important in the future.

“The acquisition and use of bulk provides vital and unique intelligence that the security and intelligence agencies cannot obtain by any other means. The security and intelligence agencies use the same techniques that modern businesses increasingly rely on to analyse data in order to overcome the most significant national security challenges.”

