Internal documents from UK spy agencies have revealed how mass surveillance techniques are used to collect dossiers of citizens' personal data. The files — which include internal memos from MI5, MI6, and GCHQ — show that the intelligence services have been gathering personal information into searchable "bulk personal datasets" or BPDs for years. These files can include travel records, financial data, and communications data such as telephone records, and are kept even on citizens that the agencies themselves acknowledge are "unlikely to be of intelligence or security interest."

The papers were obtained following a legal challenge by Privacy International, and show spies' often casual approach to proper data-handling protocol. "We’ve seen a few instances recently of individuals crossing the line with their database use," reads one letter circulated by the Secret Intelligence Agency (SIS) in 2011. "Looking up addresses in order to send birthday cards, checking passport details to organize personal travel, checking details of family members for personal convenience."

"every search has the potential to invade the privacy of individuals."

The memo continues: "Another area of concern is the use of the database as a ‘convenient way’ to check the personal details of colleagues when filling out service forms on their behalf. Please remember that every search has the potential to invade the privacy of individuals, including individuals who are not the main subject of your search, so please make sure you always have a business need to conduct that search and that the search is proportionate to the level of intrusion involved."

BPDs have been kept secret by the UK government for more than 15 years

Accessing an individual's data from a bulk personal dataset requires a signed warrant from a senior government minister, and all database access is monitored. But these internal memos do not make it clear exactly what information requires a warrant, and this situation is confusing when each agency has its own separate database. The government only admitted that it was amassing these searchable databases in March 2015 and even then, insisted they did not constitute mass surveillance. The documents show the existence of BPDs has been known about by senior government ministers from at least the late 1990s onward.

Millie Graham-Wood, a legal officer at Privacy International, said the documents showed "the staggering extent" to which UK spy agencies collect data. "These documents show the intelligence services using bulk personal datasets over a long period of time," Graham-Wood told The Financial Times. "The intelligence agencies have secretly given themselves access to potentially any and all recorded information about us." She added that access to these files "leaves us with more questions than it answers."

A Home Office spokesman said: "Bulk powers have been essential to the security and intelligence agencies over the last decade and will be increasingly important in the future [...] The security and intelligence agencies use the same techniques that modern businesses increasingly rely on to analyze data in order to overcome the most significant national security challenges."

Many of the powers described in these memos are likely to be enshrined in law by the investigatory powers bill — a controversial piece of legislation that is currently being debated by parliament and various committees. The government has said the bill will help fight terrorism, organized crime, and even cyberbullying, but it has been roundly criticized by tech experts. Facebook, Microsoft, Google, and Apple have all described the bill as a backward step for citizens' rights, while privacy groups say it could provide a template for authoritarian regimes who want to spy on their citizens.