Classified mass-surveillance manuals for UK spies have been published today amid a legal battle against the British government.

The newly obtained documents set out Blighty's secret do's and don'ts for monitoring populations. The files acknowledge that chapter and verse on the lives of people "of no security interest" lie within the spooks' secret databases – and analysts and agents are simply told to avoid pulling up their information.

If you're the kind of person who thinks spies aren't interested in normal citizens, and thus they have nothing to worry about, guess again: your information is in the UK government's hands, and you are just a few keystrokes away. And, it appears, some Brit spies have no problem looking up their families, colleagues and even themselves for trivialities such as sending birthday cards.

As we've long known, for decades now GCHQ, MI5 and MI6 routinely extract people's personal information from files kept by private and public organizations, and plonk it all in highly classified searchable databases.

Now the rules and policies in place since 2001 overseeing this mining of data have been revealed. They include:

Advice urging agents to not search for themselves in the databases, especially for things like looking up where they've been traveling – presumably to remind themselves of their trips and how much to put down on their expenses: An example of an inappropriate ‘self search’ would be to use the database to remind yourself where you have travelled so you can update your records. This is not a proportionate use of the system, as you could find this information by another means (i.e. check the stamps in your passport or keep a running record of your travel) that would avoid collateral intrusion into other people’s data.

In fact, just don't search for anyone who isn't actually suspected of wrongdoing: We've seen a few instances recently of individual users crossing the line with their database use, looking up addresses in order to send birthday cards, checking passport details to organise personal travel, checking details of family members for personal reasons. Another area of concern is the use of the database as a 'convenient' way to check the personal details of colleagues when filling out service forms on their behalf. Please remember that every search has the potential to invade the privacy of individuals, including the privacy of individuals who are not the main subject of your search, so please make sure you always have a business need to conduct that search and that the search is proportionate to the level of intrusion involved.

The databases contain people "of no security interest" and yet there they are completely searchable, so analysts are simply told mind where they're going: In order to identify leads and counter fast-moving threats we have access to data from large external databases – about individuals of interest to us – but also, inescapably, about those who are of no security interest. Use of bulk data is therefore a particularly sensitive area, requiring careful consideration and strict adherence by users to that which is necessary and proportionate for their work.

There are mechanisms in place to catch snoopers: The database must not be a 'free for all' ... The use of analytical systems is monitored on a continuing basis through a variety of means — including technically — in order to detect misuse of the system and any unusual activity that gives rise to security concerns. Users will also be subject to random and routine spot checks to account for their activities on analytical systems at any time.

And there are apparently consequences for trawling databases for innocent people: You will be challenged on searches where you have not followed this guidance and asked to justify your approach. Failure to follow the guidance with no good reason could lead to a breach.

There are hundreds of pages, here, here and here, that we're still poring over. So, where did it all come from?

'Staggering extent to which the intelligence agencies hoover up our data'

In March of 2015, the UK government first admitted to the use of Bulk Personal Datasets (BPDs) by its intelligence staff – databases consisting of public and private information including call logs, internet traffic, and medical, financial, and travel records keyed to British individuals.

In June that year, Privacy International (PI) filed a legal challenge to this blanket spying, and was able to pry out of the UK government written answers to its questions and accompanying internal files. PI has today dumped hundreds of pages of these discovery documents online, after giving The Reg and others a few hours' heads up to leaf through them.

"The information revealed by this disclosure shows the staggering extent to which the intelligence agencies hoover up our data," said Millie Graham Wood, legal officer at PI.

"The agencies have been doing this for 15 years in secret and are now quietly trying to put these powers on the statute book for the first time, in the Investigatory Powers Bill, which is currently being debated in Parliament. These documents reveal a lack of openness and transparency with the public about these staggering powers and a failure to subject them to effective Parliamentary scrutiny."

Fifteen years? We fear it's been a little longer than that. Modern mass data slurping came to Blighty five years before the invention of the World Wide Web, with the UK Telecommunications Act of 1984. Section 94 of the 110-section bill authorised the Secretary of State to request any data from telcos operating in the UK on the grounds of "national security or relations with the government of a country or territory outside the United Kingdom, or the commercial interests of some other person."

Under that law, the Treasury would reimburse the telcos for any costs involved in funneling this data to the intelligence services, but they wouldn't be allowed to reveal what they were doing under a Section 94 order.

In 2001, under the administration of the then-Prime Minister and dull conference speaker Tony Blair, Section 94 collection was stepped up and reviewed every six months. BPD files now account for about five per cent of the data GCHQ holds, and an unspecified amount by the UK's domestic intelligence service MI5, the government filings state.

The intelligence agencies claim analysis of BPD files gave them the information to arrest and imprison terrorists in 2010 who were planning a bombing campaign in the UK, and others on weapons charges. The same databases were used to check the background of people with access to venues for the 2012 Olympic Games in London.

The UK government gives little detail as to how this data was handled initially, but in 2010 and 2015 the filings mention two code of conduct reviews regarding BPD handling. On the face of it they look about how you'd expect.

Analysts have to provide a valid "business" case for a BPD request and all access is logged and monitored. Analysts cannot look up themselves, their family members, or friends, and there's a maximum sentence of three months in prison and a fine for misuse of data.

Currently the storage of BPDs is reviewed every six months by an oversight committee, which in the case of MI5 consists of senior agency staff, including the ethics counselor and the legal adviser. GCHQ inspectors can audit files at will on visits.

It all sounds very proper, the kind of thing that human resources departments issue in three-ring binders to staff. Whether they are adhered to is another question. ®