Another document from the original cache leaked by NSA whistleblower Edward Snowden in 2013 has just been published by The Intercept, and it further bolsters the view that government intelligence agencies’ bulk collection of data for investigatory purposes is counterproductive to national security efforts.

The release has been timed to coincide with the UK government’s move to update surveillance legislation and enshrine bulk collection powers at the heart of the security state. The controversial Investigatory Powers Bill is currently being debated in parliament.

Critics of the UK’s IP bill have included the security-focused Intelligence and Security Committee, which slammed an earlier draft for privacy failures, vague language and overly broad intrusive powers. Although the opposition Labour party, which withheld support in an earlier vote on the bill, now appears to be moving towards active support — after winning what it claimed were significant concessions from the government. The bill is continuing its progress through parliament, with further debate in parliament today, followed by a third reading.

The classified MI5 briefing document, which is a draft version dated February 2010, provides details of the UK domestic intelligence agency’s so-called ‘Digint’ program (digital intelligence), and notes that the agency’s efforts to collect and exploit data from UK web users’ digital footprints have “grown significantly over the last few years”.

However it also goes on to describe these efforts as being in “imbalance”.

“It can currently collect (whether itself or through partners such as NTAC[*]) significantly more than it is able to fully exploit,” the document states of MI5.

This creates a real risk of ‘intelligence failure’ ie from the Service being unable to access potentially life-saving intelligence from data that it has already collected.

“This creates a real risk of ‘intelligence failure’ ie from the Service being unable to access potentially life-saving intelligence from data that it has already collected,” it adds.

Or to put it another way, pile yet more hay onto the stack and don’t be surprised when it gets harder to keep tabs on a few interesting needles. Spy agencies ‘drowning in data’ has been a warning refrain sounded for multiple years on both sides of the pond.

And yet the UK government continues to seek to greatly expand the volume of data available to state agencies via the IP bill — including, for example, a requirement that ISPs log the web activity of all users for a full year, as well as provisions for thematic warrants to authorize the interception of the communications of multiple people, and for bulk equipment interference to sanction the mass hacking of devices by the state.

Despite MI5 flagging up a risk of intelligence failures owing to having collected more data than it could profitably analyze, the 2010 briefing document also reveals the agency continued to push the case for gathering “substantial” amounts more data.

“Given the increasing importance of a target’s ‘digital footprint’ to investigations, a substantial increase in coverage is required,” it said, identifying increasing coverage as the third of its three main ambitions.

The other two were: fixing the “immediate challenge” posed by its data vs analysis imbalance; and getting multiple UK security agencies to work together on a joint, strategic approach to cross-leverage data collection and analysis capabilities.

Elsewhere in the document the agency writes of increased usage of the digint program generating “a vast body if information for us to work through if we are to build a clear intelligence assessment”.

One of the most significant intelligence failures in the UK in recent times was the 2013 murder of British soldier, Lee Rigby, by two men who had espoused extremist views online. Both of the killers were on the radar of intelligence agencies prior to the murder, and an ISC report subsequently detailed a series of agency failures (although it did not point the full finger of blame at the intelligence agencies — concluding instead that only Facebook could have prevented the killing by sharing more data with the security agencies).

Returning to the 2010 briefing document, it is unclear how MI5 specifically intended to improve its ability to analyze the increasing amounts of data it was intending to pull in — although it also writes of a need to “develop at pace… capabilities (i.e. processes, people and technology) which will enable it to improve investigative value from its collection investment”.

It adds that successfully fixing its data volume vs analysis imbalance would be “essential if we are to protect our existing capabilities in the face of evolving technology and target behaviors”.

*NTAC, the National Technical Assistance Center, was a Home Office agency established for data decryption and analysis, whose responsibilities have since been folded into GCHQ