Advisory body says ‘technological magic’ cannot resolve security objectives and privacy concerns about phone records and details of other communications

No “technological magic” can reconcile the security objectives and privacy concerns sparked by the National Security Agency’s bulk collection of communications data, an academic advisory panel reported on Thursday.

Convened by the US director of national intelligence, James Clapper, in response to a request from President Barack Obama, the panel did not specifically endorse any bulk collection conducted by the NSA on Americans’ phone records and international communications and foreigners’ emails, phone calls and internet searches.

Instead, the panel, heavy on computer science experts under the aegis of the National Academy of Sciences, rejected the idea that technology can resolve what 18 months’ worth of policy debate over surveillance, sparked by the intelligence whistleblower Edward Snowden, have not.

Examining a range of technical alternatives to bulk data collection, the panel concluded that none can fully replicate its primary potential investigative value: identifying new intelligence targets in the future from a massive pool of previously collected information.

“A choice to eliminate all forms of bulk collection would have its costs in intelligence capabilities,” the panel concluded, though it neither cited real-world cases nor assessed the efficacy of bulk surveillance in practice.

But the panel suggested that narrowing the parameters of what intelligence agencies seek to investigate would reduce the utility of bulk data for cases in which old information is unimportant. It likened bulk data collection to “telephone directories”, providing intelligence analysts with reference material for additional investigation, not necessarily usable threat information.

“It is the context of the investigation, rather than the technique for using collected metadata, that most influences the value of bulk collection,” the panel found.

Shunting away policy and legal questions but mindful of the unresolved privacy debate, the panel looked favorably on additional and automated safeguards on the use of collected data by intelligence analysts. However, judges on the secret surveillance court overseeing the NSA have found the agency to have misrepresented how its automated querying functions actually operate.

“Technical means can isolate collected data and automatically restrict queries that analysts make, and the way that these means work can be public without revealing sources and methods,” the panel judged.

Clapper convened the panel after Obama’s January 2014 directive constraining the NSA’s analysis of domestic bulk phone data. A component of the directive encouraged exploring “the feasibility of creating software that would allow the Intelligence Community more easily to conduct targeted information acquisition rather than bulk collection”.

Since the Guardian and other news organizations began publishing Snowden’s revelations, the intelligence agencies have portrayed bulk surveillance as a stark binary choice: either collect everyone’s data, or accept catastrophic terrorist attacks as inevitable. In August 2013, a senior US intelligence official, who would not speak for the record, told reporters that the NSA had no technical alternatives to collecting screengrabs of email inboxes, even though emails from legally protected US persons would be visible.

As acknowledged by the panel, a different advisory body, the government’s Privacy and Civil Liberties Oversight Board, found that in practice, the bulk collection of US domestic phone records did not prevent any terrorist attacks, and recommended shuttering the program. The same board found that the NSA’s bulk collection of foreigners’ communications and Americans’ international conversations was effective against terrorism, though its chairman said the program walked “right up to the line of constitutionality”.

The NSA and the Obama administration have stopped defending the necessity of bulk collection of US phone data, following a political uproar after Snowden revealed it. But a bill aimed at making the phone companies and not the NSA the repository of the call data failed last year, leaving surveillance reform with an uncertain future in the current Congress. The legal authority undergirding the ongoing bulk collection of US call records expires in June.

Mark Rumold, a lawyer with the Electronic Frontier Foundation, said the report’s assessment of bulk collection ought to have come before US intelligence began the practice in secret, even if its technical assessments were agnostic on policy recommendations.

“If what you’re trying to do is re-create past events, there’s no computer program that you can create that will substitute for that. They rightfully note that it’s a policy decision, whether or not the need to re-create past events is worth the tradeoff for citizens’ privacy,” Rumold said.

The panel also found that government’s definitions of “bulk” and “targeted” surveillance were often at odds with “the plain meaning of the words bulk and targeted”, as “with a broad discriminant such as ‘Syria’, collection is targeted”.

Additionally, the panel suggested that surveillance would inevitably expand beyond traditional communications platforms, and into networked, internet-accessible home appliances like Nest’s Wi-Fi enabled thermostat.

“In the committee’s view, [signals intelligence] has come to embrace almost any data stored on an electronic device. In a future that contains the Internet of Things, the scope will be even greater,” it found.