Almost exactly two years before Barack Obama signed a bill to end the National Security Agency’s bulk collection of US phone records, the US president defended a domestic surveillance dragnet he would later claim credit for stopping.



“You can’t have 100% security and also then have 100% privacy and zero inconvenience. You know, we’re going to have to make some choices as a society,” Obama said on 7 June 2013, two days after the Guardian, thanks to whistleblower Edward Snowden, began revealing the reach of US surveillance.

The Snowden revelations began a process of classic Obama vacillation, bringing him from public defender of domestic mass surveillance to its reluctant and partial critic. It was a path crafted by politics, accelerated by law and fueled by fear.



On 1 June 2015, after the Senate conceded it would pass a bill ending the phone records dragnet, Obama’s spokesman claimed full ownership of the end of a program the US president had two years before said posed mere “modest encroachments on privacy”.



“To the extent that we’re talking about the president’s legacy, I would suspect that that would be a logical conclusion from some historians that the president ended some of these programs,” said the White House press secretary, Josh Earnest.

Many of those who voted for Obama thought that was the president they would get on day one of his tenure. They had good reason to think so. A former constitutional law professor, Obama began his national political career at the 2004 Democratic convention, where he gestured toward dissatisfaction with the emerging US surveillance apparatus: “We don’t like federal agents poking around our libraries.”

But they also had less prominent warnings that Obama was more comfortable with that apparatus than he let on. In July 2008, with the Democratic presidential nomination in hand, Obama cast a Senate vote to expand the NSA’s grasp, retroactively blessing a program that secretly collected Americans’ international communications in bulk. The bill Obama voted for extended the powers that ultimately created the internet dragnet known as Prism.

Once in power, Obama embraced domestic bulk data collection – even as the NSA’s massive appetite for phone records was in secret legal peril. A judge on the typically pliant foreign intelligence surveillance court, or Fisa court, was so incensed that the NSA’s software for querying US phone records went beyond the technological boundaries described to the panel that he effectively shut down NSA access to the data for much of 2009. Yet Obama continued the program.

Similarly, Obama permitted the NSA to gather Americans’ internet records in bulk until 2011 – and only disclosed that he shut the program down in response to the Guardian’s post-Snowden queries. It remains unclear why Obama ended it. But a recent Justice Department report makes clear that the FBI has for years been amassing “large collections” of the same data, raising questions about the meaningfulness of Obama’s actions.

But Obama’s defense of bulk surveillance began to show cracks as the Guardian, the Washington Post, the Intercept, Der Spiegel and publications worldwide continued publishing revelations from Snowden’s treasure troves. In August 2013, Obama announced he would empanel surveillance reviews to examine the programs’ necessity.

Barack Obama, then a candidate for the US Senate, told the 2004 Democratic convention: ‘We don’t like federal agents poking around our libraries.’ Photograph: Jim Rogash/WireImage

Those reviews savaged the domestic phone records collection. His own advisory body, heavy with intelligence veterans, told him in December 2013 to end the program. The government’s privacy watchdog agreed in harsher tones, judging that it never prevented a terrorist attack and was illegal. The NSA never conceded the illegality, even as a federal judge said the “almost Orwellian” program was probably unconstitutional, but abandoned its previous insistence that it was vital for counter-terrorism.

Obama agreed in January 2014 to divest NSA of its domestic phone records collection and called on Capitol Hill, where the NSA had not lost a fight since 1978, to pass legislation. But his administration was conspicuously silent on the leading congressional push for surveillance reform.

It was called the USA Freedom Act, and it had an unlikely architect. James Sensenbrenner, the Wisconsin Republican who authored the 2001 Patriot Act, pronounced himself stunned not just by the bulk collection, but how the NSA had justified it on a part of his law, known as Section 215.

Despite Obama’s reversal, Sensenbrenner and his allies received the cold shoulder from the White House. Administration officials refused to publicly indicate their support or opposition to the Freedom Act. At a 4 February 2014 hearing, members of the House judiciary committee predicted they would either back the bill or watch as the House failed to reauthorize Section 215 the following year.

The NSA did not like the Freedom Act, which went further than shutting down bulk phone records collection. It banned the NSA and FBI from warrantlessly querying hoards of international communications for Americans’ data, permanently put a privacy advocate on the Fisa court and permitted greater transparency for the companies receiving surveillance orders.

But the NSA could not get its preferred legislation, from its allies on the House intelligence committee, through a skeptical Congress. So it began working with the Freedom Act architects to soften the bill’s restrictions.

Intelligence-agency lawyers met with House GOP potentates and Freedom Act architects in May 2014 after the bill passed the judiciary committee. By the time discussions ended, the Freedom Act would bear only superficial resemblance to what committee members voted for. Gone was the so-called “backdoor search” ban. All the privacy and transparency restrictions were weakened, without any vote, to the point where the bill’s vague definitions left unclear how much bulk surveillance it actually banned.

A judiciary committee member and mass-surveillance critic, Democrat Zoe Lofgren of California, observed: “This is not how American democracy is supposed to work.” Yet the House passed the bill that month by a wide margin.

As they watched privacy groups abandon support, the bill’s architects defended the compromises out of political reality. They had a product that would end NSA bulk domestic phone records collection, reform the Fisa court somewhat, and provide more surveillance transparency than the status quo. They had also quieted the objections of the NSA and its allies and produced a bill that was passable in both senses of the word. And they had a belated White House endorsement.

But the Freedom Act did not have the US Senate’s support. Republicans set up procedural obstructions to kill it using a tried and true method: the specter of a terrorist attack. “God forbid we wake up tomorrow and [the Islamic State] is in the United States,” Florida Republican Marco Rubio said in a typical comment. Despite Obama’s allies pushing through a vote during the final weeks of Democratic Senate control, the USA Freedom Act failed in November 2014.

Senator Rand Paul, centre, has led congressional opposition to the NSA surveillance dragnet. Photograph: Andrew Harnik/AP

So the Freedom Act’s architects regrouped. To revive the bill in 2015, they set about placing surveillance advocates on the horns of the dilemma they had warned about the previous year: either pass the bill or lose a broader swath of FBI surveillance powers. For good measure, they conceded even more to the NSA’s allies.

The strategy depended on introducing the bill just weeks before the Patriot Act portions it needed for leverage expired. Meanwhile, bulk collection grew less tenable: a federal appeals court, the final step before the US supreme court, ruled it was illegal. The House passed the bill on 16 May. GOP presidential candidate Rand Paul, the rare Senate Republican to oppose bulk surveillance, used procedural obstructions of his own to force the partial Patriot Act expiration this week – which ultimately ensured passage of the Freedom Act on Tuesday afternoon by a wide, bipartisan, 67-32 margin.

Paul considers the bill insufficient. His allies in the House are now preparing to supplement it. They outlined to the Guardian on Tuesday a strategy to chip away at more of the NSA’s bulk surveillance powers, as well as its ability to insert flaws in encryption software standards. Cognizant that the USA Freedom Act leaves the vast majority of NSA powers untouched, they said they intend to fight for years – particularly through 2017, when the bill Obama voted for as a senator in July 2008 expires.

Obama will be gone from office by then. But his vacillations over surveillance continue. No sooner had the Freedom Act passed, than an administration official confirmed to the Guardian that the NSA would actually restart the bulk surveillance the bill bans – for another six months, to “transition” to a system where the telecom companies retain control of the data and provide it to the NSA or the FBI based on a court order that does not even require the typical probable-cause standard.

Snowden remains in exile in Russia. Obama has never shown him the slightest regard. Photograph: Barton Gellman/Getty Images

Snowden remains a pariah stranded in Russia, facing jail time for telling Americans about what their government was doing secretly, and illegally, in their name. Obama has never shown Snowden the slightest regard.

But Obama’s caveated, cautious and mutable stances on surveillance have earned him no allies. Despite his endless defenses of the NSA, its veterans demanded greater deference. Civil libertarians now view Obama as a tragic disappointment. With Hillary Clinton a longstanding security hawk, their only presidential hope is Rand Paul.

After the Freedom Act passed on Tuesday, Obama gave no indication that further reforms to the vast, labyrinthine apparatus of surveillance he commands are needed. Instead, he issued a statement praising the passage and insisting the bill will “provide greater public confidence in these programs”, which has been the NSA’s explicit goal since the Snowden leaks began on 5 June 2013.

Obama’s statement did one more thing. He portrayed himself as a stalwart and consistent champion of surveillance reform : “For the past 18 months, I have called for reforms that better safeguard the privacy and civil liberties of the American people while ensuring our national security officials retain tools important to keeping Americans safe.” It was a gesture toward history, toward the kind of president Obama might have been.