THICK and fast they came at last, and more and more and more. On June 5th the Guardian, a British newspaper, reported that America’s National Security Agency (NSA) was collecting the telephone records of millions of Americans not suspected of crimes. A day later, the Washington Post reported the existence of a programme code-named PRISM, under which the NSA collects an unknown quantity of e-mails, internet phone-calls, photos, videos, file transfers and social-networking data from big internet companies, including Google, Facebook, Apple, YouTube, Skype, Microsoft and PalTalk—a video-chat service popular in the Middle East and among Muslims.

Members of the Senate Intelligence Committee confirmed that widespread collection of telephone records had been going on for years. As for PRISM, on June 8th America’s director of national intelligence, James Clapper, issued a rare public statement acknowledging its existence, but stressing that it is lawful and operates under a secret court that oversees intelligence-gathering. The leaker revealed himself the next day: Edward Snowden, a 29-year-old who had worked as a security contractor at the NSA for the past four years, employed by several private contractors.

In an interview with the Guardian (from Hong Kong, where he had holed up in hope of avoiding extradition to America), Mr Snowden said the NSA had built the capacity to ingest massive quantities of information from people not suspected of crimes. “I do not want to live in a world where everything I do and say is recorded,” said Mr Snowden. He believes that the public, not spies and secret courts, ought to decide whether this is right. He chose to reveal himself to avoid hiding behind the secrecy he abhors.

Since its creation in 1952 the NSA has been listening in on the world’s communications, from drunk Soviet leaders to Osama bin Laden’s satellite phone. Its thirst for information is well known. For decades, under a programme called Echelon, it has operated listening stations around the world that intercept troves of phone and data traffic.

Yet the latest disclosures suggest a scale of data-collection bigger than many experts had expected. A former high-ranking American official with ties to intelligence says more programmes skirting legality have still to be exposed. Mr Snowden has handed over “thousands” of classified documents, according to Glenn Greenwald, the Guardian journalist who broke the story, so more disclosures are probably on the way. His revelations have already prompted condemnation—and vigorous debate over the proper role and extent of modern government surveillance.

Insight into the telephone-data collection came from a leaked order from a FISA (Foreign Intelligence Surveillance Act) court instructing Verizon, one of the country’s biggest telecoms firms, with 121m American customers, to hand over information about all calls on its network “on an ongoing daily basis”. The FISA court was created in 1978 to approve or deny government requests to listen to foreigners’ calls on the ground of national security. Other telecoms firms are believed to deliver data under similar FISA orders, which appear to be renewed every three months.

The order does not give the government the right to listen to the content of calls, as Barack Obama, in response to the leak, emphatically told Americans. For that, law-enforcement agents need a separate warrant: one far harder to obtain because it requires suspicion of particular individuals and proof that “normal investigative procedures have been tried and failed”. Instead, the NSA has hoovered up “metadata”—the records of who people call, when, for how long, and so on.

Back when telephones were plugged into walls and data analysis was done by humans, the usefulness of metadata was limited: hence the lower evidentiary standards required to obtain them. But thanks to powerful computers that can map people’s associations, and mobile phones that pinpoint a person’s movements, metadata can now provide a detailed portrait of who people know, where they go and their daily routines. The NSA may be able to use metadata to identify connections between people even if they have never shared a direct link, just as Facebook can predict which people a user may know. From a security point of view, what matters is getting all the information available. At the same time, the need to examine data at a moment’s notice has shifted the regime to “collection first” and analysis later, under FISA approval.

The details of PRISM are murkier. The initial leak for the programme was a computer slide presentation, in which the NSA said it had access to a cornucopia of customer information from American web firms. That stoked fears that the NSA is hoovering up information on a grand scale. But according to Mr Clapper, PRISM is not a data-gathering tool; it is an “internal government computer system” for accessing content that a court has already ordered companies to provide.

Stewart Baker, a former homeland-security official, compared PRISM to FTP (file transfer protocol)—a way to transfer files over a network. In America’s system of law-enforcement wiretapping, operators must provide access to the line when they are served with a court order to do so. Big internet companies may have simply designed a similar system for requests for content. There is no evidence yet that all the world’s Skype conversations, e-mails and Google docs are being sucked into NSA headquarters.

Hands off my metadata

The leaks have shaken the Obama administration, and drew swift criticism in Congress. Two Democratic senators, Ron Wyden and Mark Udall, who have warned about state intrusions into privacy for years, demanded that the government should reveal more about its data-gathering. Congressman Jim Sensenbrenner, a Republican and the author of the Patriot Act, the legal basis for the sweeping surveillance, called the activities “an abuse of that law”. A bipartisan group of eight senators has introduced legislation to force the government to make public its interpretation of the laws that seem to condone the surveillance. On June 11th the American Civil Liberties Union (ACLU), an advocacy group, sued the government over the surveillance programmes.

But both the metadata programme and PRISM appear to be legal. Both were approved by a FISA court, even if the breadth of surveillance of American citizens seems at odds with the privacy protections in FISA. Many criticise FISA courts for excessive deference to the government: in 2012 the government made 1,856 applications for electronic surveillance to FISA, and none was denied.

Benjamin Wittes of the Brookings Institution argues that the metadata programme rests on a “very aggressive reading” of section 215 of the Patriot Act. That section allows the FBI or others to apply to a FISA court for a warrant compelling businesses to turn over “any tangible things”, as long as they are “relevant to an authorised preliminary or full investigation to obtain foreign intelligence information not concerning a US person”. The authorities seem to believe that obtaining records of every telephone call made in America is either relevant to an investigation or an essential bulwark against international terrorism.

As for PRISM, on paper the protections against privacy abuse seem robust. The government does not “unilaterally obtain information” from company servers, nor does it target anyone for information-gathering without “an appropriate, and documented foreign-intelligence purpose to the acquisition”. It does not intentionally target any American citizen. The process is monitored by a FISA court, by Congress (through twice-yearly reports) and by independent inspectors-general. The information is subject to “minimisation procedures” designed to protect Americans unconnected to an investigation whose information is accidentally gathered.

Yet that does not reassure everyone. Just three months ago Mr Wyden asked Mr Clapper, who was testifying under oath before the Senate, whether the NSA collects “any type of data at all on millions or hundreds of millions of Americans”. Mr Clapper said it did not; thanks to Mr Snowden’s leak, everyone now knows that it does. As a candidate, Mr Obama applauded the courage of whistle-blowers (and rode into the White House on their disclosures); as president he has prosecuted them far more vigorously than his predecessors did. Then there is the data centre that the NSA is building near Salt Lake City, Utah. It is likely to cost at least $1.2 billion, and some expect its computers to provide five trillion gigabytes of storage. The agency did not build it to stand empty.

Still, the American public may not mind too much. A poll taken in the days after the metadata programme was exposed found that a majority of respondents (56%) believe that monitoring their phone calls is an “acceptable” way to investigate terrorism—though a substantial minority (41%) disagreed. (On the question of e-mail monitoring, the split went the other way: 52% said it was unacceptable while 45% approved.)

Separate from the question of trust is the subtler issue of data-mining’s efficacy. Bruce Schneier, a security expert, does not believe that a data-mining dragnet works. Terrorism, he says, “is a needle-in-a-haystack problem, and dumping more hay on the stack isn’t going to solve [it].” He advocates “going from person to person with targeted warrants”.

The government claims that information gathered has disrupted plots and stopped potential attacks, though the details remain classified. On June 12th the head of the NSA, Keith Alexander, said the surveillance programmes had helped prevent “dozens of terrorist events”—though they did not avert the Boston bombings.

Whatever the truth, the leaks are damaging America’s telecoms and internet firms, especially the companies whose cheerful logos appear at the top of the leaked slides describing PRISM. The bosses of Google and Facebook, Larry Page and Mark Zuckerberg, both strongly denied that the NSA has special access, and said they had not received orders to supply communications data, like the one issued to Verizon. Yet it is possible to speculate that they are simply unaware of some data-hoovering. According to a lawyer at a telecoms company and the retired boss of a large telecoms group operating in the United States, telecoms companies have long been required to employ technicians with security clearances who assist in government surveillance, but are not allowed to disclose their activities to their uncleared bosses. The same request may, perhaps, have been extended to web firms. Google, Facebook and Microsoft have requested permission to publish the numbers of national-security requests they receive, including FISA orders. So far there is no sign that the big web firms are losing users, and their share prices have not been hit. But the boss of a large European telecoms operator says he plans to market his services on the basis that they protect customer data from America’s prying eyes. American officials keep repeating that they hoover up very little content belonging to their own citizens. That is no comfort to the many millions of foreigners who visit American websites or whose traffic happens to pass along networks owned by American firms. On June 10th William Hague, Britain’s foreign minister, promised that his country’s spies would explain to a parliamentary committee how they may have benefited from America’s surveillance. British MPs fear that spooks are asking American agencies to fish out information on Britons they are forbidden to collect themselves—a claim Mr Hague said was “fanciful”.

Snoopers international

China dined out on the surveillance saga, with the state-run China Daily remarking that it was “certain to stain Washington’s overseas image”, and citing a Chinese academic who condemned “the unbridled power of the [American] government”. Peter Schaar, Germany’s data-protection chief, said the alleged scale of the spying was “monstrous”. Europe’s politicians have long fretted about FISA. In October a report prepared for the European Parliament warned that the law had granted American spies “heavy-calibre mass-surveillance firepower” and recommended that cloud-storage providers should be required to warn European users of the risks.

The weaker powers granted to European spooks are part of a pattern. In April the British government was forced to drop plans to make it easier for investigators to see whom troublemakers contact online. It aimed to require more phone and internet firms to store data about what their customers do, but would probably not have allowed authorities to download and store it daily, as in America. Critics mauled the proposal, but appreciated that it had been made public and debated. European privacy groups blame American lobbying after the September 11th attacks for the EU’s own limited data-retention law. Germany, Belgium and the Czech Republic have failed to ratify it fully; Austria and Ireland have asked a European court to rule on it.

But America’s energetic snooping is part of a broader global trend. Each year authorities in South Korea make more than 37m requests to see communications data stored about the country’s 50m people (police in Britain make about 500,000). New laws in Kenya let the government snoop on suspects indefinitely once an application is approved. India is considering a plan to route communications through government equipment, helping it to eavesdrop without alerting service providers. A report presented on June 4th by Frank La Rue, the UN’s special rapporteur on free expression, warned that broad interpretations of outdated laws were enabling sophisticated and invasive surveillance measures to flourish around the world. He called for governments to draw up new regulations that properly acknowledge the growing power of modern spying equipment.

Flourishing surveillance abroad may have a surprising impact back home. As more communications are stored on servers far from the citizens who created them, domestic intelligence services are increasingly trying to track activity overseas, says Carly Nyst of Privacy International, a lobby group. South Africa and Pakistan have both passed laws that give agencies more power to intercept communications between foreign citizens and to peruse material on servers abroad. Dutch spies want approval to hack into foreign machines and infect them with spyware. One risk is that security services from friendly countries will collaborate to evade domestic limits on their power, says Mr La Rue. Everyone is a foreigner to someone.

Driving all this is a dramatic expansion in the information people create, transmit and store. The fact that the scale and scope of surveillance has widened too should raise no eyebrows. That does not make the NSA’s work legitimate, but it makes it likely to continue—even if better protections emerge against abuse. When asked what the best outcome of the present furore would be, a former intelligence official said: “It’s that we have a debate and keep doing what we’re doing in better conscience.” That is only half the answer.