“THE COMPUTER WAS born to spy,” says Gordon Corera, who covers intelligence for the BBC, Britain’s national broadcaster. The earliest computers, including Colossus and SEAC, were used by signals intelligence (known as SIGINT) in Britain and America to help break codes. But computers also happen to have become supremely good at storing information. Searching a database is a lot easier than searching shelves of files like those compiled by the East German secret police, the Stasi—which stretched for 100km.

The job used to be to discover what a hostile country was up to by attaching crocodile clips to telephone lines emerging from its embassy, intercepting communications, collecting data and decrypting them. It was an industrial process. Breaking code was laborious, but once you had succeeded, the results endured. “Twenty years ago we had a stable target, a stately pace of new technology and point-to-point communications,” says a senior intelligence officer. Cryptography evolved slowly, so “when you cracked a code it could last from ten to 30 years.”

The internet changed everything. Roughly $3.4trn a year is being invested in networked computers, phones, infrastructure and software. The pace is set by businesses, not spooks. Individual packets of data no longer travel on a dedicated phone line but take the route that is most convenient at that instant, blurring the distinction between foreign and domestic communications. Signal intelligence used to be hard to get hold of. Today it gushes in torrents. The trick is to make sense of it.

Civil-liberties groups rightly claim that this new world presents untold opportunities for surveillance. This has been especially true for the NSA and GCHQ. Most of the traffic has passed through America, which contains much of the infrastructure of the internet, and much of the rest passed through Britain, even if it originated and terminated elsewhere. Everyone uses the same hardware and software, so if you can break one device, you can break similar devices anywhere.

Knowing who communicates with whom is almost as revealing as what they say. In a technique called contact chaining, agencies use “seed” information—the telephone number or e-mail address of a known threat—as a “selector” to trace his contacts and his contacts’ contacts. A burst of activity may signal an attack. In 2015 contact chaining let GCHQ identify a new terrorist cell that the police broke up hours before it struck.

You are never alone with a phone

Mobile phones show where they are. According to Bruce Schneier, a cyber-security expert, the NSA uses this information to find out when people’s paths cross suspiciously often, which could indicate that they are meeting, even if they never speak on the line. The NSA traces American intelligence officers overseas and looks for phones that remain near them, possibly because they are being tailed. Location data can identify the owner of a disposable phone, known as a “burner”, because it travels around with a known phone.

The technical possibilities for obtaining information are now endless. Because photographs embed location data, they provide a log of where people have been. Touch ID is proof that someone is in a particular place at a particular time. Software can recognise faces, gaits and vehicles’ number plates. Commercially available devices can mimic mobile-phone base stations and intercept calls; more advanced models can alter texts, block calls or insert malware. In 2014 researchers reconstructed an audio signal from behind glass by measuring how sound waves were bouncing off a crisp packet. The plethora of wired devices in offices and houses, from smart meters to voice-activated controllers to the yet-to-be-useful intelligent refrigerator, all provide an “attack surface” for hacking—including by intelligence agencies. Britain’s government has banned the Apple Watch from cabinet meetings, fearing that it might be vulnerable to Russian hackers.

The agencies can also make use of the billows of “data exhaust” that people leave behind them as they go—including financial transactions, posts on social media and travel records. Some of this is open-source intelligence (known as OSINT), which the former head of the Bin Laden unit of the CIA has said provides “90% of what you need to know”. Private data can be obtained by warrant. Data sets are especially powerful in combination. Facial-recognition software linked to criminal records, say, could alert the authorities to a drug deal.

The agencies not only do more, they also spend less. According to Mr Schneier, to deploy agents on a tail costs $175,000 a month because it takes a lot of manpower. To put a GPS receiver in someone’s car takes $150 a month. But to tag a target’s mobile phone, with the help of a phone company, costs only $30 a month. And whereas paper records soon become unmanageable, electronic storage is so cheap that the agencies can afford to hang on to a lot of data that may one day come in useful.

Vague, very vague

But not everything is going the agencies’ way. Indeed, many SIGINTers believe that their golden age is already behind them. As the network expands, more capacity is being added outside America. By 2014, according to Mr Corera, the proportion of international data passing through American and British fibres had nearly halved from its peak. And the agencies have the capacity to examine only a small fraction of what is available. The NSA touches 1.6% of data travelling over the internet and selects 0.025% for review. Its analysts see just 0.00004%.

Data are also becoming harder to trace. Some protocols split a message in such a way that it passes over different networks—a phone connection and Wi-Fi, say. Others allocate IP addresses dynamically, so that they may change many times in a single session, or they share one between many users, which complicates identification. Still others take computing closer to the user, which means that messages bypass the core network.

The internet has many channels and communications apps, each with its own protocol. Work on new tools is 20-30% of the spooks’ job. Even so, there are too many apps for the agencies to reverse-engineer, so they have to choose. An easy protocol might take a day to work around. A difficult one might take months. A routine upgrade of an app can mean having to start from scratch. And some means of communication are intrinsically hard to break. Messages worth collecting that are contained in apps like FaceTime and Skype are hard to tell apart from entertainment in Netflix and YouTube when they pass through networks. Jihadists can contact each other through online gaming chat rooms. Steganography hides messages inside images.

Encryption is becoming standard. If a message is sent via an app provider like Telegram or WhatsApp, the identity of the receiver might be encrypted, too. In principle modern encryption is uncrackable. Unless someone can build a quantum computer, which could search for multiple solutions simultaneously, working through the permutations would take a chunk out of the rest of history.

To get in, therefore, analysts often depend on human error. But the targets are becoming more sophisticated. The New York Times has reported that Abdelhamid Abaaoud, who directed a wave of bloody attacks in Paris in November last year, ordered a soldier to ring a mobile phone on Syria’s northern border so that his call would pass through a lightly monitored Turkish network.

The result, case officers say, is that tracking jihadists takes increasing effort and skill. A few years ago one officer might watch several jihadist targets; today you need to throw a lot more manpower at the task. Too many jihadists have travelled to Syria for GCHQ to monitor them all. The intelligence services catch glimpses of what is going on, but not the full picture. “With encryption,” says a British officer, “maybe you see a bit of content, a bit of the puzzle.”

Some Western intelligence chiefs have tried to curb encryption, or argued that at least they should be given a set of secret keys. That would be impractical and unwise. Impractical, because watertight encryption programmes will then be written outside America and Europe, and there is little the authorities can do to stop it. Unwise, because the intelligence services are not the only ones prowling the web. Organised criminals and fraudsters would like nothing better than weaker encryption.

A better way to cope with the difficulties of intercepting traffic is to hack into machines sitting at the end of the communications chain. Once in, the agencies can look at a message before it has been encrypted, split into packets and scattered across the network. Again, though, that poses a dilemma, because governments are responsible for cyber defence as well as cyber offence. To gain entrance to a machine, hackers use flaws in software. The most prized of these are undisclosed and called zero-day vulnerabilities (because software engineers have zero days to write a patch). Stuxnet, a computer worm written by the Americans and the Israelis that attacked centrifuges in Iran’s uranium-enrichment programme, exploited five zero-day flaws.

There is a market in such tools. When Hacking Team, an Italian cyber-company, was itself hacked in 2015, the world learnt that zero-day vulnerabilities were for sale. According to Wired, a magazine, the price started at hundreds of thousands of dollars. Among the buyers were governments and criminals. In their role as defenders, the NSA and GCHQ should be revealing software faults so that companies can write patches. In their role as attackers, they need some in reserve.

When machines are so powerful, where do people fit in? Certainly, signal intelligence is relatively cheap, versatile and safer than running human agents. Yet human spies still play a vital complementary role. One task is to furnish seed information that can serve as selectors for tracing contacts. Another is to gain access to computers that are well-defended or “air-gapped” from the internet. Most valuable of all is the human ability to bring judgment and context.

People also provide oversight. There was a time when the constraints on the agencies were technical and budgetary, because codes were hard to break and agents costly to deploy. In an era of cheap technology, it is difficult to know precisely what the technology will be able to accomplish. The constraints on the intelligence services’ conduct must therefore be legal—and robust.

Edward Snowden and others have suggested that the agencies are unwilling to live within the rules. But is that criticism deserved? In the anxious times after the attack on America on September 11th 2001, how far did the CIA and the NSA really go?