While most of the N.S.A. was reeling on September 11th, inside [the Signals Intelligence Automation Research Center] the horror unfolded “almost like an ‘I-told-you-so’ moment,” according to J. Kirk Wiebe, an intelligence analyst who worked there. “We knew we weren’t keeping up.” SARC was led by a crypto-mathematician named Bill Binney, whom Wiebe describes as “one of the best analysts in history.” Binney and a team of some twenty others believed that they had pinpointed the N.S.A.’s biggest problem—data overload—and then solved it … .

As Binney imagined it, ThinThread would correlate data from financial transactions, travel records, Web searches, G.P.S. equipment, and any other “attributes” that an analyst might find useful in pinpointing “the bad guys.” By 2000, Binney, using fibre optics, had set up a computer network that could chart relationships among people in real time… . ThinThread processed information as it was collected—discarding useless information on the spot and avoiding the overload problem that plagued centralized systems. Binney says, “The beauty of it is that it was open-ended, so it could keep expanding.”

Pilot tests of ThinThread proved almost too successful, according to a former intelligence expert who analyzed it. “It was nearly perfect,” the official says. “But it processed such a large amount of data that it picked up more Americans than the other systems.” Though ThinThread was intended to intercept foreign communications, it continued documenting signals when a trail crossed into the U.S. This was a big problem: federal law forbade the monitoring of domestic communications without a court warrant. And a warrant couldn’t be issued without probable cause and a known suspect. In order to comply with the law, Binney installed privacy controls and added an “anonymizing feature,” so that all American communications would be encrypted until a warrant was issued. The system would indicate when a pattern looked suspicious enough to justify a warrant.

But this was before 9/11, and the N.S.A.’s lawyers deemed ThinThread too invasive of Americans’ privacy. In addition, concerns were raised about whether the system would function on a huge scale, although preliminary tests had suggested that it would. In the fall of 2000, [General Michael Hayden, the director of the N.S.A.,] decided not to use ThinThread, largely because of his legal advisers’ concerns… .

The N.S.A.’s failure to stop the 9/11 plot infuriated Binney: he believed that ThinThread had been ready to deploy nine months earlier. Working with N.S.A. counterterrorism experts, he had planned to set up his system at sites where foreign terrorism was prevalent, including Afghanistan and Pakistan. “Those bits of conversations they found too late?” Binney said. “That would have never happened. I had it managed in a way that would send out automatic alerts. It would have been, Bang!”