On November 5, 2009, an Army psychiatrist stationed at Fort Hood, Texas shot and killed 12 fellow soldiers and a civilian Defense Department employee while wounding 29 others. US Army Major Nidal Malik Hasan, the American-born son of two Palestinian immigrants, reportedly shouted “Allahu Akbar!”—“God is great!”—before launching his 10-minute shooting rampage at the Soldier Readiness Center. The shooting—the worst ever on an American military base—occurred as Hasan was facing imminent deployment to Afghanistan. A civilian police officer shot Hasan and placed him under arrest.

In the investigation that followed, the FBI and Defense Department investigators found that Hasan had been communicating with Anwar al-Aulaqi (sometimes spelled "al-Awlaki"), an American radical Islamic cleric living in Yemen. In the process of reviewing the evidence, investigators found that the FBI’s Joint Terrorism Task Forces in San Diego and Washington, DC had been aware of Hasan’s interactions with Aulaqi for over 11 months before the attack. Yet Hasan had never even been interviewed about his connection with the imam who would later be tied to “underwear bomber” Umar Farouk Abdulmutallab and to attempts to bomb US bound cargo planes with explosives packed in laser printer cartridges. (al-Aulaqi would later be killed by a US drone strike in Yemen.)

As federal officials looked into whether they had somehow missed leads that might have prevented the shooting, they found that the information technology at the heart of the FBI’s efforts to prevent terrorist attacks was fractured, overburdened, and running on aging and underpowered hardware.

Two weeks ago—coincidentally, just hours before another gunman would kill 12 and wound many more in Aurora, Colorado—an FBI independent commission led by former FBI director and federal judge William H. Webster filed its final report on the FBI’s performance leading up to the Fort Hood shooting. That report found no evidence the FBI's data would have set off alarms that Hasan was planning to kill fellow soldiers; he received no explicit instructions from al-Aulaqi and never mentioned his plans. But the report strongly implies that FBI IT systems and the bureau’s poor state of information sharing with other agencies played a role in the failure to take a harder look at Hasan.

Much has been made of government's power to survey citizens using technologies such as packet capture and deep packet inspection. Even used in a limited fashion, these technologies can gather massive amounts of data on the online behaviors of individuals, and when taken together they can create an electronic profile of people's lives. That potential—and concerns about its abuse—have driven privacy advocates to push for the repeal or alteration of laws such as the PATRIOT Act.

At the same time, US law enforcement and intelligence agencies have struggled over the past decade to take all of this information and put it to use. The poor search capabilities of the FBI's software, inadequate user training, and the fragmented nature of the organization's intelligence databases all meant there was no way for anyone involved in the investigation to have a complete picture of what was going on with Hasan.

While much has changed since November of 2009, the FBI’s intelligence analysis and sharing systems remain a work in progress at best—and there's no telling what other potential threats may have gone unnoticed.

Packet captured

Hasan first drew the interest of the Joint Terrorism Task Force in the FBI’s San Diego field office back in December of 2008, while he was a captain assigned to Walter Reed Army Medical Center and he attempted to make contact with Aulaqi via a message form on Aulaqi’s personal website.

The San Diego JTTF—a team made up of FBI agents and analysts, along with officers from the Defense Criminal Investigative Service (DCIS) and Navy Criminal Investigative Service (NCIS)— had been investigating Aulaqi since the late 1990s, when he was an increasingly radical imam at a San Diego mosque. As part of that investigation, the FBI monitored his electronic communications under a secret warrant, intercepting traffic to his personal webpage, his e-mails sent to a Yahoo webmail account, and his instant messages.

While the tools to do this could be primitive at the time, they did work. Back in 1997, the FBI began intercepting e-mails and other network traffic with the custom tool "Carnivore" (later given the bland name "DCS-1000" after copious criticism), a Windows-based packet sniffer that could capture specific types of communications as part of warranted surveillance. (In 2005, the FBI dropped its bespoke sniffer and switched to commercial deep packet inspection technology, which by then offered better features and performance.)

So when Hasan visited Aulaqi's website in 2008 and used its "Contact the Sheik" page to send Aulaqi a message, he identified himself with his name and his own AOL e-mail address. The FBI’s surveillance software scooped it up and noted Hasan's IP address, which resolved to Reston, Virginia:

Hasan’s message then entered an FBI database called the Data Warehousing Service (DWS), a database originally designed when the FBI was still using Carnivore. The FBI's Special Technologies and Applications section had designed DWS in 2001, but it was misnamed. DWS wasn’t a data warehouse per se but instead was designed as a transactional database for storing intercepted communications captured in criminal investigations—not for doing analysis on large data sets.

DWS wasn’t the only system used for handling surveillance data, though. In 2002, the Bureau had launched the Electronic Surveillance (ELSUR) Data Management System (EDMS), a separate system for handling foreign intelligence surveillance. The goal of EDMS was to help language analysts translate and annotate electronic content ranging from audio (collected from wiretaps and telephone monitoring) to intercepted e-mail and seized electronic media.

At the time that Hasan’s message ended up in the hands of the San Diego JTTF, the two systems were in turmoil. In February 2009, the systems were merged under a single user interface called DWS-EDMS as part of an effort to improve and consolidate surveillance data access—combining both criminal investigations and intelligence activities.

But with the Global War on Terror in full swing, DWS and EDMS had hit the wall—neither was really intended to handle the volume of surveillance that began rolling in to support counterterrorism investigations.