EXCERPT The Graveyard of Empires and Big Data The Pentagon's secret plan to crowdsource intelligence from Afghan civilians turned out to be brilliant — too brilliant.

The only tiki bar in eastern Afghanistan had an unusual payment program. A sign inside read simply, “If you supply data, you will get beer.” The idea was that anyone — or any foreigner, because Afghans were not allowed — could upload data on a one-terabyte hard drive kept at the bar, located in the Taj Mahal Guest House in Jalalabad. In exchange, they would get free beer courtesy of the Synergy Strike Force, the informal name of the American civilians who ran the establishment. Patrons could contribute any sort of data — maps, PowerPoint slides, videos, or photographs. They could also copy data from the drive. The “Beer for Data” program, as the exchange was called, was about merging data from humanitarian workers, private security contractors, the military, and anyone else willing to contribute. The Synergy Strike Force was not a military unit, a government division, or even a private company; it was the self-chosen name of the odd assortment of Westerners who worked — or in some cases volunteered — on the development projects run out of the guest house. The Synergy Strike Force’s Beer for Data exchange was a pure embodiment of the techno-utopian dream of free information and citizen empowerment that had emerged in recent years from the hacker community. Only no one would have guessed that this utopia was being created in the chaos of Afghanistan, let alone in Jalalabad, a city that had once been home to Osama bin Laden. Or even more unlikely, that the Synergy Strike Force would soon attract the attention of the Defense Advanced Research Projects Agency (DARPA). Founded in 1958 to help the United States win the space race, DARPA is best known for its role in futuristic technology, be it driverless cars or implantable brain chips. During the Vietnam War, DARPA had led a counterinsurgency research program that spanned the globe from Saigon to Tehran and employed hundreds of personnel, but the agency has been largely forgotten. However, DARPA’s interest in the potential of open-source information came at a critical juncture in Afghanistan. In January 2009, Barack Obama was sworn in as the 44th president of the United States, the war in Afghanistan was in its eighth year, and a resurgent Taliban, whose regime had rapidly collapsed after the American invasion in 2001, was challenging the central government’s tenuous authority in the provinces. Counterinsurgency, the doctrine that had been promoted in Vietnam decades earlier, was back in vogue. For DARPA, getting involved in Afghanistan represented a return to its Vietnam-era roots, after nearly three decades of staying out of war zones. In 2009, the agency launched an ambitious initiative in data mining. Eventually, DARPA brought two data-mining programs to Afghanistan: a highly secretive program meant to predict insurgent attacks based on “big data” science used by companies like Amazon to predict customers’ purchases; and a program based on the emerging science of social networks, attempting under the guise of humanitarian work to enlist an unwitting army of Afghan civilians to spy for the American military. And so DARPA’s first deployment to a war zone since Vietnam began with a group of well-intentioned hacktivists trading beer for data at Afghanistan’s only tiki bar. In Vietnam, DARPA had made the mistake of trying to “win the war with technology,” as one senior agency official in charge of the work had later recalled. The question once again was whether cutting-edge science and technology could do any better in Afghanistan. In 2009, a team from MIT won $40,000 cash prize from DARPA for being the first to locate 10 red weather balloons like these spread across the United States by incentivizing crowd-sourced reconnaissance on social media. (Photo credit: DARPA) In February 2009, Tony Tether, the longest-serving director of DARPA, was ordered to resign. Obama’s pick to replace him, announced in July 2009, was a public watershed for the agency. Regina Dugan became the agency’s first female director. As a program manager at DARPA in the 1990s, Dugan had earned a reputation for boldness — and some alleged recklessness — visiting minefields and combat zones to test bomb-detection technology. When she started making the rounds in the Washington Beltway as DARPA director, her choice of attire — short skirts, stiletto heels, and leather jackets — generated as much buzz as her credentials. A self-styled technology enthusiast, Dugan liked talking about theories of innovation. Her lecture style was often better suited to the world of TED, the popular technology conference, than to old-school military briefing rooms. “There is a time and a place for daydreaming, but it is not at DARPA,” she told Congress. “DARPA is not the place of dream-like musings or fantasies, not a place for self-indulging in wishes and hopes.” One of Dugan’s first moves was to hire Peter Lee, a prominent university computer scientist, to run a key DARPA office. The week before Lee was supposed to start, Dugan asked him to come to Washington immediately. She wanted him to meet with Secretary of Defense Robert Gates, who was scheduled to visit DARPA’s headquarters. As he drove from Pittsburgh to Washington, Lee grew nervous. He was an ivory tower academic, and now that he was about to become a senior DARPA official, he realized he had no concrete plans for the agency. His apprehension grew worse when he was pulled over for speeding. He had visions of multiple tickets and having his license suspended. The speeding incident, however, provided Lee with inspiration for the new office. He had recently learned about Trapster, a smartphone app that allows users to map and share information about speed traps using GPS. Trapster enabled a virtual army of tipsters to create a real-time map warning drivers of areas where the police may be lying in wait. Trapster finally provided Lee with an idea that he thought might interest the Pentagon chief. Instead of plotting speed traps, he imagined a Trapster-like application that could track potential bomb attacks in Afghanistan. Already, communities of people were collaborating online to track nuclear proliferation, spotting potential test sites in North Korea. Humanitarians were using crowdsourcing to monitor elections and respond to natural disasters. If crowdsourcing could plot speed traps and spot election fraud, perhaps it could be used in war zones. When Lee presented the idea, the defense secretary seemed to like it. So did Dugan, who encouraged Lee to pursue it. That was the beginning of Lee’s Transformation Convergence Technology Office, a name so awkward it seemed tailor made for hiding the secret programs it would develop. Dugan assigned to the office a group of military officers working at DARPA on a short-term basis, a sort of professional internship called the Service Chiefs Fellows Program. Normally, the officers toured military laboratories and did not do much substantive work, but Dugan wanted them to create a project with Lee. Soon, Lee and the fellows brainstormed a contest where participants would use social media in something resembling a national treasure hunt. The fellows proposed having teams compete to locate red weather balloons that DARPA would release across the United States. Regina Dugan speaks at The Disruptive Innovation Awards on April 25, 2014 in New York City. (Photo credit: SLAVEN VLASIC/Getty Images) The Network Challenge, as it was called, offered a $40,000 prize to the first team that could, on a specific day, identify the locations of the 10 red weather balloons placed across the United States. The idea was that teams would use social media to help locate the balloons. The contest would test the teams’ ability to leverage a network, figuring out how to motivate people to participate while weeding out possible fake sightings, and to do it quicker than other competitors. On Dec. 5, 2009, the day of the challenge, Lee’s biggest fear was that no team would identify all the balloons, undermining the point of the challenge. In the end, it took only nine hours for a team from MIT to win. They beat the competitors by using a sliding scale of financial incentives that rewarded not just those who spotted balloons, but those who recruited others who successfully spotted balloons. Alex “Sandy” Pentland, an MIT computer science professor who headed the winning team, called the task “trivial.” The MIT scientist had reason to be confident. Pentland had already established a reputation as one of the nation’s leading “big data” scientists. His specialty was sifting through data to predict patterns of human behavior, an area he called “social physics.” Pentland’s team had created a novel financial incentive system based on the assumption that people’s actions are dictated not purely by profit but by intangible benefits that come from exchanges that strengthen positions in social networks. “If you look at the models for incentives, or for management through the army, through companies, through economics, they’re all about individual incentives, and they ignore the social fabric. What I just said about red balloons was that it wasn’t about economics; it was about the social fabric,” Pentland said. Pentland theorized that someone’s position in the network — his or her social standing — was the primary motivator. In his calculation, people act to make their social fabric stronger, not necessarily to earn a bit of money. “I give you a favor. Maybe in the future you’ll give me a favor. That’s what drove this thing,” he said. “That’s a very different way of thinking about things. Instead of paying attention to individuals, you pay attention to relationships.” Taking what was learned from the Network Challenge and moving it into a formal DARPA program was the next step. “Someone made the observation and brought it to my attention that it might be possible for DARPA to get direct near-real-time access to several hundred data intelligence feeds from the theater in Afghanistan,” Lee recalled. “I thought that was very interesting. Most of the data feeds were classified only at the secret level. Some were even unclassified. One immediate question was, what might be possible if we did large-scale data mining on all of those feeds?” Lee began to contact all the experts he knew in data mining, including Werner Vogels, the chief technology officer at Amazon, who “provided a lot of framing for how we would approach this problem because it’s very similar to the kinds of data mining that Amazon does on its customers.” What Lee eventually formulated was a data-mining program based on the latest predictive analysis work being done in the commercial sector, but using military data from Afghanistan. “For example, we were trying to understand if the price of potatoes at local markets was correlated with subsequent Taliban activity, insurgent activity, in the same way that Amazon might want to know if certain kinds of click behaviors on Amazon.com would correlate to higher sales of clothing versus handbags versus computers,” Lee said. Big data were about to be enlisted in a program to predict whether a village in Afghanistan was being taken over by the Taliban or when insurgents might plan the next attack. More important, big data were going to take DARPA back to war.

An aerial reconnaissance drone is launched by members of the U.S. Army operating in eastern Afghanistan on Sept. 7, 2012. (Photo credit: TONY KARUMBA/AFP/GettyImages)

In February 2010, two months after Lee’s red balloon contest, Dugan did something that no DARPA director had done since the Vietnam War: She traveled to a war zone to see what the agency might be able to contribute. Gen. Michael Oates, the head of the Joint Improvised Explosive Device Defeat Organization, the Pentagon’s bomb-fighting agency, invited Dugan on a three-day tour of Afghanistan. Military personnel expressed surprise to see her. “You’re from DARPA,” she recalled their general reaction. “We call you when we have three- to five-year problems.”

When Dugan returned to Washington she assembled the office directors and their deputies and gave them a month to come up with ideas for technologies DARPA could contribute immediately to the war in Afghanistan. By April, DARPA had identified about a dozen projects that could have an immediate effect on the war, and then Dugan narrowed those down to a final list. The technologies ranged from a blast gauge that would go in soldiers’ helmets to detect exposure to possible blast waves from IEDs to an imaging sensor, called the High Altitude LIDAR Operations Experiment, which could be used to create three-dimensional maps of Afghanistan.

Dugan’s priority, however, was a new program based on Lee’s big data work, called Nexus 7, which would help predict insurgency in Afghanistan. In August, Dugan met with the chairman of the Joint Chiefs of Staff and laid out DARPA’s plan for Afghanistan. The data-mining project, her briefing noted, would “sequester [a] team of the Nation’s leading researchers in large scale computation techniques and social science.” She called Nexus 7, a data-mining program named after a humanoid robot in the movie Blade Runner, “the potentially big win.”

In the summer of 2010, Nexus 7 was launched with Randy Garrett, a former NSA official involved in big data, as its head. At the National Security Agency, Garrett had been working on creating a cloud that would allow analysts to search through real-time data as it was vacuumed up by intelligence agencies. This cloud would include “essentially every kind of data there is,” Garrett said, from the millions of calls a day around the world the NSA intercepts, to various forms of internet traffic, from emails to Skype calls. Afghanistan, after 10 years of war, was one of the NSA’s top targets for cell phone interception.

Garrett’s goal with Nexus 7 was to “actually build this big data aggregated environment, a cloud, and then see how you would use it,” according to a scientist who was involved in creating the DARPA program. Nexus 7 was a direct carryover of work started at the NSA, according to the scientist.

Other key members of the Nexus 7 team came from Sandy Pentland’s Human Dynamics Lab at MIT, drawing on the same ideas that drove his team’s win with the balloon contest and applying them to an entire society. Pentland described his contribution as informal, providing more of an intellectual framework than nuts-and-bolts work. At the core of Nexus 7 was Peter Lee, who created the program and ran it from his office. “Nexus 7 turned out to be a bunch of desks, laptops, and secure computers literally in the hallway outside of my office,” Lee said. “It was just a zoo.”

The direct relationship between the NSA and DARPA was one of the hallmarks of Nexus 7, but it was also the most problematic, because working with data from the NSA required navigating a maze of legal and statutory requirements that often prevent sharing and aggregating data among government agencies. As for why DARPA wanted the NSA data, L. Neale Cosby, a retired Army officer who consulted on the program, invoked the bank robber Willie Sutton: “Why rob a bank? Because that’s where the money is.” The NSA was the bank; it had all the data.

Nexus 7 was unabashedly interested in data of all kinds. It was particularly interested in using patterns of daily life, including the costs of transportation and exotic vegetables, to make predictions about insurgencies in Afghanistan. “We were really using the latest research in quasi-experimental design, in machine learning, and data mining literally on hundreds of intelligence feeds to make inferences about what would happen next,” Lee said.

The program was kept secret to avoid any controversy. DARPA workers one office down had little idea what was going on when the group of young computer scientists set up shop in the agency’s headquarters. “We had to have cover stories to tell people if various Beltway people came to visit me in my office and they were walking through this pandemonium,” Lee said. In budget documents, Nexus 7 was obliquely described as a program that combined data analysis and forecasting with social network analysis.

According to Dugan, Nexus 7 started making its “first discoveries”— or meaningful predictions — just 82 days into the operation. But the program soon met resistance. Gen. Stanley McChrystal, the head of the International Security Assistance Force in Afghanistan when Nexus 7 started, was interested in the data-driven work promoted by DARPA. But in 2010 he was forced to resign after a Rolling Stone magazine profile depicted him and his staff as mocking senior White House leaders. Gen. David Petraeus returned to Afghanistan to take over, but he was not enthusiastic about Nexus 7. A disastrous meeting between Petraeus and Dugan in Afghanistan almost brought it to a halt. DARPA’s proposal for algorithms did not sit well with a general who believed he wrote the book, metaphorically and literally, on counterinsurgency.

At that point, however, Nexus 7 had support from Gen. James Cartwright, the vice chairman of the Joint Chiefs of Staff and a technology enthusiast, who had given an official green light to the deployment of the program and its personnel. (Dugan said Cartwright’s response to Nexus 7’s first discoveries was “Go and go faster.”)

DARPA would eventually deploy more than 100 people across Afghanistan, working on Nexus 7 and other technology programs. Peter Lee, the creator of Nexus 7, was not among them. He abruptly left DARPA after less than a year to become the head of research at Microsoft. On the day he left for Seattle in September 2010 to start his new job, the Nexus 7 team, some members as young as their mid-20s, was departing for Afghanistan. “I should have been with them,” he said.

“It was the first operational deployment from DARPA since the Vietnam War,” Dugan later recounted. The program also became Dugan’s top priority as she shuttled back and forth to Afghanistan with Cartwright. In congressional testimony in 2011, Dugan did not use the Nexus 7 name, but simply described “a 90-day Skunk Works activity” that involved scientists and counterinsurgency experts working on “crowdsourcing and social-networking technologies.”

DARPA’s “army of technogeeks,” as Dugan affectionately referred to them, was young and had no military experience, and the culture shock soon became apparent. Military officials in Kabul were reluctant to share intelligence with computer scientists just out of graduate school, and the intelligence they did provide was not nice and neat, like consumer data. Once in Afghanistan, the analysts began to gather as much intelligence as they could: phone records from the NSA, radar feeds from the military, and intelligence reports. But much of the data that came into Nexus 7 were qualitative, rather than quantitative, which were not easy to plug into a computer program. Even when the data were quantitative, like from radar, they rarely covered the exact same place over time.

By late 2010, DARPA was touting Nexus 7’s successes within the Pentagon, but it was not clear what, if anything, it had accomplished. As members of the team worked on a base crunching numbers from military and intelligence data feeds, another team of contractors, the Synergy Strike Force, was working in the provinces of Afghanistan, swapping beer for data and using crowdsourcing techniques honed in the red balloon hunt.