This article was taken from the September 2012 issue of Wired magazine. Be the first to read Wired's articles in print before they're posted online, and get your hands on loads of additional content by subscribing online.

On the afternoon of February 15, 2011, Jaime Zapata, a 32-year-old special agent with US Immigration and Customs Enforcement, was shot dead by members of a drug cartel as he drove along a four-lane highway in Mexico.

Zapata's partner, Victor Ávila, 38, who survived the attack, later said that as many as 15 gunmen opened fire with automatic rifles, even though Zapata had identified himself as a diplomat and the armoured SUV in which the pair were riding had number plates identifying it as an official vehicle.


Back in Washington DC, senior US administration officials wanted revenge for what they saw as the deliberate killing of a federal agent and the attempted murder of another. Michele Leonhart, head of the US Drug Enforcement Administration (DEA) and a veteran of the drug wars, asked, "What can we do to make an immediate impact against the cartels, to send a message?" recalls Derek Maltz, the special agent in charge of the DEA's Special Operations Division. "We decided to do a law-enforcement strike," Maltz says.

To conduct what became known as Operation Fallen Hero, investigators turned to a little-known Silicon Valley software company called Palantir Technologies.

Read next Email is broken and this company has a plan to fix it Email is broken and this company has a plan to fix it

Palantir's expertise is in finding connections between people, places and events in large repositories of electronic data. Federal agents had a trove of reporting on the drug cartels, their members, their funding mechanisms and smuggling routes. They had dossiers, informants' reports, surveillance images, intercepted electronic communications, footage from drones. But investigators lacked a way to assemble and share all that intelligence with one another, and to quickly find leads buried in mountains of information.

Investigators with Zapata's agency bought Palantir's software, plugged it into their databases and used it to track down members of the cartel. The results were astonishing. Palantir helped to identify connections among key individuals and organisations. Officials reported that this kind of painstaking detective work -- reading reports, piecing together clues, drawing links between people -- would have taken months without technological assistance. With the help of Palantir, large amounts of data from disparate sources were analysed within days.


Law-enforcement officers across the US, Mexico and South America confiscated 467 kilograms of cocaine, 30 kilograms of methamphetamine and 282 weapons, and arrested 676 people -- including the cartel member suspected of killing Zapata.

Officials were so impressed with Palantir's software that seven months later they bought licences for 1,150 investigators and analysts across the country. The price, including training, was £4.8 million a year. The government chose not to seek a bid from some of Palantir's competitors because, officials said, analysts had already tried three products and each "failed to provide the necessary comprehensive solution on missions where our agents risk life and limb".

As far as Washington was concerned, only Palantir would do.

Read next Inside Cameo, the celebrity shoutout app hungry for fame Inside Cameo, the celebrity shoutout app hungry for fame

Such an endorsement would be remarkable if it were unique. But over the past three years, Palantir, whose office in Tysons Corner, Virginia, is just ten kilometres from the CIA's headquarters, has become a darling of the US law-enforcement and national-security establishment -- and its business in the UK is growing. US national security agencies now use Palantir for numerous variation on the challenge that bedevilled analysts in Operation Fallen Hero -- how to organise and catalogue intimidating amounts of data and then find meaningful insights that defy humans alone.


In the US, Palantir has sold its software to the CIA, the military's Special Command and the Marine Corps. The FBI, the Defense Intelligence Agency, the National Counterterrorism Center and the Department of Homeland Security are all customers. The director of the National Security Agency has said that Palantir's software could help the agency to "see" into cyberspace to defend against hackers and spies attempting to breach government computer networks. The Los Angeles Police Department uses Palantir. So does the New York Police Department, whose intelligence and counterterrorism unit rivals the sophistication of the FBI and the CIA.

The company also has a small London office in Covent Garden, and it plans to move to a larger, 930m2 space elsewhere in central London this year. Palantir's main UK customers are part of "Five Eyes", an international partnership comprising intelligence and security services from the UK, the US, Canada, Australia and New Zealand. The member countries share intelligence with one another that's particularly focused on cyber security and signals intelligence -- intercepted phone-calls, emails and other electronic messages. The British "Eye" is the Secret Intelligence Service (MI6) and the Government Communications Headquarters (GCHQ), the signals-intelligence agency.

Palantir is extraordinarily tight-lipped about its UK customers, much more so than about its American ones. (British secrecy laws are much stricter and more readily enforced than in the US.) According to sources, Palantir has few or no customers in domestic law enforcement, such as Scotland Yard. However, because British agencies routinely share foreign and domestic information with one another, there are domestic security organisations in the UK receiving the benefit of Palantir without having actually used the software themselves.

Asher Sinensky, who's in charge of Palantir's UK business, says the British government will be using Palantir during the Olympics. "There will be half a million more people in London every day during the games," he says. "The transportation infrastructure will be swept in ways that it hasn't before." He says there's also "a lot of concern" among British officials about defending borders and controlling who comes in and out of the country. "These outsiders, we don't have the same record of their interactions with people as we do with [British] citizens and others we've been tracking."

Read next The pandemic changed how everyone uses Slack The pandemic changed how everyone uses Slack

Palantir was launched in 2004 by Alex Karp, a financial adviser with a PhD in neoclassical social theory and no experience running a technology company, and Peter Thiel, a billionaire venture capitalist who'd helped start PayPal and was an early investor in Facebook. Karp, a self-described progressive, knew Thiel, a prominent libertarian, from their days at Stanford Law School. After 9/11, Karp had reconnected with Thiel, who had the idea that Silicon Valley should do something to improve national security and secure civil liberties. Karp, who is tall and thin with a bushy crop of hair, has none of the swagger of so many Silicon Valley entrepreneurs. He's a restless academic, more at home in a seminar than a pitch meeting. During an interview at his office in Tysons Corner, he stands up, paces the room for a few moments, sits down, gets up again and starts sketching out diagrams and graphs on a white board to explain how Palantir works.

They and three other cofounders wanted to use PayPal's fraud-detection technology as the model for a new counterterrorism software, which would be used by analysts to crunch huge amounts of data.

The parallels between moving money and fighting al-Qaeda might not be immediately obvious, but the ascendancy of PayPal, which was founded in 1998, was largely because of its ability to prevent criminals from stealing its customers' money. Engineers designed an algorithm that let human fraud experts -- many of whom were former law-enforcement officers -- quickly sift through transaction data, look into the transaction network and map out connections among suspected criminals. That approach made PayPal the world's most trusted system for online payments. In 2002, the company was acquired by eBay for £964 million. "The bread and butter of PayPal was to look at a transaction and to know if it came from a bad IP address," says Bob McGrew, Palantir's director of engineering. That same approach became the heart of Palantir, which Karp says is an "attribution" software, meaning it's used for finding the people behind pieces of data.

Palantir wanted to beat terrorists the way PayPal beat Russian criminal gangs. Thiel, who's something of a policy dilettante, also thought the idea could make a lot of money.

Palantir is considered one of the most valuable startups in the US -- VC firm Globespan Capital Partners estimates Palantir's market valuation to be as high as £2.5 billion. A top executive at JPMorgan Chase says it's poised to do for information locked inside organisations what Google did for information on the web.

Read next To survive the pandemic’s doom, startups need to work together To survive the pandemic’s doom, startups need to work together

Before Karp and Thiel ever signed their first contract, they sought counsel from some of the most important national-security experts in Washington. Not long after the company was officially formed, in 2004, the two cofounders met with John Poindexter, a former national-security adviser to Ronald Reagan, at the home of Richard Perle, who was chairman of the Defense Policy Board, a group of influential Pentagon advisers, in 2001. "I told them I thought they had an interesting idea," says Poindexter, who, from 2002 to 2003, ran a Defense Department initiative called Total Information Awareness, which bore striking similarities to Palantir's approach to data analysis. The programme was shut down following outcries from privacy activists -- TIA proposed to mine not just government intelligence databases, but privately held records such as credit-card transactions, email and phone records.

Experts such as Poindexter helped Palantir open doors. In a short time, the company has assembled a legion of advocates from the most influential strata of government. Karp counts former CIA director George Tenet as a friend; he says the same about Tenet's employer, Herb Allen, who runs the enigmatic investment bank Allen

& Co, a Palantir investor. And another top adviser, Bryan Cunningham, was a CIA intelligence officer and a senior staffer to former secretary of state Condoleezza Rice.

Early prospective investors were sceptical of Palantir. Venture-capital firms were looking for the next Facebook or Google -- most presumed it would come in the form of a consumer technology, probably a new social-media site. No one wanted to back an expensive software platform for large organisations. The Palantir founders met several venture-capital firms, and all turned them down. "We believed it would work," Karp says. "No one else did."

Potential investors were also leery of Washington. "The government was unpopular in Silicon Valley,"

Karp says. At the end of one failed pitch meeting, an investor who'd turned him down said there was a group he should talk to "that does this kind of thing". The group was In-Q-Tel, the venture-capital arm of the CIA that was set up in 1999 to bypass the cumbersome government procurement process and to fund technologies that might be useful to intelligence agencies.

Read next Uber’s London rivals circle as the company faces its judgement day Uber’s London rivals circle as the company faces its judgement day

Palantir got a meeting with In-Q-Tel's CEO, Gilman Louie, a former computer-game designer.

Stephen Cohen, one of Palantir's founders, was 22 at the time, but had been writing code since he was a teenager. He spent the next eight weeks with another cofounder, hammering out a version of Palantir they could take to the meeting. They worked -- and slept -- in an office Thiel had used when he founded PayPal.

In-Q-Tel invested a relatively insignificant amount of money -- about £1.3 million, a small chunk of the nearly £26 million that Karp says Palantir's investors spent before the company saw its first dollar in revenue. But it led to a meeting with another interested backer, the venture wing of Reed Elsevier, the publishing and information conglomerate. One of its partners saw Karp give a presentation at an In-Q-Tel meeting, and was so impressed that he invested a few million dollars.

Crucially, In-Q-Tel put Palantir's founders in the room with frontline US intelligence analysts, the people they hoped would use their product. The analysts gave Palantir the software equivalent of a test drive. "They'd say, 'I love that, I hate that,'" Karp explains.

The founders spent the next three years flying to Washington, taking notes and then returning to Palo Alto to tweak the software. Cohen says he was getting "most of my calories from Red Bull".

Read next We need to talk about investors’ problem with vaginas We need to talk about investors’ problem with vaginas

Karp estimates that he and Cohen had more than 300 meetings with likely users, people far down the government hierarchy. The Silicon Valley techies found themselves deep in an unfamiliar culture: some people introduced themselves only by their first names and refused to say where they worked in the government.

While shuttling between the coasts, Palantir's founders discovered that intelligence analysts wanted a way to search their own databases and to know what their colleagues in other agencies had available. But just as important, agencies needed to restrict access, so that only those with the proper security clearances could, for instance, look at the video of a drone attack or read a classified interrogation summary. Palantir developed a method for indexing information so the system would match up a particular data point with the user's security clearances. If he didn't have the authority to read it, the information was unavailable. This technique had the added benefit of creating an audit trail of what the Palantir users were reading, whether they'd handled the information properly, and whether they'd modified it in any way.

It's difficult to overstate the importance of this security regime. Without such nuanced controls -- down to the level of a single person or one nugget of intelligence -- the kinds of collaboration necessary to prevent terrorist attacks just won't happen. An audit trail like this also lets analysts check their own prior judgments to see if there was a flaw in their logic.

Palantir also developed a way to organise data that spoke to a great yearning in the spy world: the need to quickly assimilate new information into an unfolding narrative. Once data is put into Palantir, the software uses a model called "dynamic ontology" to show how names, places and events relate to one another. For instance, imagine a suspected terrorist who's being tracked by MI6 makes contact with someone whom the service hasn't seen yet. That person's name goes into Palantir's system, and the entire dossier on the original target changes to account for any previously unseen connections between the two. The network of relationships between the target and anyone the new person knows can be seen as well. Every time an analyst adds a new piece of data, the picture changes automatically. And this new picture can be shared with other analysts using the software.

Before it had even landed a customer, Palantir was given a rare audition with the agency that knew better than any other about the dangers of misguided analysis.

Read next How London’s Silicon Roundabout dream turned into a nightmare How London’s Silicon Roundabout dream turned into a nightmare

According to a government official, the CIA allowed Palantir to set up its software in the agency's counterterrorism centre, the hub of its global campaign to track down terrorists. The official was astounded that a little-known company from Silicon Valley was allowed to place its equipment on a network that pulses with the most highly classified government intelligence. Palantir didn't disappoint. The official says the company worked for several months without pay and convinced the CIA that its technology could do what it claimed. Yet heading into mid-2008, Palantir still hadn't won a government contract or earned any revenue. The small number of investors the company had managed to attract were getting impatient. Karp had already delayed the release of the software by a year because he felt it wasn't ready. "I'm not motivated by money," Karp says. "It's not what gets me out of bed in the morning."

Patience paid off. At yet another analyst meeting, Cohen showed a group of more senior government officials what enhancements the engineering team had made. Out of the corner of his eye, Cohen claims, he saw two stoic men in grey-flannel suits turn to each other and, without speaking a word, give each other high fives. "At that moment, it was really clear to me: we're going to have a very, very valuable business," Cohen says. <span class="s1">Back in Palo Alto, Palantir moved into a 650m2office.

The founders were about to sign their first contract with a government agency. Karp won't say whom, and although sources' accounts conflict, it was likely that the client was the CIA or a Defense Department group set up to fight improvised explosive devices and bomb makers. Palantir began recruiting top students from Stanford and other elite computer-science schools.

It offered the typical tech-employee perks, including free dry cleaning and three meals a day. On the company intranet, Karp sent out motivational videos that instructed employees how to talk about Palantir with customers. Employees nicknamed it KarpTube. "The office was like a fraternity for very smart people," says Tim Su, who worked for two years as a software engineer.

The cultures of Washington and Palo Alto found enough common ground that Palantir's initial contract turned into a second and then a third. Palantir's early business grew based on word of mouth -- Karp calls it "a rumour mill of people who'd worked with the product". They called friends in other agencies and urged them to buy it.

Read next WeWork triggers second wave of mass layoffs in the UK WeWork triggers second wave of mass layoffs in the UK

Beyond government, Palantir's business now includes prominent banks and financial institutions -- such as JPMorgan Chase and the hedge-fund manager Bridgewater -- and is moving into healthcare, helping to spot fraud and inefficient spending.

Media reports estimate that the company earned just over $250 million in 2011.

Palantir's corporate ethos sometimes feels more connected to fantasy than reality. In public remarks, several employees have said their job is to assist the people "who are out saving The Shire". It's an allusion to JRR Tolkien's

The Lord of the Rings. The Shire is the home of the hobbits, who band together with their elf, dwarf and human compatriots to save the world from the armies of Sauron, the master of evil. The company is permeated with Tolkien references.

Its Palo Alto office is known as The Shire, and the office in Virginia is Rivendell -- the home of the elves. The London office is Grey Havens, an elvish port. The firm's name is also from Tolkien -- a palantir is a magical stone that lets its holder see across great distances. The stone is also used by Sauron to conduct surveillance as he wages war. It's an apt allusion: just like Tolkien's palantir, the ends to which the Palantir software is used depends on who's manipulating it. And that includes not just Palantir's clients but its own employees, some of whom have embraced their powerful status as an arm of the surveillance state.

Read next Inside the plot to topple Europe’s last dictator Inside the plot to topple Europe’s last dictator

In the autumn of 2010, Palantir employees partnered with an ex-US Navy intelligence analyst named Aaron Barr, the new CEO of HBGary Federal, a company specialising in identifying computer viruses, on an ill-conceived project that carried the promise of big money -- but also a lot of risk. Palantir and HBGary Federal teamed up with a third intelligence contractor, Berico Technologies, to provide information on groups and individuals deemed hostile to the US Chamber of Commerce. The law firm Hunton & Williams first approached Palantir about the work, which was to include reconnaissance of various websites and social media in order to build dossiers on the Chamber's opponents. According to a proposal, Palantir would "serve as the foundation for all of the data collection, integration, analysis, and production efforts".

For its work, Palantir asked to be paid $1.1 million. Anticipating that its client might balk at such a price, Matthew Steckman, a Palantir employee in the Washington office, wrote an email to his teammates urging them to emphasise, "We are the best money can buy! Damn it feels good to be a gangsta."

A few days afterwards, the law firm asked whether it could offer a proposal for another job, this time targeting the anti-secrecy group WikiLeaks, which at the time was threatening to release internal records from Bank of America. As Steckman explained to his team, Bank of America wanted to sue WikiLeaks and enjoin it from releasing the information.

The US Justice Department, which had been looking for a way to prosecute WikiLeaks's founder, Julian Assange, called Bank of America's attorneys and told them to get in touch with Hunton &

Williams. "Apparently, if they can show that WikiLeaks is hosting data in certain countries, it will make prosecution easier," Steckman wrote. Barr said that the Palantir team should target WikiLeaks's "global following and volunteer staff" as well as people donating money to the group: "[We] also need to get people to understand that if they support the organisation we will come after them. Transaction records are easily identifiable." He said they should submit fake documents to WikiLeaks and try to foment distrust among different camps of supporters. Barr also wanted to launch "cyberattacks" on a server WikiLeaks used in Sweden in order to "get data" about people who were anonymously submitting information.

But, in February 2011, an article appeared in the Financial Times quoting Aaron Barr, who bragged that he'd been able to penetrate the inner ranks of another hacker-activist group, Anonymous. The group retaliated by breaking into Barr's email account and publishing several years' worth of his correspondence, which included the proposals relating to Palantir.

Read next As WeWorks lie empty, coworking spaces face their day of reckoning As WeWorks lie empty, coworking spaces face their day of reckoning

For Palantir, a company founded on the idea that technology should protect personal freedoms, it was a humiliating revelation: the company risked looking like a cybermercenary. That image ran counter to the core values of the company. Karp was apparently unaware of what his subordinates had been doing. Palantir has what Karp calls a "flat hierarchy": employees are encouraged to act like entrepreneurs and not to seek approval for every decision they make. Karp says this structure is essential to Palantir's success: "No company in the Bay Area is disruptive with multiple layers of hierarchy."

Palantir placed Steckman on leave, pending a review of his action. Barr resigned from HBGary Federal.

Karp ended all contacts with HBGary and issued a statement apologising to "progressive organisations... for any involvement that we may have had in these matters".

Karp says that Palantir hired the law firm Boies, Schiller &

Flexner to investigate the company's role. It recommended that Palantir should keep Steckman as an employee, which Karp says that he did. But the incident begged a question: what's to stop a government intelligence agency from turning off Palantir's privacy-protection features and using the software for illicit purposes? Karp insists that the controls are "very hard to circumvent" and that it would take a "world-class software team" to do it.

Read next Coronavirus might spell the end of tech hubs. That’s a good thing Coronavirus might spell the end of tech hubs. That’s a good thing

He doesn't cite an example, but one agency Palantir employees claim hold it in high regard -- the US National Security Agency (NSA) -- would have both the skill and the motivation to modify Palantir for its own ends. The agency employs the largest and most skilled cadre of software experts in the US government. And for more than four years after 9/11, it conducted a secret campaign of electronic surveillance against US citizens that bypassed federal courts. The NSA also took over many of John Poindexter's Total Information Awareness programmes after they were officially shut down, but it rejected one: building privacy-enhancing technology into computer software.

The cultural distinctions between employees in Palo Alto and Virginia have become more pronounced.

Palantir is neither of the Valley nor entirely of Washington. It's a kind of techno-military hybrid. In Virginia, there are predictable trappings of a startup -- a pool table, Razor scooters, a well-stocked kitchen -- but there are also employees wearing desert boots and customers in camouflage fatigues.

Palantir, like so many government contractors, has installed a Washington-style revolving door. Most of its "embedded analysts" -- employees who work on high-priority national-security threats -- are former users from the military and intelligence community. A job description on Palantir's website describes the ideal embedded analyst this way: "Although you loathe the bureaucracy, you have a deeply held belief that a revolution in intelligence affairs is not only possible, it is imminent. Help us craft that revolution."

Has Palantir created more of a cult than a culture? Karp -- whom employees call Dr Karp -- insists they've built "a culture that's not based on money". Palantir caps all salaries at $127,000 (£82,000). Employees are compensated with bonuses and equity stakes, but most of the engineers could make far larger salaries if they defected to Facebook or Google. If someone goes to work for Palantir, it's probably because he or she believes in Palantir and its mission. Despite Palantir's obvious trajectory toward a public offering of stock in the near future, he says, "I don't want an IPO. The minute you have it, people wake up and ask,

'How rich am I?'"

Today, some current and former US government officials say Palantir's star has dimmed in the intelligence community. They complain that the software has a hard time analysing very large databases and that it takes a lot of time on the front end to arrange information in a format Palantir can use.

Still, Palantir has managed to build what it claims, and, despite its shortcomings, it is a technology that has made possible several significant contributions to solving some of the US's most important national security challenges. "The contradiction that we wanted to remove was between civil liberties and fighting terrorism," Karp said at a recent Palantir conference. "Do we really want to live in a world where everyone sees everything without any kind of permissions? Solving this problem... that's a really cool idea."

Security and liberty are competitors now. That's not a natural condition; it's a product of our time, of the decisions that we have all made -- or failed to make -- over the past decade. Could a piece of software allay that uneasy tension? Perhaps. But as any good student of Tolkien knows, whether a palantir is used for good or for evil depends on who's holding the stone.

Shane Harris is the author of The Watchers: The Rise of America's Surveillance State (Penguin Books)

Five ways in which Palantir has helped its clients to crunch data

1 --

Intelligence

In 2011, Palantir provided a software platform to investigators at the Center for Public Integrity, in conjunction with Georgetown University, to support a three-and-a-half-year project identifying the kidnappers who killed US journalist Daniel Pearl in 2002. Palantir's software platform analysed the investigation data and helped identify the links between the key individuals involved in the crime.

2 -- Defence

Palantir provides a large selection of tools for government defence analysts and soldiers to improve their access to battlefield intelligence. One example is their "human terrain" platform, which stores photos, notes and summaries of key political, ethnic and economic leaders and trends in a war zone, so that shuffled units don't have to constantly relearn the information.

3 -- Disaster relief

After the 2010 Haiti earthquake, Palantir developed a free, open database, including names and locations of collapsed buildings, Internally Displaced People camps and SMS messages within specific administrative sectors. Also, a Palantir analyst spent a week examining federal government spending associated with Hurricane Katrina, using geospatial, temporal and relational analysis.

4 -- Finance

Using data from the Regulatory Data Corporation, Palantir has helped US banks to crunch the numbers to spot patterns of fraudulent mortgage loans perpetrated by organised-crime rings and street gangs. Their platform can also sniff out individual fake transactions and identify corporate and human networks operating under the radar.


5 --

Healthcare

The Palantir Health platform is used by US government health authorities, such as Medicare and Medicaid, to analyse electronic health records. The software enables Palantir to reconstruct a history of interactions between patients, doctors, insurers and others involved, in order to find subtle but useful patterns.

Madhumita Venkataramanan