Introduction: The following story was the first article I wrote during my short-lived career with First Look Media, the media outlet founded by eBay billionaire Pierre Omidyar, which attracted name-brand journalists such as Glenn Greenwald, Laura Poitras and Jeremy Scahill. As I have recounted elsewhere, it was not a good fit. One of the main differences in point of view was that there was no chance the editors there would ever be supportive of anything the CIA or NSA might ever do. I myself am hugely critical of the U.S. government and intelligence agencies, and especially the CIA’s role undermining and overthrowing popular leftist governments for the past 65-odd years. But I do see the need to protect this country from lunatics like Al Qaeda and ISIS, and that requires robust intelligence gathering operations.

That’s why I object to a certain strain common among watchdog journalist types—a juvenile, reflexive anti-Americanism—something like the extreme leftism that Vladimir Lenin once referred to as an “infantile disorder.” At one point Mr. Scahill even worried that a story by an Intercept staff reporter that was critical of Al Qaeda might jeopardize his relationship with an Al Qaeda spokesman who was a good “source” of information for him.

In any case, this is one of many stories I wrote for First Look that never saw the light of day.

[Note: I very lightly edited this story, however, I have not updated it. When reading it, recall that it was originally written in March of 2014.]

How can the United States spend upwards of $50 billion a year on intelligence, and still be surprised by something like the Russian invasion of Crimea?

Anonymous intelligence officials are trying to blame Edward Snowden—as if Vladimir Putin had no idea until this summer that the U.S. was trying to eavesdrop on him.

‘Drones don’t help you understand tribal politics in Pakistan, but they make money for a lot of people in Northern Virginia.’

The documents Snowden has made available to reporters didn’t cause this latest intelligence failure—but they can help us understand it a bit better. Because, in addition to telling us a lot about the incredible breadth of the intelligence community’s electronic surveillance program, the documents have also shown us that the U.S. government is well aware of its significant shortcomings.

A Washington Post story based on a top-secret summary pegged last year’s “black budget” at a staggering $52.6 billion, of which a sizeable chunk was devoted to the “particularly indispensable” task of signals intelligence, or SIGINT. And yet, despite having “resources and a reach beyond those of any adversary,” the budget narrative acknowledged that America’s vast espionage empire was “unable to provide critical information to the president on a range of national security threats,” including key “blind spots” regarding Pakistan, China, Iran, North Korea—and Russia.

And it’s not just that SIGINT gives the U.S. pretty poor bang for the buck. Talk to veteran intelligence officers, like I did, and they’ll tell you the over-reliance on passive, one-dimensional SIGINT tools has come hand in hand with a sharp decline in the kind of vital information that can only be collected by humans on the ground, or HUMINT.

Even as contractors dangle new gadgets that promise to vacuum up ever more data, they say, old-school spying has become a dying art, mismanaged by bureaucrats, squeezed for resources and severely restricted by security rules.

“The NSA’s technological capabilities truly are breathtaking…but everybody out there knows it has a giant ear,” said a former senior CIA officer who worked for the agency for more than two decades, mostly in the Middle East. “When they [NSA targets] don’t want us to hear it, they write it on a piece of paper and walk it down three flights of stairs or use a courier.”

SIGINT, he added, “doesn’t tell you anything about context or your target’s plans and intentions. To get that you need human intelligence.”

The increasing dependence on SIGINT is all the more troubling given the NSA’s outsized role in targeting suspects for capture and assassination. “The system is knocking out multiple judgments about somebody’s alleged support or membership in a terrorist group,” said a former CIA division chief. “Do all these guys they point a finger at really need to be taken out? Even if they’re getting 90 percent right that still leaves room for a lot of mistakes…when those judgments are turned into actionable intelligence.”

The Rise of SIGINT

SIGINT’s role in U.S. intelligence gathering has been growing for decades, driven by the rise of the Internet and new forms of electronic communications, and by parallel leaps in technology that allow for greater ability to gather and exploit that data. “By the late-1990s you heard a lot of people at NSA saying that everything was going digital and they had to own the web,” says a former intelligence officer who specialized in counterterrorism, “9/11 was just what was needed for them to get the budget and authority they wanted.”

A Los Angeles Times story last July described SIGINT as the “primary vehicle for U.S. intelligence collection” and the source of considerably more than half of the President’s Daily Brief—the classified intelligence summary prepared for the White House.

“Policymakers are always frustrated by gaps in human intelligence and Americans are obsessed with data and technology,” a former senior official at the National Intelligence Council told me in explaining the growing reliance on SIGINT. “We’re always looking for technical solutions, because we believe they are pure and unbiased.”

One source, who has worked in multiple hotspots for the State Department and contractors, said the big NSA contractors fuel the acquisition frenzy because high-tech toys, hardware and intelligence analysis services bring them such lavish profits. “Contractors are never going to lobby to put more spies on the ground in Peshawar because that doesn’t make them money,” he said. “They make money by putting high-tech equipment in the air. Drones don’t help you understand tribal politics in Pakistan, but they make money for a lot of people in Northern Virginia.”

The human side of SIGINT is also a problem. There just aren’t enough NSA analysts, linguists and other personnel to process all the data the agency collects. And a retired veteran of both the CIA and the NSA told me that the great majority of NSA employees he worked with lived in suburban Maryland and had never traveled outside of the United States for pleasure or work. “The top overseas job at NSA is working as a special U.S. liaison and that’s usually at the embassy in London working with the GCHQ, the British counterpart agency, or with the Germans, or the Australians or maybe the Israelis,” he said. “They’re not mixing it up with Pakistanis and Jordanians, and they have a very limited understanding of the outside world.”

(Obviously many intelligence officers don’t like Snowden but he is not universally reviled, as some might believe. This source—who is highly conservative—told me: “I don’t know what he gave away, but he’s highlighted government spying on its citizens and that is something I fought against during the cold war, when Communist regimes were spying on their citizens…[L]et’s say you write me a letter and you put it in an envelope and you put your return address on it and my address on it, and drop it in the mail. The government takes that letter and they file your address and my address and they open the letter but they say we promise not to read it, we’ll just keep a copy of it and we’ll only read it later if we need to. And that’s exactly what the NSA is doing now. It’s the most un-American thing I’ve ever heard of. They say they’ll only read it if the letter writer or recipient does something wrong and they need to investigate it, but they can always manufacture a reason. Everyone I know does something ‘wrong’ every day—we all commit felonies five times a day in our conversations, so if the government wants to investigate you they will find something you did that was wrong and read that letter…What happened to the Fourth Amendment? There are going to be consequences of this, there’s going to be civil disobedience and I’m going to be part of it.”)

Another challenge is structural: the NSA is a “blue-collar” intelligence organization. Individuals are given narrow and discrete tasks—for example, to collect or analyze or decrypt signals—and information is passed along in assembly-line fashion. Few understand the broader context of a given assignment other than the team writing the final report. At the CIA, my sources told me, collectors understand the bigger picture and subject matter, what they’re going after and why they’re going after it. That allows them to ask better questions, and challenge or reject preconceived notions from superiors.

SIGINT also has some inherent limitations. For instance, while the core of SIGINT is capturing human communication, overhearing a conversation doesn’t make it true. “When you have a phone conversation between two people, you have to assume that 80 percent of the time you’re getting bullshit because people are always fabricating or modifying stories to make themselves look good and exaggerate their own role,” said the CIA/NSA veteran. But communications collected through SIGINT aren’t treated with sufficient skepticism, he said.

A number of sources told me that the NSA often disseminates information of questionable value that is touted as high-grade material—for example, intercepts of top personnel at foreign intelligence services. But if the intelligence service in question is corrupt and incompetent and its own product is unreliable, then the intercept has little value.

Just like criminals, intelligence targets can make surprisingly stupid revelations even when they know their communications are being captured. But SIGINT’s value is diminished when it’s assumed by everyone that the NSA is snooping. “It’s great that the NSA can listen in on Karzai’s staff meeting, but guess what?” said the former counterterrorism specialist. “He knows you’re listening. He can solve that problem with a walk in the garden.”

One problem seen throughout the history of SIGINT is that targets routinely dupe eavesdroppers by planting false narratives. “During World War II, we knew the Japanese and Germans were listening to us on certain channels so we fed them garbage, yet we’re still slow to imagine that people are doing the same thing to us,” said a former senior CIA officer. “Anyone halfway smart is going to put out information that’s 90 percent right and ten percent bullshit, and they’ll do it in a way that you latch onto the bullshit.”

The Collapse of HUMINT

When conducted properly, HUMINT provides a more textured, three-dimensional perspective than SIGINT. Officers in the field meet with agents (and sometimes targets) and can assess an individual’s role and importance in the bigger picture, as well as his motivation and reliability. A case officer can drill down if he sees that an agent isn’t looking him in the eye or tells a story that conflicts with previously provided information.

Where SIGINT is data and imagery and by definition passive, HUMINT can guide, as well as inform. “Sometimes you recruit a guy as a student activist and 30 years later he’s working in the prime minister’s office,” said a former intelligence officer. “You can go to that guy and say, ‘We’re going to give you a big bonus but we need you to do A, B, C and D.’ ”

One veteran officer scathingly dismissed it as the “Directorate of Drones”—that ran secret prisons and tortured terrorism suspects.

Post-9/11 reforms were intended to improve intelligence collection, but instead created layers of new bureaucracy, crowned by the office of Director of National Intelligence, and accelerated the push to hire office workers at the expense of field officers. Back in the mid-1980s, the CIA had an annual budget of $1.5 billion and about 8,000 employees, of whom 950 were spies posted overseas, said the former CIA officer who worked mostly in the Middle East. Fast-forward three decades and the agency has 23,000 employees (plus about 15,000 contractors), but the vast majority are administrative while the number of spies, sources say, has grown at a far slower pace.

“Officials today complain that Iran is a hard target,” said the former officer. “Well, fuck you; Russia was also a hard target. We’ve had six presidents since the Iranian revolution and we still can’t answer basic questions: What’s the status and goals of its nuclear program, and what are its strategic goals in the region?” (On March 16, the Los Angeles Times reported that the CIA’s chief of Iran operations, Jonathan Bank, was sent home on paid administrative leave “after an internal investigation found he had created an abusive and hostile work environment that put a crucial division in disarray.”)

Although there was a swell of CIA recruits after 9/11, turnover since then has been huge, especially in the spy service. “The people who know the most, the people with institutional knowledge, are mostly gone,” said the former CIA/NSA veteran. “They can’t keep talent because people get frustrated and leave or they go to work for contractors and get an immediate 50 percent salary increase.”

Even the CIA devotes a significant chunk of its budget to SIGINT collection, and after 9/11 became an increasingly paramilitary organization—one veteran officer scathingly dismissed it as the “Directorate of Drones”—that ran secret prisons and tortured terrorism suspects. “The thinking is that problems can be solved with military solutions so HUMINT has been put in second place,” another source told me.

When the CIA does have spies on the ground, they are constrained by rules of force protection and are too often holed up in embassies, multiple sources told me. American missions abroad have been turned into fortresses and approval is needed to leave diplomatic compounds, at least outside of relatively safe stations like London or Paris. CIA officers in high-risk areas are glued to flatscreens and may as well be based in Washington. “Traditional HUMINT is SADR — spotting, assessing, developing and recruiting—but you can’t do that when you’re heading out of the Green Zone in a convoy or a warlord in Afghanistan has to send his emissary to the embassy in the back of a car,” said the counterterrorism specialist.

Because of all these limitations, the CIA has increasingly outsourced its intelligence gathering to its foreign liaison partners, especially in the Middle East. Those partners were happy to cooperate with the agency’s rendition program and torture Al Qaeda suspects, but they certainly weren’t going to alert the CIA to the internal weaknesses of their respective regimes.

That’s a primary reason that the CIA was blindsided by the mass uprisings across the Middle East of the past few years. “The primary function of liaison relationships is counterterrorism, so who cares that the jails are filled with political prisoners or that workers are striking,” the same source said. “Meanwhile, you stop collecting intelligence on issues like regime stability and corruption and the role of organized crime because if you get caught, your counterterrorism liaison team is going to get kicked out of the country and the White House is going to be pissed off.”

For example, the CIA’s chief interlocutor in Egypt for decades was Omar Suleiman, the country’s longtime spy chief. “Guess what intelligence he gave us when the Mubarak regime came under threat?” said a Middle East expert who briefed the CIA during that period and was stunned by its ignorance of political realities. “He told us, ‘Don’t worry, we’ve got everything under control’.”

A January report by the Senate Intelligence Committee said there was no “tactical warning” leading up to the attacks on the U.S. compounds in Benghazi, citing a “dearth of clear and definitive HUMINT” prior to the events (as well as inaccurate and misleading SIGINT in their aftermath).

In July 2012, 16 months after the uprising began in Syria, The Washington Post reported that the CIA had virtually no presence in that country and American spy agencies were “largely confined to monitoring intercepted communications and observing the conflict from a distance.”

As a result, the Obama administration didn’t understand the composition of opposition forces to President Bashar al-Assad. “We’ve got to figure out who is over there and we don’t really know that,” an unnamed U.S. official told the Post before speculating that “there could be a number of extremist elements.”

And then of course there was the invasion of Crimea.

Stewart Baker, a former NSA general counsel, agreed that HUMINT was generally preferable to SIGINT, but said getting a human source well-placed is expensive and time consuming. “Our global presence means that we often have to gather intelligence on countries that weren’t high priorities for the CIA two years ago,” Mr. Baker told me in an email. “Having someone in [deposed Ukrainian president Viktor] Yanukovych’s inner circle would have been much more valuable than trying to intercept his calls but that wasn’t a priority last year and it won’t be a priority next year the way things are going.”

“They stopped spying on Russia,” former CIA officer Bob Baer told Politico. “I guarantee Putin wasn’t on Facebook working through this in his mind,” Mr. Baer said. “He wasn’t on Twitter. And he wasn’t on his cell phone… The only way to know in advance about planning for an invasion like this is to have someone in a military office at the Kremlin.”