When ex-government contractor Edward Snowden exposed the NSA's widespread efforts to eavesdrop on the internet, encryption was the one thing that gave us comfort. Even Snowden touted encryption as a saving grace in the face of the spy agency's snooping. "Encryption works," the whistleblower said last June. "Properly implemented strong crypto systems are one of the few things that you can rely on."

But Snowden also warned that crypto systems aren't always properly implemented. "Unfortunately," he said, "endpoint security is so terrifically weak that NSA can frequently find ways around it."

>Since the Heartbleed bug has existed for two years, it raises obvious questions about whether the NSA or other spy agencies were exploiting it before its discovery.

This week, that caveat hit home – in a big way – when researchers revealed Heartbleed, a two-year-old security hole involving the OpenSSL software many websites use to encrypt traffic. The vulnerability doesn't lie in the encryption itself, but in how the encrypted connection between a website and your computer is handled. On a scale of one to ten, cryptographer Bruce Schneier ranks the flaw an eleven.

Though security vulnerabilities come and go, this one is deemed catastrophic because it's at the core of SSL, the encryption protocol so many have trusted to protect their data. "It really is the worst and most widespread vulnerability in SSL that has come out," says Matt Blaze, cryptographer and computer security professor at the University of Pennsylvania. But the bug is also unusually worrisome because it could possibly be used by hackers to steal your usernames and passwords – for sensitive services like banking, ecommerce, and web-based email – and by spy agencies to steal the private keys that vulnerable web sites use to encrypt your traffic to them.

A Google employee was among those who discovered the hole, and the company said it had already patched any of its vulnerable systems prior to the announcement. But other services may still be vulnerable, and since the Heartbleed bug has existed for two years, it raises obvious questions about whether the NSA or other spy agencies were exploiting it before its discovery to conduct spying on a mass scale.

"It would not at all surprise me if the NSA had discovered this long before the rest of us had," Blaze says. "It's certainly something that the NSA would find extremely useful in their arsenal."

NSA Sets Its Sights on SSL ————————–

Although the NSA could use the Heartbleed vulnerability to obtain usernames and passwords (as well as so-called session cookies to access your online accounts), this would only allow them to hijack specific accounts whose data they obtained. For the NSA and other spies, the real value in the vulnerability lies in the private keys used for SSL that it may allow attackers to obtain.

Cracking SSL to decrypt internet traffic has long been on the NSA's wish list. Last September, the Guardian reported that the NSA and Britain's GCHQ had "successfully cracked" much of the online encryption we rely on to secure email and other sensitive transactions and data.

According to documents the paper obtained from Snowden, GCHQ had specifically been working to develop ways into the encrypted traffic of Google, Yahoo, Facebook, and Hotmail to decrypt traffic in near-real time, and there were suggestions that they might have succeeded. "Vast amounts of encrypted internet data which have up till now been discarded are now exploitable," GCHQ reported in one top-secret 2010 document. Although this was dated two years before the Heartbleed vulnerability existed, it highlights the agency's efforts to get at encrypted traffic.

The Snowden documents cite a number of methods the spy agencies have used under a program codenamed "Project Bullrun" to undermine encryption or do end-runs around it – including efforts to compromise encryption standards and work with companies to install backdoors in their products. But at least one part of the program focused on undermining SSL. Under Bullrun, the Guardian noted, the NSA "has capabilities against widely used online protocols, such as HTTPS, voice-over-IP and Secure Sockets Layer (SSL), used to protect online shopping and banking."

Security experts have speculated about whether the NSA cracked SSL communications and if so how the agency might have accomplished the feat. Now, Heartbleed raises the possibility that in some cases the NSA might not have needed to crack SSL. Instead, it's possible the agency used the vulnerability to obtain the private keys of companies to decrypt their traffic.

The Good News ————-

So far, though, there's no evidence to suggest this is the case. And there are reasons why this method wouldn't be very efficient for the NSA.

First, the vulnerability didn't exist on every site. And even on sites that were vulnerable, using the Heartbleed bug to find and grab the private keys stored on a server's memory isn't without problems. Heartbleed allows an attacker to siphon up to 64kb of data from a system's memory by sending a query. But the data that's returned is random – whatever is in the memory at the time – and requires an attacker to query multiple times to collect a lot of data. Though there's no limit to the number of queries an attacker can make, no one has yet produced a proof-of-concept exploit for reliably and consistently extracting a server's persistent key from memory using Heartbleed.

"It is very likely that it is possible in at least some cases, but it hasn't been demonstrated to work all the time. So even if a site is vulnerable, there's no guarantee you're going to be able to use [Heartbleed] to actually get keys," Blaze says. "Then you've got the problem that it's an active attack rather than a passive attack, which means they need to be able to do multiple round trips with the server. This is potentially detectable if they get too greedy doing it."

>The vulnerability didn't exist on every site. And even on sites that were vulnerable, using the Heartbleed bug to find and grab the private keys stored on a server's memory isn't without problems.

The security firm CloudFlare, which has spent the last three days testing various configurations to determine if, and under what conditions, it's possible to extract private keys using the Heartbleed vulnerability, says it hasn't been able to do so successfully yet, though its tests have been limited to configurations that include the Linux operating system on Nginx web servers.

Nick Sullivan, a Cloudflare systems engineer, says he has "high confidence" that a private key can't be extracted in most ordinary scenarios. Though it may be possible to obtain the key under certain conditions, he doubts it has occurred.

"I think it is extremely unlikely that a malicious attacker has obtained a private key from an Nginx server of a busy website," he says.

So far, they believe private keys can't be extracted from Apache servers either, though they don't have the same level of confidence in that yet. "If it is possible with Apache, it's going to be difficult," he says.

A few other researchers have claimed on Twitter and on online forums that they have retrieved private keys under various circumstances, though there doesn't appear to be a uniform method that works across the board.

Either way, there are now signatures available to detect exploits against Heartbleed, as Dutch security firm Fox-IT points out on its website, and depending on how much logging companies do with their intrusion-detection systems, it may be possible to review activity retroactively to uncover any attacks going back over the last two years.

"I suspect there are many people doing exactly that right now," Blaze says.

So what might the world's spy agencies say about all this? The GCHQ has a standard response for anyone who might wonder if the spooks used this or any other vulnerability to undermine SSL for their BULLRUN program. In a PowerPoint presentation the British spy agency prepared about BULLRUN for fellow spies, they warned: "Do not ask about or speculate on source or methods underpinning BULLRUN successes." In other words, they'll never say.