Heartbleed aftermath

LWN.net needs you! Without subscribers, LWN would simply not exist. Please consider signing up for a subscription and helping to keep LWN publishing

By many accounts, the "Heartbleed" vulnerability in OpenSSL was one of the worst security-related bugs seen in years. The actual cost of this vulnerability will take some time to work out, though; rumors of long-time exploitation by the intelligence community notwithstanding, there is not much hard evidence of real-world compromises enabled by this bug. In truth, the nature of the bug makes such evidence hard to come by, and, in any case, the cost of updating much of the Internet, changing passwords, and replacing keys will be certainly be high. While the full picture emerges in the long term, we can profitably look at a few interesting themes that have already become apparent.

Resources

One commonly heard assertion is that the OpenSSL project simply does not have the resources needed to maintain a code base of that size (about 300,000 lines of code), complexity, and importance. Companies take OpenSSL for granted, it is said; they make vast amounts of money using it, but do not contribute back to its development. Steve Marquess, the OpenSSL "money guy," put it this way:

Even if those donations continue to arrive at the same rate indefinitely (they won’t), and even though every penny of those funds goes directly to OpenSSL team members, it is nowhere near enough to properly sustain the manpower levels needed to support such a complex and critical software product. While OpenSSL does “belong to the people” it is neither realistic nor appropriate to expect that a few hundred, or even a few thousand, individuals provide all the financial support. The ones who should be contributing real resources are the commercial companies and governments who use OpenSSL extensively and take it for granted.

There may be value in asking, though, whether support levels are the real problem and, if so, what form the solution would take. There are calls for users and companies to donate to the project, but nobody seems to be pointing out one simple fact: it is a rare development project indeed that operates out of cash donations. It just does not seem to be a sustainable way to keep developers employed working on a project. Companies don't donate cash to OpenSSL because, almost without exception, they do not donate to free software projects at all.

That does not mean that resources are not available for OpenSSL development, though. It is interesting to continue reading in Steve's posting on the subject. The OpenSSL Software Foundation offers consulting services starting at $250/hour. Those rates are apparently not high enough to put off clients:

At the moment OSF has about a hundred grand in open contracts — these are executed contracts with purchase orders, not just contracts in discussion or negotiation — that aren’t being worked because no one in this very small “workforce” of qualified OpenSSL developers is available to work on them.

Given that, one might reasonably conclude that the limiting resource is not money, it's qualified developers. That is a different sort of problem.

Coming up with more qualified developers is not something that will happen overnight. Cryptographic programming is not easy, and the TLS protocols are seemingly not designed to make it any easier. If companies are waiting with contracts in hand for qualified developers to just show up, they may find themselves waiting in vain while the quality of the software continues to suffer. Perhaps there is a better way.

Projects that have large numbers of commercially supported developers are typically not employing those developers through a donation-supported foundation. Instead, those developers are employed directly by the companies that have an interest in the development of the software. In the most successful settings, these companies have in-house programs to bring developers up to speed on the projects they care about the most. That is the sort of support that projects like OpenSSL need; the companies that depend on it need to be directly employing (and training) developers to get this work done. Once that happens, OpenSSL should improve quickly.

The perils of closed enhancements

One of the companies that arguably should be employing OpenSSL developers is Akamai, which uses OpenSSL heavily on its massive content distribution network. On April 11, Akamai proudly told the world that it could not have lost SSL keys via the Heartbleed vulnerability, despite having run vulnerable versions of the software. Why?

Akamai uses a custom memory allocator with a separate heap for SSL private keys. We replace the OPENSSL_malloc call with our own secure_malloc. Our edge server software sets up two heaps. Whenever we're allocating memory to hold an SSL private key, we use the special "secure heap." When we're allocating memory to process any other data, we use the "normal heap."

The reasoning was that this secure heap, being kept far away from normal working memory, could not be exposed via Heartbleed. Akamai kept this code to itself; it was not contributed back upstream to the OpenSSL project. Or, at least, it was not contributed until Akamai posted it on April 11.

It took less than two days for an independent reviewer to look over the code and conclude that, in fact, it was buggy and offered no protection at all. By late on April 13, Akamai had conceded the point and begun the process of changing out all customer SSL keys. The company deserves credit for facing up to its problems once they were pointed out, but it can't have been Akamai's proudest moment.

Akamai, it seems, had been using its custom allocator for many years. Had that code been contributed upstream, there is a good chance that the bug would have been found a long time ago. Akamai would have benefited with secure code that was actually secure and the damage to the net as a whole caused by the Heartbleed vulnerability would have been considerably reduced. There really is value in this whole open-source development idea, but it seems that this lesson has to be learned the hard way over and over again.

End notes

There does seem to be one influx of new developer effort, though: the OpenBSD project has apparently decided to rip the code apart and clean it up to its standards. The result seems likely to be a cleaner and more secure OpenSSL in OpenBSD, but there is no evidence of any sort of coordination or communication with the OpenSSL project or of any intent to push these changes back upstream. In other words, it seems likely that OpenBSD has just forked OpenSSL. This move may allow some immediate problems to be addressed but, in the long term, this project (which, like most projects, has resource constraints of its own) has just taken on the maintenance of a complex, security-critical body of code. The cost of that decision will have to be paid continuously into the indefinite future.

Finally, we have had a few queries about LWN's server. As LWN readers know, we are strong proponents of running the latest and greatest software with all the newest fixes and features. That is why the main LWN server is running on CentOS 5. First impressions notwithstanding, this version of CentOS does not actually predate the World Wide Web, but it is old enough to have never had a vulnerable version of OpenSSL, so LWN's servers were not vulnerable to Heartbleed for even small period of time. The company that does LWN's credit card processing (Braintree) was vulnerable, but responded quickly once the vulnerability was disclosed. PayPal, our other payment provider, claims to have never been vulnerable but says nothing about why that is true.

In summary: there should be no need to change LWN passwords or to worry about payments made for LWN subscriptions.

The Heartbleed vulnerability is a reminder that there are classes of bugs that can affect a substantial portion of the Internet. Heartbleed may or may not have been extensively exploited against people on the net, but, in a sense, it almost does not matter. The chances that other vulnerabilities exist and are known to hostile entities would appear to be quite high. Free software may well offer the best defenses we have against security problems, but it is clear that those defenses are nowhere near as strong as they need to be. For years we have been telling the world that it can rely on our software; now that the world is, to a great extent, convinced of that fact, we need to accept the responsibility to make sure that this claim is actually true.

