July 3, 2012

Update: Heroku’s official response.

Last week I discovered a major security flaw in the Heroku Cedar stack build system. This vulnerability exposed sensitive information including API keys, private keys and server credentials.

Once I realized the extent of the vulnerability, I immediately informed Heroku. I have been in regular contact with their security team and the problem has since been fixed.

Understanding the issue requires operational knowledge of the Cedar stack build system.

Cedar Build Process

Since Heroku runs on Heroku, after receiving a git push of an application, the build request is dispatched to a regular Heroku app named Codon that handles builds. Codon runs a buildpack which compiles the application so that it can be deployed.

Normally apps running on Heroku are entirely isolated using Linux Containers, but to perform builds Codon runs untrusted code in this container.

Source Code Exposure

I encountered a Ruby exception and backtrace from the Heroku build system while experimenting with custom buildpacks. Ruby backtraces look like this:

app.rb:2:in `foo': undefined method `a' for nil:NilClass (NoMethodError) from app.rb:5:in `<main>'

Backtraces include the paths to the source files that encountered the exception. This pointed me to the source files for Codon, which indicated the possibility of gaining read access to the code.

I then ran a custom buildpack that copied the source code into my Heroku app and verified that it was possible to view the source code of Codon.

While examining the source code I discovered that there was another vulnerability that was much more serious than source code exposure.

Sensitive Credential Exposure

Like most Heroku apps, Codon uses environment variables to configure runtime options including sensitive credentials. This ensures that credentials are not checked into version control. However, due to the constraints of Heroku containers, Codon is running as the same user as the buildpack, which is untrusted. This allows the buildpack to dump the environment variables of Codon from the Linux process table:

cat /proc/*/environ

The environment variables exposed included critical credentials such as internal API keys, a SSH private key with access to source code repositories, Redis connection details, and a key with access to their Campfire account.

Disclosure Timeline

Immediately after discovering this vulnerability, I sent an email to Heroku’s security team to start the disclosure process. I requested a PGP key first, as they did not provide one on their website. Here is the discovery and disclosure timeline:

2012-06-26 19:45 PDT Encountered backtrace and began experimenting. 2012-06-26 20:25 PDT Sent email to Heroku asking for PGP key. 2012-06-26 22:40 PDT Received PGP key from Heroku. 2012-06-26 22:56 PDT Received follow-up email with mobile phone number of a Heroku security engineer. 2012-06-26 22:58 PDT Sent PGP encrypted description of the vulnerability. 2012-06-26 23:06 PDT Received confirmation of receipt. 2012-06-27 12:01 PDT Received confirmation that an interim patch would be pushed in a few hours, and full patch by Tuesday (2012-07-03). 2012-06-28 20:44 PDT Checked validity of credentials, SSH and Campfire keys were still valid. 2012-06-29 16:13 PDT Checked validity of credentials, all credentials were invalid. 2012-07-03 13:35 PDT Received confirmation that the issue had been patched.

Customer Impact

The build system appears to have been vulnerable since the Cedar stack launched about a year ago. Customer applications and credentials could have been compromised at some point due to the credential exposed by the vulnerability. Anyone who ran applications on Heroku during this period should immediately reset all sensitive credentials, and audit their access logs to determine if any infrastructure or data has been accessed.

I suspect that a variant of this vulnerability may exist in other Platform as a Service build systems. Further research is warranted.

Full Disclosure: I remain a Heroku customer with several apps in production, and I have no plans to change platforms. Heroku offered me a paid penetration test contract, but required that I sign a retroactive non-disclosure agreement which would have precluded publishing this article.

If you liked this, then you should check out my article on security disclosure policy best practices.