Environment Variables Considered Harmful for Your Secrets

Storing your secret keys securely yet easily accessible for your code running in your production environment is a challenging task. With secret keys I mean for instance your keys for accessing 3rd-party APIs, keys used to encrypt/sign cookies, hash user passwords and so on. There are dire consequences if your production secret keys would get into the wrong hands. You'll want to tightly control how and when your secret keys are accessible.

We are probably in agreement on how not to store your secret keys, that is: in your code or in the same source control repository as your code. There are usually just too many people or 3rd-party services with (potential) access to your code repository who don't need to (and thus shouldn't) know your production secret keys.

One great document on how to architect modern web applications is the The Twelve Factor App. It documents 12 principles (or factors as they call it) every web developer should know. Their chapter on "Config" states: "Store config in the environment". While I agree that the OS's environment is a good place for general configuration, I can't recommend this for anything you'd like to keep secret.

When you store your secret keys in the environment, you are still prone to accidentally expose them -- exactly what we want to avoid.

Let's have a closer look at the environment and environment variables. This is a feature of any OS you'd be using to host web apps, including Linux, MacOS and Windows. Environment variables have been used for system configuration since basically forever and are designed to pass configuration from the parent process to its child processes. A child process by default inherits a copy of all environment variables (called the environment) from its parent process.

And within those properties of the environment lie the issues when it is used to store secret keys:

The environment is implicitly available to the process. It's hard to track access and how its content gets exposed. It's easy to grab the whole environment and print it out (can be useful for debugging) or send it as part of an error report for instance. The whole environment is passed down to child processes (if not explicitly filtered) which violates the principle of least privilege. So your secret keys are implicitly made available to any 3rd-party tools you're using (like ImageMagick to resize an image). It's hard to say what those 3rd-party tools will do with the environment, especially in rare occasions like crashes. External developers are not necessarily aware that your environment contains secret keys. By doing so your requirements (tightly controlled access) no longer match with most developers implicit expectation (nothing special in the environment, just generic system configuration). This is a dangerous mismatch.

Our solution thus has been to use a configuration file which is managed by the server's configuration management software (chef in our case). This way we can store secret keys outside of the code repository, and manage them using the same tool used to configure our servers. This solution, while not perfect, doesn't expose our secret keys to child processes and requires explicit access, which also communicates how developers should work with it.

But no matter which solution you choose, be aware of the different caveats on how you store your secret keys.