After going through some of the most common ways that companies screw up when managing their passwords and secrets, let’s consider a number of better options.

Passwords

To start, passwords. If there’s one thing all the experts agree on, it’s that they are lame.

Nobody likes them. We don’t like remembering them, and we like forgetting them even less.

But like them or not, we do need them, and we therefore need some way to keep track of them and distribute them. Enter the password manager.

Password Managers

In 2018, there’s really no excuse not to be using a password manager. If you aren’t, it’s very likely that you’re resorting to insecure password practices, needlessly wasting time and mental energy, or (most likely) all of the above.

There are a lot of password managers out there, and digging into the details of each one is more than I want to get into here. I personally use 1Password, and I’ve also heard good things about Dashlane and Keepass. Lastpass is also popular, though they’ve had some security issues in the past.

In the end, as long as it’s end-to-end encrypted and has a good track record, just use one that you like. Apart from the security benefits, it makes life a lot easier.

Application Secrets

Application secrets are things like API keys, database credentials, and admin passwords. Your app needs them to function, but they can cause a lot of damage if exposed.

These are tricker to deal with than passwords, and there’s also a wider variety of techniques being used to protect them.

Why is that?

They’re multi-dimensional — API keys have test and production versions. Credentials for databases, caches, and other services also differ across environments. They usually need to be distributed far more widely than passwords. Every developer working on a project needs a copy of the development secrets, and test servers, staging servers, and production servers all need a version of the secrets too. They’re used in development and server contexts. That means when things go wrong, it can halt development, cause downtime, and lead to major security issues. Keeping passwords organized is important too, but there are fewer moving parts involved. Development workflows, server architectures, and deployment strategies vary widely across teams and projects. Because secrets management touches all these areas, approaches to it tend to vary a lot as well.

Now that you understand the challenges, let’s look at a number of reasonably secure strategies for dealing with these things.

Strategy 1: Cloud-Hosted, Not End-To-End Encrypted

Secrets are stored with a cloud hosting provider. This includes AWS Parameter Store, Google Cloud’s Firebase, Heroku config vars, and others.

Pros

They tend to be pretty easy to integrate and make available to servers and other services on the same platform.

While you do need to trust the provider to protect your secrets, you’re presumably already trusting them to run your production servers, so there isn’t much additional risk.

No additional accounts or services are needed.

Cons

They usually give you a web UI to edit secrets. This is convenient, but as I mentioned in Part 1, browsers aren’t really secure enough to manage secrets. This can be mitigated by avoiding full-permission browser extensions or using CLI tools instead of web UIs, but it’s better not to have the temptation.

AWS Parameter Store, in particular, can still be pretty difficult to configure and integrate with an application. It’s not a big problem for AWS experts, but it will likely increase the amount of time you spend on server setup, deployments, integrations, maintenance, and other related things. If you’re an AWS newbie, you can expect it to make the already steep learning curve a fair amount steeper.

Access control systems like IAM can become complicated and error-prone as teams and infrastructure expand.

They don’t help you protect development-level secrets. Securing these as well as production secrets is an important component of a really solid secrets management approach.

Strategy 2: Self-Hosted, End-To-End Encrypted

Access to secrets is controlled by a central server on your own infrastructure. In most cases, secrets are also end-to-end encrypted so that the server itself doesn’t have access.

The well known options for this approach are open source. There’s Hashicorp’s vault, Lyft’s confidant, Square’s keywhiz, and a bunch of others.

Pros

Secrets remain completely under your control and never end up on external servers (whether encrypted or not).

Secrets are end-to-end encrypted. This allows you to decouple access to secrets from access to the storage layer.

It depends on the tool, but some can be used to encrypt and control development secrets in addition to production secrets.

Useful features like secrets generation, auto-expiration, and audit logs.

Cons

Setting up a secrets server and integrating it with your applications is a significant project. It requires a fairly high level of devops sophistication, and will likely take between n days and n weeks depending on your infrastructure’s complexity and your team’s capabilities.

There’s generally not enough attention paid to user friendliness and developer friendliness in these tools. Adding complexity to development and deployment workflows can slow down and frustrate engineers. They might then be tempted to route around the system in insecure ways when they’re in a hurry.

Apart from setup and integration, you’ve got a new piece of critical infrastructure to maintain, upgrade, monitor, and make highly available. These are non-trivial, time-consuming, and highly unforgiving tasks.

While some of these tools can be used to manage development-level secrets, production secrets tend to be given a lot more focus in terms of design and documentation. This means that you’re generally left to your own devices on fitting it into your development workflow.

Strategy 3: Encrypted With Configuration Management Tools

Many configuration management tools like Ansible, Chef, Kubernetes, and others have built-in or community-provided secrets management libraries.

Pros

Fits in with your existing toolset and deployment workflows.

Helps you to manage all your production secrets and configuration in one place.

Cons

Managing encryption keys securely is often left to the user. It’s easy to make mistakes here that defeat the purpose of the whole thing.

It can add complexity to your code, deployment scripts, and test server environments.

It doesn’t do anything for development secrets. As mentioned above, a solid strategy should protect these as well as production secrets. The line is fuzzy.

You can still end up with configuration sprawl as the number of services and applications you manage grows.

Strong access control can still be difficult to achieve, particularly in a way that doesn’t require trusting additional third parties.

Strategy 4: End-To-End Encrypted Messages or Files

Tools like Keybase, Signal, and others can be used for this.

Pros

Messages are end-to-end encrypted so that you don’t need to trust any third parties.

Any tool worth its salt will offer native apps so that you aren’t dealing with secrets in a browser.

Generally pretty quick to get up and running. It provides some basic security without needing to go all in on a more comprehensive approach.

Cons

Relying on messages to keep developers and servers in sync is error-prone, tends to get disorganized quickly, and is a constant source of delays and interruptions.

Some tools, like Keybase, have web messaging options in addition to their native apps. This tempts people to share secrets in a browser, which as mentioned above can be a problem due to all those devilishly convenient browser extensions looking over your shoulder.

While tools like Keybase and Signal are the real deal when it comes to zero-knowledge encryption implementations, there are a lot of pretenders out there in the messaging space. If you choose a messenger that’s web-based or gets the crypto wrong, then you’re back in Anti-Pattern land.

Strategy 5: EnvKey

Disclaimer: EnvKey’s creator is the author of this post.

EnvKey is like a password manager, but for application secrets and configuration. It’s a cloud-hosted, end-to-end encrypted system that allows you to securely configure a development machine or server with a single environment variable, and then keep it in sync automatically going forward.

It also provides a desktop app where you can manage configuration and access across all your apps, teams, and environments in one place.