Secrets management is a hard problem. Many different approaches and tools are out there as well as new innovations in the space. Because we are deeply focused on this emerging space at CryptoMove, we put this post together as a resource to anybody who is thinking about or trying to learn more about secrets management. If this is an area of interest for you, we are running a private beta for CryptoMove’s Tholos Key Vault now—and we’re hiring!

Outline

I. Default secrets management approaches without tooling

II. Why secrets management matters

III. Options, tools, and commercial solutions for secrets management

IV. Key objectives for secrets management (pun intended)

V. Challenges around secrets management

Alpacas have to manage secrets too

I. Default secrets management approaches

Usually secrets management defaults to whatever is easiest and fastest. When devs are working together on an image in AWS, GCP, Azure or some other cloud native environment, it’s not uncommon to see .PEM files, SSH keys, and other configuration secrets passed around in cleartext and across a variety of communications channels. Some of the most popular “anti-patterns”:

Email

Slack

GitHub

Hard-coding

Clear text config files

Unencrypted laptop filesystems

There are plenty of examples of situations where the default “fast & easy” approach to secrets management and sharing have led to issues with devops & cloud transformations.

II. Why secrets management matters

Due to changes in infrastructure and software development processes, secrets are proliferating widely. Here are a few ways enterprise transformations are affecting secrets management at scale:

Cloud native development & multi-cloud infrastructure leads to secrets proliferation. As teams develop cloud-native applications, secrets to services for storage, compute, analytics, logging, and dozens of other services become important to share and manage. AWS has close to, if not more than, 100 services alone, all that need to be mediated with secrets, including API keys, SSH keys, tokens, certificates, and configuration.

leads to secrets proliferation. As teams develop cloud-native applications, secrets to services for storage, compute, analytics, logging, and dozens of other services become important to share and manage. AWS has close to, if not more than, 100 services alone, all that need to be mediated with secrets, including API keys, SSH keys, tokens, certificates, and configuration. Machine identity and machine-to-machine communication matters as much or more than user identity. In traditional enterprise architecture, human users were the key. Human identities were important to access documents, spreadsheets, email, and other tools. Modern enterprises, however, often have tens of thousands of machine-based identities that need to be managed and mediated via tokens, API keys, certificates, and other secrets. Automation and AI/ML will continue to shift landscapes from human user identity to machine-based identity.

and machine-to-machine communication matters as much or more than user identity. In traditional enterprise architecture, human users were the key. Human identities were important to access documents, spreadsheets, email, and other tools. Modern enterprises, however, often have tens of thousands of machine-based identities that need to be managed and mediated via tokens, API keys, certificates, and other secrets. Automation and AI/ML will continue to shift landscapes from human user identity to machine-based identity. DevOps processes & microservices based architecture also leads to secrets proliferation. Teams undergoing DevOps transformations move fast and manage many different infrastructure environments and services for development, testing, integration, and deployment. Secrets management for DevOps environments is vital as part of the secure software development lifecycle.

also leads to secrets proliferation. Teams undergoing DevOps transformations move fast and manage many different infrastructure environments and services for development, testing, integration, and deployment. Secrets management for DevOps environments is vital as part of the secure software development lifecycle. AI & data analytics leads to lots of secrets to manage for these pipelines as well.

leads to lots of secrets to manage for these pipelines as well. IoT, robotics, and embedded device proliferation leads to secrets proliferation, due to the need to have encryption and certificates for each IoT endpoint.

proliferation leads to secrets proliferation, due to the need to have encryption and certificates for each IoT endpoint. Blockchain projects in the enterprise also lead to more private keys than typically are used in applications. There becomes a need for an “enterprise wallet” to manage all those private keys.

Many mainstream Global 2000 and Fortune 500 enterprises are behind on this, stuck with legacy HSMs focused on encryption keys, and struggling to manage secrets in multi-cloud and services based architectures.

But why don’t HSMs and traditional encryption key management products work for modern secrets management?

There are legacy key management products for encryption, databases, even cloud services in some cases. The difficulty many teams encounter is when these legacy tools are used for devops and cloud infrastructure related keys like API keys, pem files, micro-services secrets, tokens, and others. We are seeing a gap in capability where they are not able to address these needs for a variety of reasons. Here are a few challenges of legacy key management solutions we’ve collected speaking with teams working on a modern cloud-based devops workflow:

Focused on encryption and PKI related keys and certificates, not really API keys

Don’t have cloud-native architectures

Not built for DevOps processes

Lack capability for dynamic secrets generation and rotation with multi-cloud services

Lack capability for containers & microservices

One example of a legacy company trying to plug a hole in its offerings here is CyberArk. They actually purchased an open source secrets management tool called Conjur to add that capability. Here’s how CyberArk announced the Conjur acquisition:

CyberArk aims to revolutionise DevOps security by using the fruits of the Conjur acquisition to deliver an “enterprise-class, automated privileged account security and secrets management” platform to secure the DevOps lifecycle and cloud-native environments. Which is nice. Privileged account credentials — SSH keys, API keys and more — are proliferating throughout IT infrastructure because of DevOps practices. CyberArk has acquired Conjur in an attempt to train and saddle this wild horse of software development.

Other traditional password and key management and encryption vendors haven’t addressed this capability gap — so they’re somewhat behind the 8-ball.

This gap in capability around secrets management for cloud infrastructure & devops is why a lot of companies have built their own secrets management tools in-house and helps explain why there are so many open source approaches — per below. This phenomena of many varied open source solutions to the secrets management problem shows it is a pretty large pain point for modern development organizations.

III. Options for secrets management

No real industry standard exists yet, as everything is fairly early in this space. At CryptoMove, we’re working on an enterprise-scalable solution for secrets management leveraging moving target defense technology under the hood. More on that later.

Due to the absence of too many off the shelf solutions, many companies have tried to build their own secrets management tools. Here are a few:

Square — Keywhiz

Keywhiz is a tool built by Square and operated in production for some time. It has a lot of features for storing and managing application and infrastructure secrets. Square describes Keywhiz as follows:

“Keywhiz makes managing secrets easier and more secure. Keywhiz servers in a cluster centrally store secrets encrypted in a database. Clients use mutually authenticated TLS (mTLS) to retrieve secrets they have access to. Authenticated users administer Keywhiz via CLI. To enable workflows, Keywhiz has automation APIs over mTLS.”

Pinterest — Knox

Pinterest also built its own secrets management tool and open sourced it, calling it Knox (as in Fort Knox). Here were the problems Pinterest was facing which led to the creation of Knox:

“Pinterest has a plethora of keys or secrets doing things like signing cookies, encrypting data, protecting our network via TLS, accessing our AWS machines, communicating with our third parties, and many more. If these keys become compromised, rotating (or changing our keys) used to be a difficult process generally involving a deploy and likely a code change. Keys/secrets within Pinterest were stored in git repositories. This means they were copied all over our company’s infrastructure and present on many of our employees laptops. There was no way to audit who accessed or who has access to the keys. Knox was built to solve these problems.”

Per Pinterest, the goals of Knox are:

“Ease of use for developers to access/use confidential secrets, keys, and credentials

Confidentiality for secrets, keys, and credentials

Provide mechanisms for key rotation in case of compromise

Create audit log to keep track of what systems and users access confidential data”

Lyft — Confidant

Lyft’s approach to secrets management resulted in a tool called Confidant. Lyft keeps it short and sweet on the description: “Your secret keeper. Stores secrets in DynamoDB, encrypted at rest.”

In describing the problem Lyft was trying to solve with Confidant, Lyft explains that with all the new internal and external services, keeping track of keys and secrets became a very laborious process:

“As Lyft has grown, we’ve added numerous services to our infrastructure. These services have credentials to internal services and external services, SSL keys, and other types of secrets. Each of these services has multiple environments, and to insulate these environments from each other, they have a version of each of these secrets for each environment. In many cases some of these secrets may be shared across a few services. Given a large number of services, this leads to a very large number of credentials.

The rotation of these secrets can be a laborious process, especially credentials for external services, since a large number of external services don’t have rotation methods that can be done without some amount of downtime and coordination. Coordination of the rotation of secrets became a difficult and time-consuming process for us pretty early on, and we knew the problem would only get worse as we added more internal services and more external dependencies.”

Here is a discussion of Confidant on Hacker News, with a lot of interesting feedback and compliments.

Many other tech companies have tried to are or are trying to create their own secrets management tool. Name a major tech company and they are probably working on this. More mainstream Fortune 500 or Global 2000 companies often are not aware of the problem yet, but as they increasingly transform their infrastructure and software delivery model, it’s likely they will run into the same issues that led to the creation of Keywhiz, Confidant, Knox, and others.