The “automagic” between Vault Secrets Engine, Kubernetes, and GitLab Pipelines.

Introduction

If you ever tried to modernize a large company’s process and infrastructure, you will probably come across some insufficient configuration/secrets management or lack thereof. Think of a .env file in the project root with shared database credentials in production, sound familiar? If not, then you are in a good place.

I have been both a web developer and currently a DevOps engineer. As a software developer, I only want to focus on my application and shouldn’t care about external tools. As a DevOps engineer, I want to automate the process and keep secrets secure. If secrets are compromised, mitigate risk as much as possible.

This is my opinionated way of deploying applications with rotating secrets into a scalable infrastructure automatically.

Requirements

There is a separation of responsibilities of developers to DevOps. In this context of configurations and secrets, I have a few requirements to follow:

My application should read configuration and secret variables from a file. Keep it simple and straightforward.

Only my application will go into a Dockerfile . No configuration management tools will be included in the Dockerfile .

. No configuration management tools will be included in the . When a better secret management tool comes out, I should be able to swap with ease.

Tooling

Container orchestration:

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

CI/CD Pipelines

Gitlab Pipelines are the top-level component of continuous integration, delivery, and deployment.

Secrets and configuration management

Hashicorp Vault: The shift to managing secrets and protecting data in dynamic infrastructure.

Secrets and configuration retriever: confd (preference) or consul-template. I will show both implementations.

No logos to show here 🙁

confd is a lightweight configuration management tool focused on keeping local configuration files up-to-date using data.

consul-template: This project provides a convenient way to populate values from Consul into the file system using the consul-template daemon.

Mitigation of Risk

Path of Credential Retrieval

The Kubernetes service account JWT is passed to Vault’s Kubernetes Auth Method. Once Vault validates the JWT via the token reviewer API, a Vault token with a TTL is returned. This vault token has the ability to generate new database credentials and renew its own TTL. Vault token is used to generate new database credentials with a TTL and written to .env for application to consume.

Strategies

Progressing from secure to more secure in order:

One Container

The application and tool used to retrieve secrets are packaged into one container.

Pros: The application is unaware of any configuration management tool and reads off the .env file.

Cons: If the container is compromised — the database credentials, Vault address, Vault token, and service account JWT will also be compromised. Secondly, this violates one of the requirements. The Dockerfile is not just the application, it has an added tool to retrieve secrets.

App Container with Sidecar

This improves the One Container approach. The application and the tool used to retrieve secrets are separated. They only share a volume containing the credentials.

Pros: The application container only contains the application. If the application container is compromised, only the database credentials are at risk until the TTL expires. The Vault token and service account JWT are not compromised because they are only in the sidecar.

Cons: The sidecar still holds the service account JWT for no reason. If an attacker gets hold of the JWT, he/she can keep generating Vault tokens.

InitContainer, App Container, and Sidecar (preferable)

The initContainer would use the service account JWT to authenticate, pass the Vault token to a shared volume for the sidecar to consume, and die. Then the application container and sidecar container would startup. The sidecar container would have the Vault token and be responsible for retrieving the credentials and updating the shared volume once the TTL has expired.

Pros: Everything that is compromisable has a TTL.

This approach was originally demonstrated by Seth Vargo in his Vault Kubernetes Workshop.