Setup

In my mind, there are two main workflows for setting up. One is for setting a new cluster to communicate with Vault, primarily focused on the authentication method (this happens per cluster). The second is for onboarding a new application with a new database connection (this happens per application).

In both cases, we can think of four main sections of responsibility that can be configured and pass information.

Kubernetes — Home for the application. Responsible for making requests to Vault and validating tokens. Vault Kubernetes Authentication Method — Perform authentication and are responsible for assigning identity and a set of policies to a service account. Vault Policies — Policies provide a declarative way to grant or forbid access to certain paths and operations in Vault. Vault Database Secrets Engine — Generates database credentials dynamically based on configured roles for the MySQL database.

I have created a small repository of scripts to help. Each section is represented in their own scripts in vault-helpers. If you are planning on using my scripts, please read the Assumptions section in README.md before continuing.

A New Cluster

Kubernetes

Create vault-auth service account in the default namespace.

kubectl create serviceaccount vault-auth

Give the vault-auth service account the token reviewer role. This allows the service account to validate other service accounts in their namespace.

kubectl apply -f -<<EOH

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

name: role-tokenreview-binding

namespace: default

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: system:auth-delegator

subjects:

- kind: ServiceAccount

name: vault-auth

namespace: default

EOH

Using vault-helpers scripts.

configure/kubernetes.sh --context apps-cluster --setup

Database Secrets Engine

Enable the database secrets engine at the default path database/

vault secrets enable database

using vault-helpers scripts

configure/vault-secrets-database.sh --enable

Kubernetes Authentication Method

In order for Vault to communicate with our Kubernetes cluster, there are a few pieces of information that are required.

Kubernetes host — Address that Vault can connect with.

k8s_host="$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")"

Cluster authority data — Certificate to verify the connection.

k8s_cacert="$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)"

User token — vault-auth service account with token reviewer role. Vault will interact with the cluster using this service account.

secret_name="$(kubectl get serviceaccount vault-auth -o go-template='{{ (index .secrets 0).name }}')" tr_account_token="$(kubectl get secret ${secret_name} -o go-template='{{ .data.token }}' | base64 --decode)"

Enable Kubernetes auth method at the default path: auth/kubernetes

vault auth enable kubernetes

Configure the Kubernetes auth method.

vault write auth/kubernetes/config token_reviewer_jwt="${token_reviewer_jwt}" kubernetes_host="${kubernetes_host}" kubernetes_ca_cert="${kubernetes_cacert}"

Using vault-helpers scripts.

configure/vault-auth-kubernetes.sh --token-reviewer-jwt $(configure/kubernetes.sh --token-reviewer-jwt) --k8s-host $(configure/kubernetes.sh --k8s-host) --k8s-cacert-base64 $(configure/kubernetes.sh --k8s-cacert) --enable --configure

New App

Kubernetes

Create a new namespace. Our demo application will go here, not default. We’re not savages.

kubectl create namespace demo

Create a service account in that namespace. This service account will be used by our deployments that require secrets from Vault. Notice the service account name, this is different than the vault-auth service account created earlier. They have different uses.

kubectl --namespace=demo create serviceaccount vault

Add a configmap with the Vault address, preferably an internal IP or DNS address.

kubectl --namespace=demo create configmap vault --from-literal "vault_addr=https://vault.local:8200"

Add Vault’s cert to a secret.

kubectl --namespace=demo create secret generic vault-tls --from-file "${VAULT_CACERT}"

Using vault-helpers scripts.

configure/kubernetes.sh --context apps-cluster --vault-addr https://vault.local:8200 --vault-cacert $VAULT_CACERT --new-namespace demo

Database Secrets Engine

Configure Vault to connect to the database. The user needs enough privilege to create and delete users and grant access. We will need to allow the demo-role role to access this connection, which we will create in a moment.

vault write database/config/demo-db plugin_name=mysql-database-plugin connection_url="{{username}}:{{password}}@tcp(demodb.local)/" allowed_roles="demo-role" username="root" password="password"

Rotate the root password. A new password will be generated and used. That means only Vault will have the password, not even you.

vault write -f database/rotate-root/demo-db

Add the role which allows for temporary users to be created in the database.

vault write database/roles/demo-role db_name=demo-db creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT SELECT ON *.* TO '{{name}}'@'%';" default_ttl="1h" max_ttl="24h"

Using vault-helpers scripts.

configure/vault-secrets-database.sh --db-name demo-db --host demodb.local:3306 --username root --password password --role-name demo-role --configure --rotate-root --role

Vault Policies

Create a policy that would allow the creation of a temporary set of credentials.

vault policy write demo-policy -<<EOF

path "database/creds/demo-role" {

capabilities = ["read"]

}

EOF

Using vault-helpers scripts.

configure/vault-general.sh --policy demo-db-r

demo-db-r is the name of the policy. This holds some assumptions. I don’t want to keep creating new policies for every database that I have. As long as I keep the Kubernetes namespace demo similar to the database role demo-role , I can use policy templates. After running the script, the policy will look something like this:

path "database/creds/{{identity.entity.aliases.auth_kubernetes_3626dffe.metadata.service_account_namespace}}-role" {

capabilities = ["read"]

}

Kubernetes Auth Method

Create a Kubernetes Auth role that grants the policy (that allows for the creation of credentials) to the service account in a specific namespace. All of which have been set up already.

vault write auth/kubernetes/role/demo bound_service_account_names=vault bound_service_account_namespaces=demo policies=demo-policy ttl=1h

Using vault-helpers scripts.