At Banzai Cloud we’re building a feature rich platform, Pipeline, on top of Kubernetes. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers - AWS, GCP, Azure and BYOC - and deploy all kinds of predefined or ad-hoc workloads to these clusters. We wanted to set the industry standard for the way in which our users log in and interact with secure endpoints, and, at the same time, we wanted to provide dynamic secret management for each application we support. With those goals in mind, and while taking into account our preference for native Kubernetes support, we selected Vault.

This operator (and the Bank-Vaults project itself) are all part of the Pipeline Platform. We strive to offer enterprise grade security to our users, and to applications deployed to Kubernetes through Pipeline. In fact, it’s a tier zero feature of our Pipeline Platform.

We’ve already described what Operators for Kubernetes are in a previous blog post. In this post we’ll explain how our Vault Operator was born out of the new Operator SDK and our Bank-Vaults project.

Banzai Cloud Vault Operator 🔗︎

There already exists a Vault Operator developed by CoreOS.

The Vault Operator makes it easier to install, manage, and maintain instances of Vault – a tool designed for storing, managing, and controlling access to secrets, such as tokens, passwords, certificates, and API keys – on Kubernetes clusters.

This is a great tool for installing a self-contained instance of Vault and etcd on top of Kubernetes. Using their work as inspiration, we’ve begun working toward our own solution. Although both ideas are similar, we started over and based our project on the brand new Operators SDK, which has really sped up development.

Here are a few features that we sorely missed from existing operators, and which we found as opensource projects on GitHub:

Automatic Vault initialization

Root Token and Unseal Keys encrypted and stored in cloud KMS systems (Azure Key Vault, AWS KMS, GCP KMS)

And in Kubernetes Secrets (however this should not be used in production, due to the current limitations of Kubernetes Secrets, see this doc for more details)

Automated unsealing

Automated re/configuration of Vault based on a YAML/JSON file like: Auth backends, Secret backends and policies

And which is not tied to etcd at all so that you can choose your own storage backend (e.g. cloud provider storages)

These features are already embedded in our Bank-Vaults project (where the heavy lifting is done for us), which is the most comprehensive open source project built on top of Vault. This means that the whole Vault experience is still cloud-agnostic, just like it was before - except the operator, which is extended with event handling.

All of the above configurations are done in a single YAML file. An example Vault Custom Resource Definition looks like this:

1 apiVersion : "vault.banzaicloud.com/v1alpha1" 2 kind : "Vault" 3 metadata : 4 name : "vault" 5 spec : 6 size : 1 7 image : vault: 0.10.1 8 bankVaultsImage : banzaicloud/bank-vaults:latest 9 10 # Describe where you'd like to store the Vault unseal keys and root token. 11 unsealConfig : 12 # In this case we are storing the root token and the unseal keys in Kubernetes secrets. 13 kubernetes : 14 secretNamespace : default 15 16 # A YAML representation of the final vault config file. 17 # See https://www.vaultproject.io/docs/configuration/ for more information. 18 config : 19 storage : 20 file : 21 path : "/vault/file" 22 listener : 23 tcp : 24 address : "0.0.0.0:8200" 25 tls_cert_file : /vault/tls/server.crt 26 tls_key_file : /vault/tls/server.key 27 ui : true 28 29 # See: https://github.com/banzaicloud/bank-vaults#example-external-vault-configuration for more details. 30 externalConfig : 31 policies : 32 - name : allow_secrets 33 rules : path "secret/*" { 34 capabilities = [ "create" , "read" , "update" , "delete" , "list" ] 35 } 36 37 auth : 38 - type : kubernetes 39 roles : 40 # Allow every pod in the default namespace to use the secret kv store 41 - name : default 42 bound_service_account_names : default 43 bound_service_account_namespaces : default 44 policies : allow_secrets 45 ttl : 1h

You may have noticed where the TLS certificates come from in this configuration and found it strange. Don’t worry, they’re automatically generated by the operator via Sprig functions, which are also used in Helm.

The main benefit of this operator, in contrast to our Vault Helm chart, is that we can react to certain Vault events in a standardized way (via the operator’s framework). Here, Vault is represented as a Kubernetes resource, and its state is stored in the Vault cluster.

Local Vault Operator 🔗︎

This is the simplest approach. The following commands will install a single node Vault instance that stores unseal and root tokens in Kubernetes secrets:

# Install the operator helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com helm upgrade --install vault-operator banzaicloud-stable/vault-operator # Create a Vault instance with the Vault CR kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/rbac.yaml kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr.yaml

After a few seconds, you can check the existing operator and vault pods:

kubectl get pods NAME READY STATUS RESTARTS AGE vault-66f484898d-lbltm 2/2 Running 0 10s vault-configurer-6c545cb6b4-dmvb5 1/1 Running 0 10s vault-operator-788559bdc5-kgqkg 1/1 Running 0 23s

Vault HA in Google Cloud 🔗︎

To demonstrate the HA setup, let’s deploy Vault on Google Cloud. For this example we’ll assume you have a GKE Cluster and have configured the Service Account with the following IAM Roles:

Cloud KMS Admin

Cloud KMS CryptoKey Encrypter/Decrypter

Storage Admin

If you don’t have a running GKE Cluster provision one with one click using Pipeline.

We’ll assume that your kubectl is pointed to the GKE cluster, so you can continue deploying the operator:

git clone git@github.com:banzaicloud/bank-vaults.git cd bank-vaults # You won't be able to change RBAC if this command isn't executed (you'll need to get the cluster-admin role): kubectl create clusterrolebinding ${ YOUR-NAME } -cluster-admin-binding --clusterrole = cluster-admin --user = ${ YOUR_EMAIL } # Install the operator helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com helm upgrade --install vault-operator banzaicloud-stable/vault-operator # First, edit this file to change the project settings, see below: kubectl apply -f deploy/rbac.yaml kubectl apply -f deploy/cr-gcs-ha.yaml

Here you can see a sample HA configuration:

1 apiVersion : "vault.banzaicloud.com/v1alpha1" 2 kind : "Vault" 3 metadata : 4 name : "vault" 5 spec : 6 size : 3 7 image : vault: 0.10.1 8 bankVaultsImage : banzaicloud/bank-vaults:master 9 10 # Describe where you would like to store the Vault unseal keys and root token 11 # in GCS encrypted with KMS. 12 # NOTE: please use your own project values, otherwise this example won't work: 13 unsealConfig : 14 google : 15 kmsKeyRing : "vault" # CHANGEME 16 kmsCryptoKey : "vault-unsealer" # CHANGEME 17 kmsLocation : "global" # CHANGEME 18 kmsProject : "continual-air-196513" # CHANGEME 19 storageBucket : "vault-ha" # CHANGEME 20 21 # A YAML representation of a final vault config file, this config represents 22 # a HA config in Google Cloud. 23 # See https://www.vaultproject.io/docs/configuration/ for more information. 24 config : 25 storage : 26 gcs : 27 bucket : "vault-ha" 28 ha_enabled : "true" 29 listener : 30 tcp : 31 address : "0.0.0.0:8200" 32 tls_cert_file : /vault/tls/server.crt 33 tls_key_file : /vault/tls/server.key 34 api_addr : https://vault.default: 8200 35 ui : true

Prometheus metrics are also in the making 🔗︎

We also plan to add Vault Prometheus metrics to the operator’s feature set (a feature that the CoreOS Operator already has), but first we’d like to carefully analyze our available options and decide what’s right for us and our users. For more details, please see the attached issue. We place a lot of emphasis on observability and do some pretty advanced monitoring of our federated clusters, so stay tuned as this will also land soon.

Unseal from mobile, through a quorum 🔗︎

There are a few ways of unsealing Vault that are already built into Bank-Vaults, but we weren’t entirely satisfied with those options. Since the operator is capable of listening to Kubernetes events - thus Vault restarts and statuses as well - we can notify alerting systems and notify operators to unseal Vault if, for example, Vaults has crashed. This allows us to unseal Vault with human intervention, using unseal keys distributed amongst a group of people, with keys stored on mobile devices (or laptops). Such a quorum would receive notifications on their mobile devices and, pending approval, the application would unseal Vault right from their phones. The Vault Operator would open up an unseal port for a limited time in order to minimize the attack window, during which Vault is sealed.

We will soon be releasing an iOS and Android app, and a Golang binary for different OSs

Learn through code 🔗︎

This project is open source, of course, and its code is tightly integrated into the Bank-Vaults project and can be found, here, in our GitHub repository.

We put a great deal of effort into securing users of Pipeline, since security is one of the platform’s core building blocks.