I needed a secrets management tool that can be highly available, on-premise, cloud-native, run on low resources, and easily orchestrated to make my life easier. Is that so much to ask for?

Kubernetes is easily the most popular container orchestration platform proven in public cloud and on-premise for production.

Hashicorp Vault doesn’t really have any competition for secrets management tool especially considering we don’t want to lock-in to a cloud vendor.

Hashicorp recommends using Consul’s key-value store as Vault’s storage backend. Unfortunately, Consul bundles all of its features together. The official Vault Reference Architecture recommends a m5.large on AWS or n1-standard-4 on GCE. No thank you, just give me the kv store. I’ll go with etcd, a simple and reliable key-value store.

I want this build to be as close to what I would deploy to production. That means I need to think about the lifecycle of the tools. That’s where the Operators come in. The operators oversee installation, updates, and management of the lifecycle of all of the Operators (and their associated services) running across the cluster. That is awesome. We will be using CoreOS etcd Operator and BanzaiCloud Vault Operator.

Prerequisites

A few things are assumed before we can start deploying our storage backend and Vault.

A Kubernetes cluster. You can always spin up a local kubernetes cluster with minikube. Installation instructions can be found here.

minikube start

Helm tiller initialized.

helm init --upgrade --wait

kubectl create clusterrolebinding tiller-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

Installation

The Vault operator includes an option to install the etcd operator. Nothing wrong with the implementation but I prefer to deploy the storage backend myself.

etcd

The CoreOS etcd-operator helm chart is stable, the etcd operator is maintained, and supports the latest version of etcd.

Create etcd_operator_values.yaml with the following contents. These are the helm chart values for the etcd operator.

# etcdOperator

etcdOperator:

image:

repository: quay.io/coreos/etcd-operator

tag: v0.9.4

# backup spec

backupOperator:

image:

repository: quay.io/coreos/etcd-operator

tag: v0.9.4

spec:

storageType: S3

s3:

s3Bucket:

awsSecret:

# restore spec

restoreOperator:

image:

repository: quay.io/coreos/etcd-operator

tag: v0.9.4

spec:

s3:

# The format of "path" must be: "<s3-bucket-name>/<path-to-backup-file>"

# e.g: "etcd-snapshot-bucket/v1/default/example-etcd-cluster/3.2.10_0000000000000001_etcd.backup"

path:

awsSecret:

## etcd-cluster specific values

etcdCluster:

name: etcd-cluster

size: 3

version: 3.4.0

image:

repository: quay.io/coreos/etcd

tag: v3.4.0

enableTLS: false

# TLS configs

tls:

static:

member:

peerSecret: etcd-peer-tls

serverSecret: etcd-server-tls

operatorSecret: etcd-client-tls

Working combinations:

As of initial write-up: v0.9.3 of etcd-operator and v3.3.13 of etcd

As of Sept 2019: v0.9.4 of etcd-operator and v3.4.0 of etcd

I will cover backing up and restoring in another write-up. Install the chart.

helm upgrade --install etcd-operator stable/etcd-operator -f etcd_operator_values.yaml

Should take a few seconds, you can run helm status etcd-operator to check.

Once the etcd operator is running, deploy the etcd cluster.

cat <<EOF | kubectl apply -f -

apiVersion: "etcd.database.coreos.com/v1beta2"

kind: "EtcdCluster"

metadata:

name: "etcd-cluster-vault"

etcd.database.coreos.com/scope: clusterwide

spec:

size: 3

version: "3.4.0"

EOF