Mutual Auto-Unseal Two Vault clusters in Kubernetes

Introduction

When I deploy Vault to Kubernetes, I realize it’s important to have auto-unseal capability to make the Vault cluster true highly available.

In my previous article “Highly available Vault cluster in Kubernetes”(link), even I’ve tried hard to make a Vault cluster as highly available as possible, without auto-unseal, the Vault cluster could tolerate partial pods failure, but won’t survive a whole cluster reboot.

This article well explained the why and how. The problem for me is I don’t have “AWS KMS service” to use, or any similar cloud vendor security services, because I run Kubernetes on self-manager cloud provider (this article explains my setup).

Vault version 1.1 adds support “Transit Auto Unseal”, which is to use a second Vault cluster B to auto unseal cluster A.

So here is the idea, deploy 2 Vault clusters A and B and set them up to do auto-unseal for each other.

If both Vault clusters A and B are highly available by itself, in case of any pod crashes/updates, the new pod can get auto-unsealed from the other cluster. The chance that all pods of both clusters are down at exactly the same time should be very small.

In this article, I deployed two Vault clusters in same Kubernetes cluster, named as “vault” and “vault-unlock”, in 2 namespaces. In actual production environment, I would suggest more robust way to deploy them in two Kubernetes clusters.

Below is the diagram.

Architecture

Updated 1/9/2020

Vault uses Consul Cluster as storage back end. To make it secure/stable, I shared my recent experience about running Consul in Kubernetes for production in this article.

Sources used in this article

Prerequisite

Highly Available Kubernetes cluster (example)

Deploy 2 vault clusters

First cluster is named as “vault”, in source code folder “vault”.

This cluster is for external service, so it has an Ingress rule.

Follow the steps described here, or similar as below.

Second cluster is named as “vault-unlock”, in source code folder “vault2”.

This cluster is dedicated to auto-unseal purpose.

Basically it’s the same steps as creating the first cluster, just everything in “vault-unlock” namespace, and without Ingress.

Check the README.md files in the source code for below commands.

# create namespace

$ kubectl apply -f vault2/vault_namespace.yml # go to ca folder

$ cd ca # create certs

$ cfssl gencert \

-ca=ca.pem \

-ca-key=ca-key.pem \

-config=config/ca-config.json \

-profile=default \

config/vault-unlock-csr.json | cfssljson -bare vault-unlock # it's important to use the same GOSSIP_ENCRYPTION_KEY as first cluster

$ kubectl -n vault-unlock create secret generic vault \

--from-literal="gossip-encryption-key=${GOSSIP_ENCRYPTION_KEY}" \

--from-file=ca.pem \

--from-file=vault-unlock.pem \

--from-file=vault-unlock-key.pem \

--from-file=vault-client.pem \

--from-file=vault-client-key.pem # go to vault2 folder

$ cd vault2

$ kubectl -n vault-unlock create configmap vault --from-file=config/vault.json # now deploy

$ kubectl apply -f . # init and unseal

$ vault operator init $ vault operator unseal

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed false

Total Shares 5

Threshold 3

Version 1.1.3

Cluster Name vault-cluster-98e11b27

Cluster ID 3804aaeb-8f62-ca79-4e34-f2cad7dfd062

HA Enabled true

HA Cluster

HA Mode active $ vault statusKey Value--- -----Seal Type shamirInitialized trueSealed falseTotal Shares 5Threshold 3Version 1.1.3Cluster Name vault-cluster-98e11b27Cluster ID 3804aaeb-8f62-ca79-4e34-f2cad7dfd062HA Enabled trueHA Cluster https://10.0.210.129:8201 HA Mode active

After all deployed, the consul UI looks like below

Vault UI is accessible via public address

First half — Use cluster vault to auto-unseal vault-unlock

I follow the directions from official document.

Precondition

cluster “vault” is fully unsealed and working

Operations in cluster “vault”

# connect to "vault" cluster

$ kubectl -n vault port-forward svc/vault 8200 & # below commands need root token $ vault login

Token (will be hidden): # enable audit, logs into container console

$ vault audit enable file file_path=stdout

Success! Enabled the file audit device at: file/ # enable transit engine

$ vault secrets enable transit # create key "autounseal"

$ vault write -f transit/keys/autounseal

Success! Data written to: transit/keys/autounseal # create autounseal policy, content from official doc

$ vault policy write autounseal autounseal.hcl

Success! Uploaded policy: autounseal # create a token with the policy, it's wrapped in wrapping_token

$ vault token create -policy="autounseal" -wrap-ttl=120

Key Value

--- -----

wrapping_token: s.AO55UYmKpT4TmPVn2AZ0UkNT

wrapping_accessor: 7N3PtIwhuAtjJZ2GvA9RM0iV

wrapping_token_ttl: 2m

wrapping_token_creation_time: 2019-06-25 19:24:46.345501602 +0000 UTC

wrapping_token_creation_path: auth/token/create

wrapped_accessor: B8kWfoLfXYOAxFI2P2TIduaI # get the token from wrapping_token

# as previous command shows, the wrapping_token is only valid for 120 seconds

$ VAULT_TOKEN="s.AO55UYmKpT4TmPVn2AZ0UkNT" vault unwrap

Key Value

--- -----

token s.LvLFn4InVdIorAFS5E9j6xd3

token_accessor KST0fJN4xMd2Yaztrmya8SNx

token_duration 768h

token_renewable true

token_policies ["autounseal" "default"]

identity_policies []

policies ["autounseal" "default"] # this is the token to be used in next step

token s.LvLFn4InVdIorAFS5E9j6xd3

Operations in cluster “vault-unlock”

In source code, copy vault2/config/vault.json as vault2/config/vault_autounseal.json and append the “seal” configuration.

Below is my example, refer to detailed doc. Fill in the token got from previous step.

Re-create the Configmap “vault”, with the content from file “vault_autounseal.json”, but still name it as “vault.json” to be used by the pod.

kubectl -n vault-unlock create configmap vault --from-file=vault.json=config/vault_autounseal.json -o yaml --dry-run | kubectl replace -f -

Re-create vault-unlock deployment. The log with auto-unseal configuration looks like below:

# Do below in source code vault2/

$ kubectl delete -f vault_deployment.yml

$ kubectl apply -f vault_deployment.yml

$ kubectl -n vault-unlock logs -f vault-544d44df85-6pmvf -c

vault

==> Vault server configuration:

Transit Address:

Transit Key Name: autounseal

Transit Mount Path: transit/

Api Address:

Cgo: disabled

Cluster Address:

Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")

Log Level: info

Mlock: supported: true, enabled: true

Storage: consul (HA available)

Version: Vault v1.1.3

Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb Seal Type: transitTransit Address: https://vault.vault.svc.cluster.local:8200 Transit Key Name: autounsealTransit Mount Path: transit/Api Address: https://vault.vault-unlock.svc.cluster.local:8200 Cgo: disabledCluster Address: https://10.0.181.115:8201 Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")Log Level: infoMlock: supported: true, enabled: trueStorage: consul (HA available)Version: Vault v1.1.3Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb 2019-06-25T19:51:54.065Z [WARN] core: entering seal migration mode; Vault will not automatically unseal even if using an autoseal: from_barrier_type=shamir to_barrier_type=transit

==> Vault server started! Log data will stream in below:

Now the “vault-unlock” cluster is in “seal migration mode”.

Do Seal Migration

# connect to "vault-unlock" cluster

$ kubectl -n vault-unlock port-forward vault-544d44df85-6pmvf 8200 & # below means seal migration in progress

$ vault status

Key Value

--- -----

Recovery Seal Type shamir

Initialized true

Sealed true

Total Recovery Shares 5

Threshold 3

Unseal Progress 0/3

Unseal Nonce n/a

Seal Migration in Progress true

Version 1.1.3

HA Enabled true # do seal migration

$ vault operator unseal -migrate

Unseal Key (will be hidden):

$ vault status

Key Value

--- -----

Recovery Seal Type shamir

Initialized true

Sealed false

Total Recovery Shares 5

Threshold 3

Version 1.1.3

Cluster Name vault-cluster-c6f60ee0

Cluster ID a152257b-813d-9eb3-57ea-f7d0683f0b1c

HA Enabled true

HA Cluster

HA Mode active # Repeat until unsealed$ vault statusKey Value--- -----Recovery Seal Type shamirInitialized trueSealed falseTotal Recovery Shares 5Threshold 3Version 1.1.3Cluster Name vault-cluster-c6f60ee0Cluster ID a152257b-813d-9eb3-57ea-f7d0683f0b1cHA Enabled trueHA Cluster https://10.0.181.115:8201 HA Mode active

Seal migration is only required for one of the pods.

Now restart the vault-unlock cluster again. All the pods of “vault-unlock” will be auto-unsealed. The log looks like below:

# Do below in source code vault2/

$ kubectl delete -f vault_deployment.yml

$ kubectl apply -f vault_deployment.yml $ kubectl -n vault-unlock logs -f vault-544d44df85-7877s -c

vault

==> Vault server configuration:

Transit Address:

Transit Key Name: autounseal

Transit Mount Path: transit/

Api Address:

Cgo: disabled

Cluster Address:

Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")

Log Level: info

Mlock: supported: true, enabled: true

Storage: consul (HA available)

Version: Vault v1.1.3

Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb Seal Type: transitTransit Address: https://vault.vault.svc.cluster.local:8200 Transit Key Name: autounsealTransit Mount Path: transit/Api Address: https://vault.vault-unlock.svc.cluster.local:8200 Cgo: disabledCluster Address: https://10.0.181.70:8201 Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")Log Level: infoMlock: supported: true, enabled: trueStorage: consul (HA available)Version: Vault v1.1.3Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb ==> Vault server started! Log data will stream in below: 2019-06-25T20:01:28.680Z [INFO] core: stored unseal keys supported, attempting fetch

2019-06-25T20:01:28.702Z [INFO] core: vault is unsealed

2019-06-25T20:01:28.702Z [INFO] core.cluster-listener: starting listener: listener_address=[::]:8201

2019-06-25T20:01:28.702Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201

2019-06-25T20:01:28.703Z [INFO] core: entering standby mode

2019-06-25T20:01:28.706Z [INFO] core: unsealed with stored keys: stored_keys_used=1

2019-06-25T20:01:28.736Z [INFO] core: acquired lock, enabling active operation

2019-06-25T20:01:28.781Z [INFO] core: post-unseal setup starting

2019-06-25T20:01:28.784Z [INFO] core: loaded wrapping token key

2019-06-25T20:01:28.784Z [INFO] core: successfully setup plugin catalog: plugin-directory=

2019-06-25T20:01:28.788Z [INFO] core: successfully mounted backend: type=system path=sys/

2019-06-25T20:01:28.788Z [INFO] core: successfully mounted backend: type=identity path=identity/

2019-06-25T20:01:28.789Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/

2019-06-25T20:01:28.811Z [INFO] core: successfully enabled credential backend: type=token path=token/

2019-06-25T20:01:28.811Z [INFO] core: restoring leases

2019-06-25T20:01:28.811Z [INFO] rollback: starting rollback manager

2019-06-25T20:01:28.813Z [INFO] expiration: lease restore complete

2019-06-25T20:01:28.816Z [INFO] identity: entities restored

2019-06-25T20:01:28.821Z [INFO] identity: groups restored

2019-06-25T20:01:28.822Z [INFO] core: post-unseal setup complete

So, cluster “vault-unlock” has got the “auto-unseal” capability.

Second half — Use cluster vault-unlock to auto-unseal vault

Basically, it’s the same thing as first half.

Just keep in mind which cluster is under operation… Especially at the step “re-create” deployment, make sure the source folder is correct.

I would suggest to backup the consul cluster frequently in the steps. Refer to doc.

Done ?

After both vault clusters switch to “Auto-Unseal” mode, it’s important to keep in mind not to destroy both deployments at the same time. If at least one vault pod is in “unsealed” mode, they will eventually all get unsealed.

Troubleshooting:

Error when trying to do seal migration

$ vault operator unseal -migrate

Unseal Key (will be hidden):

Error unsealing: Error making API request.

Code: 400. Errors: URL: PUT https://127.0.0.1:8200/v1/sys/unseal Code: 400. Errors: * 'migrate' parameter set true in JSON body when not in seal migration mode

This is because the Configmap vault still has contents from config/vault.json. Replace it with the content from config/vault_autounseal.json

Error enabling transit seal “client tls verify”

# vault container could not start

$ kubectl -n vault-unlock get pod

NAME READY STATUS RESTARTS AGE

vault-6d5fff8bd8-6d5dr 1/2 Error 1 12s

vault-6d5fff8bd8-cmfjt 1/2 Error 1 12s

vault-6d5fff8bd8-kzn9l 1/2 Running 1 12s

t

Error parsing Seal configuration: Put $ kubectl -n vault-unlock logs -f vault-6d5fff8bd8-6d5dr -c vaulError parsing Seal configuration: Put https://vault.vault.svc.cluster.local:8200/v1/transit/encrypt/autounseal : x509: certificate signed by unknown authority

This problem gave me trouble for long time… even after I believe I’ve correctly setup all the parameters for the “seal” configuration.

It’s easy to make it work by setting “tls_skip_verify” as “true”, and it might be ok as here both Vault clusters are inside the same Kubernetes cluster.

However in case of two Vault clusters in two different Kubernetes clusters, the TLS verification is must.

"tls_skip_verify": "false",

"tls_server_name": "vault",

"tls_ca_file": "/etc/tls/ca.pem",

"tls_client_cert": "/etc/tls/vault-unlock.pem",

"tls_client_key": "/etc/tls/vault-unlock-key.pem"

As the error indicated the problem must be with certificate. After many attempts, I found the current solution, which is to set VAULT_CACERT in vault_deployment.yml

env:

- name: VAULT_CACERT

value: "/etc/tls/ca.pem"

Test for peace of mind

So, the 2 Vault clusters now do “Auto-Unseal” for each other. The purpose is get a highly available Vault as service.

From user point of view, I setup test to check Vault status every 1 second with statping as below. It checks Vault health from Internet to Kubernetes’ Ingress controller then to the Vault cluster.

More details about the setup are in my article “Measure and Improve High Availability of Kubernetes Cluster During Reboot” (link).

Rolling update the deployment

As described in this article, the vault deployment combines “RollingUpdate”, “livenessProbe”, “readinessProbe” , “lifecycle-preStop”, and “PodDisruptionBudget”. It makes sure during the update, only one pod of the 3 get terminated and the newly created pod must get to “unsealed” state for next pod update.

In the many update to vault deployment that I’ve performed while writing this article, there is 0 failure detected.

Host direct reboot

While writing this article, I got one time host machine maintenance by the cloud provider. It took 2 machines down for 20 minutes, one has the haproxy and another one has a master node which Ingress Controller is also there. In this case, statping recorded 4 failures.

Whole Kubernetes cluster reboot

I use the method described in this article to test maintenance reboot scenario. I sequentially reboot all the nodes of the Kubernetes cluster, for example:

192.168.3.122 haproxy1.example.com haproxy1

192.168.2.17 haproxy2.example.com haproxy2

192.168.4.114 master1.example.com master1

192.168.5.58 master2.example.com master2

192.168.1.203 master3.example.com master3

192.168.3.72 worker1.example.com worker1

192.168.5.67 worker2.example.com worker2

192.168.7.180 worker3.example.com worker3

For each node, first drain the pods, reboot, wait till it’s back and lastly “uncordon” the node.

In this test, statping always recorded some 50x failures, around 2–6 failures per reboot cycle. It took me quite long time trying to figure out a solution. I tried the methods from this article, but it didn’t help. The discussion here explained Vault is not designed for zero-downtime. When the active Vault node is down, it takes time for the standby one to take over.

So in the source I put the best that I could do, for container vault, delay 6 seconds, and for container consul-vault-agent, delay 6 seconds then call ‘consul leave’. Please let me know if any improvement could be done.

# container vault

lifecycle:

preStop:

exec:

command: [

"sh", "-c",

# Introduce a delay to the shutdown sequence to wait for the

# pod eviction event to propagate.

"sleep 6",

] ...

# container consul-vault-agent lifecycle:

preStop:

exec:

command: [

"sh", "-c",

# shutdown vault first, then shutdown consul agent

"sleep 6 && consul leave"

]

Update 1/31/2020 I’d like to update that the availability of Vault is greatly improved after implementing highly available Consul Cluster. Read this article. In my most recent 2 cycles of “whole cluster reboot”, there were 0 and 1 error detected.

Worst case — Dead lock

What if both Vault clusters die at the same time?

I’ve hit the case once when I accidentally destroyed the Consul deployment. Although I could bring Consul back without data loss, both Vault clusters are unable to unseal.

—

Update: I’ve found another case which will bring down the Vault cluster for sure. As Vault depends on Consul as storage back end, if the Consul cluster is down, e.g. Consul enters a state that no leader exists, the Vault cluster enters all sealed state.

So it’s important to make sure the Consul cluster is highly available. I updated consul_statefulset.yml to add livenessProbe. When the consul cluster need rolling update, I found the consul cluster and vault clusters are all good.

However, if all the worker nodes do server reboot at the same time, the Vault clusters will go to dead lock state. This must be avoided.

—

Unfortunately I couldn’t find a way to recover, as both clusters stays in sealed mode forever… I hope in this case Vault could be unsealed with all the recovery keys, but not sure if this is possible. (one related post here)

$ kubectl get pod --all-namespaces -l app=vault

NAMESPACE NAME READY STATUS RESTARTS AGE

vault-unlock vault-unlock-5fc9c7cdd7-22cl4 1/2 Running 10 5h6m

vault-unlock vault-unlock-5fc9c7cdd7-pclv8 1/2 Running 11 38h

vault-unlock vault-unlock-5fc9c7cdd7-v6pvx 1/2 Running 11 38h

vault vault-5fc9c7cdd7-62qb6 1/2 Running 11 38h

vault vault-5fc9c7cdd7-gnfdj 1/2 Running 10 5h6m

vault vault-5fc9c7cdd7-nb8n7 1/2 Running 11 38h

So for this time, I had no choice but to rebuilt the 2 Vault clusters.

Solution for dead lock

Mutual Auto-Seal with backup

How to recover from the dead lock which should be rare but possible in theory(and it happened due to human error)?

Here the problem is both Vault clusters mutually depend on each other for unsealing. We must break the dependency, e.g. there should be a Vault cluster stays in “manual” unseal mode.

My idea is, as the diagram shows, to create a “Manual Unseal” Vault cluster in a second Kubernetes Cluster. This “manual unseal” Vault cluster has the same content as “Vault-unlock” while it’s in “manual unseal” mode.

High Level Steps:

Main Kubernetes cluster

Initialize “consul” Cluster Initialize “vault” Cluster Initialize “vault-unlock” Cluster Switch “vault” cluster to Auto-Unseal mode, with the help of “vault-unlock” cluster

Helper Kubernetes cluster

Create “consul-back” cluster Snapshot main cluster’s “consul” content and restore to “consul-back” cluster. Keep only “vault-unlock” K/V directory Create “vault-unlock” cluster, as the content is same as “vault-unlock” in main cluster, it could be unsealed with the same Unseal keys Keep “vault-unlock” in helper Kubernetes in this “manual unseal” mode, so it could always be unsealed manually, and it has a backup of the transit key to unseal “vault” in main cluster

Consul-back in helper Kubernetes cluster

Main Kubernetes cluster

1. Switch “vault-unlock” to Auto-Unseal mode, with the help of “vault” cluster

In case of dead lock

Make sure “vault-unlock” cluster in helper Kubernetes cluster is unsealed and reachable Delete “vault” deployment from main Kubernetes, switch to use “emergency unseal” with the help from “vault-unlock” in helper Kubernetes

kubectl -n vault create configmap vault --from-file=vault.json=config/vault_emergency.json -o yaml --dry-run | kubectl replace -f -

3. After “vault” cluster is unsealed, re-create “vault-unlock” cluster in main Kubernetes, and it will be “Auto-Unsealed” by “vault” cluster in main Kubernetes.

4. Switch “vault” cluster to use “vault-unlock” in main Kubernetes as “Auto-Unseal”

kubectl -n vault create configmap vault --from-file=vault.json=config/vault_autounseal.json -o yaml --dry-run | kubectl replace -f -

Todo: Figure out the TLS communication between main Kubernetes “vault” cluster and helper Kubernetes “vault-unlock” cluster. Currently TLS verification is skipped.

Update: Actually I found it’s super easy to do TLS verification. As the helper Kubernetes provides Vault service over Trafik Ingress, via url https://vault.backup.example.com, it’s using Let's Encrypt certificate.

So simply comment out 2 lines from vault_deployment.yml of “vault” cluster

#- name: VAULT_CACERT

# value: "/etc/tls/ca.pem"

Logs:

$ kubectl -n vault logs -f vault-6df998bb7c-5q7g8 -c vault

==> Vault server configuration:

Transit Address:

Transit Key Name: autounseal

Transit Mount Path: transit/

Api Address:

Cgo: disabled

Cluster Address:

Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")

Log Level: INFO

Mlock: supported: true, enabled: true

Storage: consul (HA available)

Version: Vault v1.1.3

Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb Seal Type: transitTransit Address: https://vault.b ackup.example.comTransit Key Name: autounsealTransit Mount Path: transit/Api Address: https://vault.vault.svc.cluster.local:8200 Cgo: disabledCluster Address: https://10.0.181.79:8201 Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")Log Level: INFOMlock: supported: true, enabled: trueStorage: consul (HA available)Version: Vault v1.1.3Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb ==> Vault server started! Log data will stream in below: 2019-07-03T04:38:10.435Z [INFO] core: stored unseal keys supported, attempting fetch

2019-07-03T04:38:10.466Z [INFO] core: vault is unsealed

2019-07-03T04:38:10.466Z [INFO] core.cluster-listener: starting listener: listener_address=[::]:8201

2019-07-03T04:38:10.466Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201

2019-07-03T04:38:10.466Z [INFO] core: entering standby mode

2019-07-03T04:38:10.469Z [INFO] core: unsealed with stored keys: stored_keys_used=1

Auto-Unseal token Refresh

After one month I found the auto-unseal failed to work.

It got “403 permission denied” when trying to unseal from the other vault.

It’s because we depend on a token with “autounseal” policy to get the job done. Remember in above steps:

# create a token with the policy, it's wrapped in wrapping_token

$ vault token create -policy="autounseal" -wrap-ttl=120

Key Value

--- -----

wrapping_token: s.AO55UYmKpT4TmPVn2AZ0UkNT

wrapping_accessor: 7N3PtIwhuAtjJZ2GvA9RM0iV

wrapping_token_ttl: 2m

wrapping_token_creation_time: 2019-06-25 19:24:46.345501602 +0000 UTC

wrapping_token_creation_path: auth/token/create

wrapped_accessor: B8kWfoLfXYOAxFI2P2TIduaI # get the token from wrapping_token

# as previous command shows, the wrapping_token is only valid for 120 seconds

$ VAULT_TOKEN="s.AO55UYmKpT4TmPVn2AZ0UkNT" vault unwrap

Key Value

--- -----

token s.LvLFn4InVdIorAFS5E9j6xd3

token_accessor KST0fJN4xMd2Yaztrmya8SNx

token_duration 768h

token_renewable true

token_policies ["autounseal" "default"]

identity_policies []

policies ["autounseal" "default"] # this is the token to be used in next step

token s.LvLFn4InVdIorAFS5E9j6xd3

The token to do the auto-unseal will expire after 768h, e.g. 32 day. As we use the token in the configmap (vault_autounseal.json), it’s unable to access remote after the token has expired.

In “vault_autounseal.json”, even “disable_renewal” is defined as “false”, the token still expired. My understanding is the “renew” didn’t happen if the token isn’t used at all for 32 days, e.g. for the time period, “auto-unseal” didn’t happen.

To resolve the problem there might be a few approaches

Extend TTL of the token

Periodically renew the token

Periodically re-generate a token and update the configmap

Summary

In this article I described my approach to create a highly available Vault service. To achieve the goal, I setup 2 Vault clusters to do mutual Auto-Unseal. One of them is to provide the Vault service to external users, and the other one is dedicated to do the “Unlock” job. To avoid the dead lock which could happen in extreme case, a third Vault cluster is acting as the backup of the “Unlock” Vault.

All source codes are shared in github.

Thanks for reading.

Reference:

Auto-unseal using Transit Secrets Engine

transit Seal configurations

vault Server configurations

tcp Listener configurations

Update 1/31/2020

Thanks for the claps and responses. I’d like to add more details about the deadlock, about the possible scenarios causing the deadlock and methods to get out of deadlock.

The idea behind mutual auto-unseal is if we have at least one unsealed Vault instance in both Vault clusters, both Vault clusters could recover to fully unsealed state.

The deadlock will happen for 100% sure if there is none unsealed instance.

So far I’ve seen these scenarios causing deadlock:

Kubernetes cluster down

In a single master Kubernetes cluster, rebooting the master host machine causes the Vault clusters fully sealed.

Solution: If both vault and vault-unlock are in same Kubernetes cluster, use HA Kubernetes with multiple masters. Alternatively, put vault and vault-lock to separate Kubernetes cluster. Avoid dangerous operations on both Kubernetes cluster at the same time.

Consul cluster destroyed

By accident or whatever reason, when the Consul cluster is down, the Vault depending on it is down too.

Solution: Use separate Consul cluster for vault and vaul-unlock. Don’t use same Consul cluster as in the demo.

Consul cluster leadership lost

There is some tolerance for leadership lost for couple seconds. I’m not exactly sure how long it is.

Solution: Make Consul cluster highly available. Read my article here.

The host machines serving the Vault/Consul down at the same time

There might be many reasons. I’ve once used Terraform to resize the hosts and they went down at the same time…

Solution: It depends on the infrastructure. Try all the methods to avoid the hosts go down at the same time.

These solutions are important for both avoiding deadlock and achieving Higly Available Vault service.

Is it possible to use alternative unseal method in case of dead lock?

The process of unseal is to use various methods to provide the master key to Vault. Vault need the master key to recover its encryption key. The encryption key is then used to decrypt its data.

The methods to provide the master key are:

Several unseal keys from ‘vault operator init’.

Auto unseal method listed here, from external trusted services, or ‘transit’, from another trusted Vault.

This question is actually like this, if the Vault cluster is currently using unseal method A, is it possible to unseal it with method B when it is sealed?

From my understanding, it’s not possible. Of course Hashicorp has the final call.

This is because the method A is currently the only method to provide the master key, as the only root of trust.

If “Vault transit” is the current unseal method, the encryption key is encrypted by a key in transit secret engine. It’s not the same one as the “several unseal keys” sums up.

It’s only possible to change the “unseal method” after Vault is unsealed, as the “Seal Migration” shows, to change “unseal method”, the command is “vault operator unseal -migrate”.

It’s not clear to me if it’s possible to change unseal method between two of the auto-unseal methods, for example, “Vault transit -> HSM Pkcs11”, without going through “Vault transit -> Shamir -> HSM”. If anyone is interested please try and let me know.

Solution to dead lock

So my conclusion is, currently the only method to recover from dead lock is to use the exactly same ‘Unseal method’, with exactly same key.

Unless Hashicorp provides a method to export the raw key stored in “Unseal method” and allow it be provided to Vault. But it looks like a super insecure idea.

Or consider this question, how to unseal Vault when its “Unseal method A” is impossible?

Like in this discussion, how to unseal Vault if GCP KMS unseal key is lost, where “Unseal method” is GCP KMS in the case?

Same as everything else, the solution is to have a backup. The method of backup might be different for different providers.

Update 3/8/2020

I’d like to say thanks for the feed backs. One reader reminded me about the method to backup a transit key via API.

I gave it a try and here is the method to backup/restore a transit key.

# Create a transit key

$ vault write -f transit/keys/transit_test

Success! Data written to: transit/keys/transit_test # Read transit key

$ vault read transit/keys/transit_test

Handling connection for 8200

Key Value

--- -----

allow_plaintext_backup false

deletion_allowed false

derived false

exportable false

name transit_test # To allow backup, change the key configuration

$ vault write transit/keys/transit_test/config exportable=true

$ vault write transit/keys/transit_test/config allow_plaintext_backup=true # Check the config again

$ vault read transit/keys/transit_test

Key Value

--- -----

allow_plaintext_backup true

deletion_allowed false

derived false

exportable true

name transit_test # Now do backup

$ vault read transit/backup/transit_test

Key Value

--- -----

backup eyJwb2xpY3kiO......VeryLooooongStrinng # After backup, the key has a mark as "it has been backuped"

$ vault read transit/keys/transit_test

Key Value

--- -----

backup_info map[time:2020-03-08T06:27:13.859635651Z version:1]

$ vault write transit/restore/transit_restore # Do restore$ vault write transit/restore/transit_restore @transit_test .json # The restored key has a mark as "it is restored"

$ vault read transit/keys/transit_restore

Key Value

--- -----

allow_plaintext_backup true

backup_info map[time:2020-03-08T06:27:13.859635651Z version:1]

deletion_allowed false

derived false

exportable true

name transit_restore

restore_info map[time:2020-03-08T06:42:45.174879348Z version:1]

I think this might be used as a backup method. When the dead lock happens, create a new instance of Vault and restore the key into it. Then point the “emergency unseal” to the new Vault to resolve the dead lock. Things to consider:

To backup, we have to modify the key’s configuration “exportable” and “allow_plaintext_backup”. Once they are set to true, they could not be set back to false. And the restored key inherited these configurations.what’s the security implication of these 2 configurations change?

How to secure the backup key value? What is the security implication if the backup is leaked?

I currently don’t have answers.