Use Vault-CRD with NGINX Ingress Controller Part I

Last week I announced my OpenSource project Vault-CRD to share secrets that are Stored in HashiCorp Vault with Kubernetes.

Here you can find the corresponding Blog post: Vault-CRD

In this Blog post I’ll show you how to use the Vault-CRD to dynamically update NGINX Ingress Controller if the certificate changes in Vault.

The Blog post is splitted into two parts.

Part I: The first part will show you how to use a PKI Secret Engine to generate NGINX Ingress Controller certificates.

Part II: The second part of this tutorial will show you how to setup and use Key-Value type for certificates that NGINX Ingress Controller can use

Setup

For a simple setup I’ve deployed a NGINX container in Kubernetes:

apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: app: nginx name: nginx namespace: vault-crd spec: replicas: 1 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:alpine ports: - name: http containerPort: 80 restartPolicy: Always --- apiVersion: v1 kind: Service metadata: labels: app: nginx name: nginx namespace: vault-crd spec: selector: app: nginx type: ClusterIP ports: - port: 80 protocol: TCP targetPort: 80

As you can see it’s a nginx:alpine-image that will be deployed as a deployment with 1 replication and a service that is connected via the selector to it. The Ingress for exposing this container to the world is also very easy:

apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: nginx name: ingress namespace: vault-crd spec: rules: - host: test-url.koudingspawn.de http: paths: - backend: serviceName: nginx servicePort: 80 path: / tls: - secretName: ingress-ssl-certificate hosts: - test-url.koudingspawn.de

Ingress will be configured to watch for test-url.koudingspawn.de Hosts in HTTP request and will proxy requests to the nginx service on port 80. Now the interesting part is the tls section inside the ingress definition. If you apply this now to Kubernetes the Ingress controller will fail with an exception, that there is no secret secret/ingress-ssl-certificate:

W0505 18:14:02.283761 5 backend_ssl.go:48] error obtaining PEM from secret vault-crd/ingress-ssl-certificate: error retrieving secret vault-crd/ingress-ssl-certificate: secret vault-crd/ingress-ssl-certificate was not found

Now comes the interresting part of Vault-CRD.

Share the Certificate with Kubernetes via PKI

In Vault there is a build in Secret Type PKI. This one is a complete certificate authority we can use to generate self signed certificates. For more information how to generate a PKI please read the instructions from HashiCorp Vault. Or you can use the simple one I’ve created for this example:

$ mountPath="testpki" $ vault secrets enable -path=$mountPath -description=$mountPath pki $ vault secrets tune -max-lease-ttl=8760h $mountPath $ vault write $mountPath/root/generate/internal \ common_name=$mountPath \ ttl=8760h $ vault write $mountPath/roles/testrole \ allow_any_name=true \ max_ttl=8760h

Now we can generate our Vault-CRD to automatically issue certificates:

apiVersion: "koudingspawn.de/v1" kind: Vault metadata: name: ingress-ssl-certificate namespace: vault-crd spec: path: "testpki/issue/testrole" type: "PKI" pkiConfiguration: commonName: "test-url.koudingspawn.de" ttl: "720h"

What will happen now

As you can see inside the Vault-CRD logs, that it detects a new Vault-resource in namespace vault-crd with name ingress-ssl-certificate.

Now Vault-CRD tries to access your Vault and issues a new certificate with common name test-url.koudingspawn.de and a time to life of 30 days (720h). After this it generates a Secret with the same name as it’s own, in the vault-crd namespace. After this there is now a Secret that the Ingress Controller can use to expose your service to the world.

The interresting part is here the naming. I’ve named the ingress resource tls secret name with ingress-ssl-certificate and the Vault resource also ingress-ssl-certificate . And as already told Vault-CRD now generates a secret with the same name.

Now you’ll see that the URL can be accessed and has a specific fingerprint for the certificate.

$ openssl s_client -servername test-url.koudingspawn.de -connect test-url.koudingspawn.de:30443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin SHA1 Fingerprint=62:98:DE:39:66:7C:A4:72:DF:FC:80:3A:71:27:3F:AD:E7:87:4A:98

Refresh of a certificate

If the 30 days are near to expire Vault-CRD will detect this and issues a new Certificate.

When a refresh happens NGINX will detect the changed secret and updates itself to use the new certificate:

I0505 18:35:02.732775 5 store.go:375] secret vault-crd/ingress-ssl-certificate was updated and it is used in ingress annotations. Parsing... I0505 18:35:02.736128 5 backend_ssl.go:59] updating secret vault-crd/ingress-ssl-certificate in the local store I0505 18:35:02.736653 5 controller.go:168] backend reload required I0505 18:35:02.914816 5 controller.go:177] ingress backend successfully reloaded... I0505 18:35:28.340035 5 backend_ssl.go:173] updating local copy of ssl certificate vault-crd/ingress-ssl-certificate with missing intermediate CA certs I0505 18:35:28.340265 5 controller.go:168] backend reload required I0505 18:35:28.479019 5 controller.go:177] ingress backend successfully reloaded...

In Part two of this tutorial I’ll show you how to use Vault-CRD with certificates that are not from a Vault PKI Secret Engine. Part II