A wildcard certificate can secure any number of subdomains of a base domain (e.g. *.example.com). This allows to use a single certificate and key pair for a domain and all of its subdomains, which can make HTTPS deployment significantly easier.

Let's Encrypt wildcard certificates support went live in March 2018.

In this blog post you will learn how to setup Kubernetes Ingress controller with Heptio Contour, automate the management and issuance of wildcard TLS certificates with Jetstack Cert-Manager and sync the TLS certs across different namespaces with AppsCode Kubed.

Pre-Requisites

You need to have Kubernetes cluster available, if you don't have it, follow one the docs below to set it up on:

Helm is installed in your Kubernetes cluster.

Contour Ingress Controller

Ok, first we need to install Kubernetes Ingress controller.

As there is no Helm chart for Heptio Contour, I wrote the chart and stored it in my Helm repository. To be able to use it you need to add my Chart repo to Helm:

$ helm repo add rimusz https://helm-charts.rimusz.net $ helm repo up

Ok, let's install contour chart:

helm install --name contour rimusz/contour

And to check that ingress controller is running:

$ kubectl --namespace=heptio-contour get pods -l "app=contour" NAME READY STATUS RESTARTS AGE contour-5d7f6fc8bd-nv474 2/2 Running 0 20s

It may take a few minutes for the LoadBalancer IP to be available. You can watch the status with:

$ kubectl get svc --namespace heptio-contour -w contour NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE contour LoadBalancer 10.64.12.134 <pending> 80:30321/TCP,443:32400/TCP 34s contour LoadBalancer 10.64.12.134 35.238.152.138 80:30321/TCP,443:32400/TCP 46s

Now please update your domain DNS record with the External IP .

Nice, we got our Ingress Controller installed.

TLS Cert-Manager

We are going to install cert-manager from Jetstack repo:

$ helm repo add jetstack https://charts.jetstack.io $ kubectl apply \ -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml $ helm install --name cert-manager --namespace cert-manager stable/cert-manager

And we check it is running:

$ kubectl get pods -n cert-manager NAME READY STATUS RESTARTS AGE cert-manager-5f8db6f6c4-jjzjc 1/1 Running 0 22s cert-manager-webhook-85dd96d87-jsfjk 1/1 Running 0 22s cert-manager-webhook-ca-sync-vwt57 0/1 Completed 0 22s

Awesome.

Generating wildcard certs with Cert-Manager

For this blog post I'm using CloudFlare (sorry I'm a big fan of their services) as DNS01 Challenge Provider , check for other supported ones here.

Now we need to create a secret with CloudFlare Global API Key, Cert-Manager Issuer with DNS1 Challenge Provider, which will use that secret and the Cert-Manager Certificate which will save the wildcard cert of *.example.com to secret example-com-tls :

$ cat <<EOF | kubectl create -f - --- apiVersion: v1 kind: Secret metadata: name: cloudflare-api-key namespace: cert-manager type: Opaque data: api-key.txt: <copy here base64 encoded CloudFlare API Key> --- apiVersion: certmanager.k8s.io/v1alpha1 kind: Issuer metadata: name: letsencrypt-staging namespace: cert-manager spec: acme: server: https://acme-staging-v02.api.letsencrypt.org/directory email: [email protected] # Name of a secret used to store the ACME account private key privateKeySecretRef: name: letsencrypt-staging # ACME DNS-01 provider configurations dns01: # Here we define a list of DNS-01 providers that can solve DNS challenges providers: - name: cf-dns cloudflare: email: [email protected] # A secretKeyRef to a cloudflare api key apiKeySecretRef: name: cloudflare-api-key key: api-key.txt --- apiVersion: certmanager.k8s.io/v1alpha1 kind: Certificate metadata: name: example-com namespace: cert-manager spec: secretName: example-com-tls issuerRef: name: letsencrypt-staging commonName: '*.example.com' acme: config: - dns01: provider: cf-dns domains: - '*.example.com' EOF

Note: Please update the example template above with your CloudFlare API key , email address and domain name .

If all went successfully you should see your wildcard cert example-com-tls :

$ kubectl -n cert-manager get secret NAME TYPE DATA AGE cert-manager-cert-manager-token-whctf kubernetes.io/service-account-token 3 4h cloudflare-api-key Opaque 1 17m default-token-cfmln kubernetes.io/service-account-token 3 4h letsencrypt-staging Opaque 1 37m example-com-tls kubernetes.io/tls 2 12m

Yay.

Sync wildcard certs between Kubernetes namespaces

Kubernetes best practices recommend to install app per namespace, but our wildcard cert example-com-tls is stored in the cert-manager namespace, so we need to sync it across namespaces when the cert gets reissued or new app in the new namespace gets added.

For that we are going to use a nice tool from AppsCode Kubed. Check their other tools, you could find more useful ones for you.

Kubed can be installed via Helm using the chart from AppsCode Charts Repository:

$ helm repo add appscode https://charts.appscode.com/stable/ $ helm repo update $ helm install appscode/kubed --name kubed --namespace kube-system \ --set apiserver.enabled=false \ --set config.clusterName=my_staging_cluster

Note: Set cluster-name to something meaningful to you, say, prod, prod-us-east, qa, etc. Which also enables kubed's ConfigSyncer feature.

To verify that Kubed has started, run:

$ kubectl --namespace=kube-system get deployments -l "release=kubed, app=kubed" NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE kubed-kubed 1 1 1 1 15s

Cool.

Ok, now let's create a few namespaces where we are going to sync our wildcard cert to:

$ kubectl create namespace demo1 $ kubectl create namespace demo2

Now we need to put the annotation as per docs to our wildcard secret, so kubed knows what to sync across namespaces:

$ kubectl annotate secret example-com-tls -n cert-manager kubed.appscode.com/sync="app=kubed" secret "example-com-tls" annotated

From now on kubed will start syncing secret example-com-tls to only namespaces which have label-selector app=kubed .

Let's annotate namespace demo1 :

$ kubectl label namespace demo1 app=kubed

Now we should be able to see the secret in the namespace demo1 :

$ kubectl -n demo1 get secret example-com-tls NAME TYPE DATA AGE example-com-tls kubernetes.io/tls 2 2h

And if we check namespace demo2 we will not see the example-com-tls there:

$ kubectl -n demo2 get secret example-com-tls Error from server (NotFound): secrets "example-com-tls" not found

Putting the annotation on to demo2 namespace will trigger an instant secret sync as well.

Now you can install web apps into demo1 and demo2 namespaces, and apps be able to use the same wildcard cert for their ingress rules.

What's next

Moving forward to production releases and receiving the real TLS certs from Let's Encrypt add new Issuer and Certificate as per example below:

$ cat <<EOF | kubectl create -f - --- apiVersion: certmanager.k8s.io/v1alpha1 kind: Issuer metadata: name: letsencrypt-prod namespace: cert-manager spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: [email protected] # Name of a secret used to store the ACME account private key privateKeySecretRef: name: letsencrypt-prod # ACME DNS-01 provider configurations dns01: # Here we define a list of DNS-01 providers that can solve DNS challenges providers: - name: cf-dns-prod cloudflare: email: [email protected] # A secretKeyRef to a cloudflare api key apiKeySecretRef: name: cloudflare-api-key key: api-key.txt --- apiVersion: certmanager.k8s.io/v1alpha1 kind: Certificate metadata: name: example-com-prod namespace: cert-manager spec: secretName: example-com-tls-prod issuerRef: name: letsencrypt-prod commonName: '*.example.com' acme: config: - dns01: provider: cf-dns-prod domains: - '*.example.com' EOF

Happy secure web content serving for you :)

Wrap Up