A quick note on keeping secrets in SCM

You have probably have your own solution to this little problem but I’d like to mention it all the same, if only in the interests of security.

sops from the folks at Mozilla is a great little tool for encrypting various file formats, including YAML. From the website:

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP.

We use it to encrypt our kubernetes secrets, thus allow us to keep them in git with using plaintext. Using the 01-clouddns-service-account.yaml from above as an example, the encrypted file looks like this:

$ cat cert-manager-utils/01-clouddns-service-account.yaml

apiVersion: v1

data:

devops-dns-admin_token: ENC[AES256_GCM,data:<redacted>,type:str]

kind: Secret

metadata:

creationTimestamp: null

name: clouddns-service-account

namespace: cert-manager

sops:

kms:

— arn: <redacted>

created_at: ‘2019–05–23T15:56:06Z’

enc: <redacted>

gcp_kms: []

azure_kv: []

lastmodified: ‘2019–05–23T18:40:17Z’

mac: ENC[AES256_GCM,data:<redacted>,type:str]

pgp: []

encrypted_suffix: _token

version: 3.2.0

Unencrypted, the file looks like:

$ sops -d cert-manager-utils/01-clouddns-service-account.yaml

apiVersion: v1

data:

devops-dns-admin_token: <redacted>

kind: Secret

metadata:

creationTimestamp: null

name: clouddns-service-account

namespace: cert-manager

One added bonus is the ability to combine this with kubectl apply and process substitution to provide on the fly decryption:

kubectl apply -f <(sops -d cert-manager-utils/01-clouddns-service-account.yaml)

Although, in the end, I really don’t mind which method is chosen as long as people are using encryption :-).

Ensure correct DNS entries exist

This was covered in my earlier post during the installation of Jenkins X (search for “You will need to create DNS records”).

Just in case you are here with a pre-installed Jenkins X platform, please ensure that you have DNS records in place for your domain. Here an excerpt from my earlier post:

You will need to create DNS records (Type A)pointing to:

YOUR-DOMAIN > loadbalancer ip

> loadbalancer ip *.YOUR-DOMAIN > loadbalancer ip

You can find the loadbalancer ip with:

kubectl get svc jxing-nginx-ingress-controller -n kube-system -o'jsonpath={ .status.loadBalancer.ingress[0].ip

Installing Kubernetes Replicator

Next we’ll install the kubernetes-replicator. Unfortunately, at the time of writing there does not seem to be an official helm chart. It is, however, very easy to install so here is a quick and dirty snippet to install v1.0.0.

Just copy and paste into a shell:

Now the replicator is installed, we can add the wildcard certificates.

Creating the wildcard certificates

First we needed DNS admin account credentials which, depending on your domain registrar, can be created as follows:

Google CloudDNS

Cloudflare (section “Create the CloudFlare DNS issuer”)

TIP: You are not forced to use multiple issuers, but having both specified allows you to use the same config files for multiple domains.

Once the credentials are created, we can create the necessary kubernetes resources. The resource files have been split into 3 groups:

the DNS admin secrets

the cluster issuers

the certificates themselves

$ tree cert-manager-utils

cert-manager-utils

├── 01-clouddns-service-account.yaml # cert-manager namespace

├── 01-cloudflare-api-key.yaml # cert-manager namespace

├── 02-clusterissuer-prod.yaml # kube-system namespace

├── 02-clusterissuer-staging.yaml # kube-system namespace

├── 03-cluster-certificate-prod.yaml # kube-system namespace

└── 03-cluster-certificate-staging.yaml # kube-system namespace

There are files for both “staging” and “prod”. These use the LetsEncrypt staging and prod environments respectively.

I highly recommend using the staging environment first to make sure the configuration is correct before moving over to the LetsEncrypt prod environment.

The DNS Secrets

The names below are just a suggestion by the way. You are, of course, free to choose your own names for the secret keys used below such as “devops-dns-admin_token”, etc.

cert-manager-utils/01-clouddns-service-account.yaml

—- apiVersion: v1

data:

devops-dns-admin_token: REPLACE_WITH_BASE64_ENCODED_JSON

kind: Secret

metadata:

creationTimestamp: null

name: clouddns-service-account

namespace: cert-manager

cert-manager-utils/01-cloudflare-api-key.yaml

—- apiVersion: v1

data:

api-key.txt: REPLACE_WITH_BASE64_ENCODED_API_KEY

kind: Secret

metadata:

name: cloudflare-api-key

namespace: cert-manager

The LetsEncrypt Staging Files

These resources will create a wildcard certificate, issued by the LetsEncrypt staging environment.

cert-manager-utils/02-clusterissuer-staging.yaml

---

kind: ClusterIssuer

metadata:

name: letsencrypt-staging-dns

spec:

acme:

server:

email:

privateKeySecretRef:

name: letsencrypt-staging

dns01:

providers:

- name: clouddns

clouddns:

serviceAccountSecretRef:

name: clouddns-service-account

key: devops-dns-admin_token

project: REPLACE_WITH_your-google-cloud-project

- name: cloudflare

cloudflare:

apiKeySecretRef:

key: api-key.txt

name: cloudflare-api-key

email:

apiVersion: certmanager.k8s.io/v1alpha1kind: ClusterIssuermetadata:name: letsencrypt--dnsspec:acme:server: https://acme-staging-v02.api.letsencrypt.org/directory email: REPLACE_WITH_your.email@your.domain.com privateKeySecretRef:name: letsencrypt-dns01:providers:- name: clouddnsclouddns:serviceAccountSecretRef:name: clouddns-service-accountkey: devops-dns-admin_tokenproject: REPLACE_WITH_your-google-cloud-project- name: cloudflarecloudflare:apiKeySecretRef:key: api-key.txtname: cloudflare-api-keyemail: REPLACE_WITH_your.email@your.domain.com cert-manager-utils/03-cluster-certificate-staging.yaml

--- apiVersion: certmanager.k8s.io/v1alpha1

kind: Certificate

metadata:

name: jxing-nginx-ingress-controller-wildcard-staging

namespace: kube-system

spec:

secretName: jxing-nginx-ingress-controller-wildcard-staging-tls

issuerRef:

name: letsencrypt-staging-dns

kind: ClusterIssuer

commonName: REPLACE_WITH_YOUR_DOMAIN

dnsNames:

- REPLACE_WITH_YOUR_DOMAIN

- "*.REPLACE_WITH_YOUR_DOMAIN"

acme:

config:

- dns01:

provider: cloudflare

domains:

- REPLACE_WITH_YOUR_DOMAIN

- "*.REPLACE_WITH_YOUR_DOMAIN"

- dns01:

provider: clouddns

domains:

- REPLACE_WITH_YOUR_DOMAIN

- "*.REPLACE_WITH_YOUR_DOMAIN"

The LetsEncrypt Prod Files

The only difference to the staging files here is the value of LetsEncrypt’s “acme.server” along with some resource names (prod vs staging to keep the two resource groups apart).

cert-manager-utils/02-clusterissuer-prod.yaml

---

kind: ClusterIssuer

metadata:

name: letsencrypt-prod-dns

spec:

acme:

server:

email:

privateKeySecretRef:

name: letsencrypt-prod

dns01:

providers:

- name: clouddns

clouddns:

serviceAccountSecretRef:

name: clouddns-service-account

key: devops-dns-admin_token

project: REPLACE_WITH_your-google-cloud-project

- name: cloudflare

cloudflare:

apiKeySecretRef:

key: api-key.txt

name: cloudflare-api-key

email: apiVersion: certmanager.k8s.io/v1alpha1kind: ClusterIssuermetadata:name: letsencrypt--dnsspec:acme:server: https://acme-v02.api.letsencrypt.org/directory email: REPLACE_WITH_your.email@your.domain.com privateKeySecretRef:name: letsencrypt-dns01:providers:- name: clouddnsclouddns:serviceAccountSecretRef:name: clouddns-service-accountkey: devops-dns-admin_tokenproject: REPLACE_WITH_your-google-cloud-project- name: cloudflarecloudflare:apiKeySecretRef:key: api-key.txtname: cloudflare-api-keyemail: REPLACE_WITH_your.email@your.domain.com cert-manager-utils/03-cluster-certificate-prod.yaml

--- apiVersion: certmanager.k8s.io/v1alpha1

kind: Certificate

metadata:

name: jxing-nginx-ingress-controller-wildcard-prod

namespace: kube-system

spec:

secretName: jxing-nginx-ingress-controller-wildcard-prod-tls

issuerRef:

name: letsencrypt-prod-dns

kind: ClusterIssuer

commonName: REPLACE_WITH_YOUR_DOMAIN

dnsNames:

- REPLACE_WITH_YOUR_DOMAIN

- "*.REPLACE_WITH_YOUR_DOMAIN"

acme:

config:

- dns01:

provider: cloudflare

domains:

- REPLACE_WITH_YOUR_DOMAIN

- "*.REPLACE_WITH_YOUR_DOMAIN"

- dns01:

provider: clouddns

domains:

- REPLACE_WITH_YOUR_DOMAIN

- "*.REPLACE_WITH_YOUR_DOMAIN"

Putting it all together

Using a little bash function we can put everything together with:

which can then be used á la:

add_certificates staging

or

add_certificates prod

After a few minutes, you should see something like this:

Using wildcard certs in preview environments

Now that we have configured all the necessary components, adding TLS to a preview environment is surprisingly simple.

Check exposecontroller version

Ensure the version in your preview requirements.yaml is 2.3.111 or higher.

See this example.

Add empty secret to replicate your wildcard certificate

Add an empty secret as described here. The kubernetes-replicator will detect this secret and copy the real secret into the preview namespace.

See this example.

Edit exposecontroller config to add TLS

Changes will be made to the following attributes:

tlsacme: true

urltemplate: "\"{{.Service}}-{{.Namespace}}.{{.Domain}}\""

tlsSecretName: "jxing-nginx-ingress-controller-wildcard-prod-tls"

See this example.

Success!

And that is it! Each preview will have TLS enabled and be using our wildcard certificate.

Congratulations!

Optional Bonus Setting

nginx has another nice little option to set a “default-ssl-certificate” for a given ingress controller.

This option, when set, adds a default ssl certificate as a fallback for any ingress requests, thus removing the need to set spec.tls.secretName in the ingress configuration.

Currently, you need to set the tlsSecretName in the exposecontroller config, meaning the above trick cannot be used. However, a huge thanks to Vincent Behar for adding the option in the first place.

If you do wish to add the default ssl certificate to the nginx controller, here is a little bash snippet to do so.