As those of you who are following us here at Banzai Cloud may or may not be aware, we are in the middle of releasing/certifying our own Kubernetes distribution — Pipeline Kubernetes Engine (PKE). PKE will be orchestrated the same way as other providers already supported by Pipeline, and will benefit from/inherit those features of the Banzai Cloud Pipeline platform that you already know and love.

If you’re interested in learning more about PKE and our vision for buidling multi and hybrid cloud managed (application) environments, please read this post

PKE has a seamless and automated bootstrap procedure, which requires a PKI infrastructure. One of the jigsaw components we use in this process is kubeadm. Kubeadm can manage PKI certificates by itself but can also integrate with preexisting PKI infrastructure.

The Pipeline platform already contains a secret management engine. We support a lot of very different secret types with that engine — from simple username/password pairs to TLS certificates. It’s possible not only to store those secrets with the engine, but to generate them as well (so long as no static input has been given). The supporting code relies heavily on Pipeline and uses Bank-Vaults to work.

Banzai Cloud PKE Kubernetes distribution clusters use our own PKI infrastructure, which is backed by Vault. This makes it possible among other obvious benefits, to seamlessly renew certificates, and to rebuild a cluster during disaster recovery or during a migration.

In the cluster setup we follow the Single Root CA arrangement, for which we have created a new secret type called distribution that holds all the certificates necessary during the kubeadm bootstrapping process. In this setup you can create a single root CA for the cluster, controlled by an administrator (which is Pipeline + Vault, in our case). This root CA can then create multiple intermediate CAs and delegate all further certificate creation to Kubernetes itself.

The required certificates are:

path Default CN description ca.crt,key kubernetes-ca Kubernetes general CA etcd/ca.crt,key etcd-ca For all etcd-related functions front-proxy-ca.crt,key kubernetes-front-proxy-ca For the front-end proxy

For each cluster or distribution type, "banzaicloud" uses a separate Vault PKI secret engine mounted to a different path (there are four of these), which is something Vault’s architecture requires us to do — for more details check out this issue as well as the Vault documentation. One secret engine is necessary for the cluster’s root CA and three for the intermediate CAs.

Example of how to call the Secret API and to generate a kubeadm TLS bundle for PKE 🔗︎

This example assumes that you already have a fully operational Pipeline instance running, in this case, on your machine. If you need some help doing that, follow along with our developer documentation.

1 # Instruct Pipeline to create a TLS bundle secret for our "banzai" distribution cluster 2 3 $ curl -H "Authorization: Bearer $TOKEN " \ 4 -H "Content-Type: application/json" \ 5 -d "{\"type\":\"distribution\", \"name\":\"banzai\", \"values\":{\"clusterUid\": \"12-345\"}}" \ 6 http://localhost:9090/pipeline/api/v1/orgs/1/secrets | jq 7 8 { 9 "name" : "banzai" , 10 "type" : "distribution" , 11 "id" : "26a8dab0bff77fe1821477c0e0b7c09883419b136ae6a745e85a01af664d2db3" , 12 "updatedAt" : "2019-01-24T12:48:11.081042376Z" , 13 "updatedBy" : "bonifaido" , 14 "version" : 1 15 } 16 17 # Check that Vault has mounted all the specific PKI engines 18 19 $ vault secrets list 20 Path Type Accessor Description 21 ---- ---- -------- ----------- 22 clusters/12-345/pki/ca/ pki pki_fb4f4c0a root PKI engine for cluster 12-345 23 clusters/12-345/pki/etcd-ca/ pki pki_041fb1d5 etcd-ca intermediate PKI engine for cluster 12-345 24 clusters/12-345/pki/kubernetes-ca/ pki pki_04aeb3fd kubernetes-ca intermediate PKI engine for cluster 12-345 25 clusters/12-345/pki/kubernetes-front-proxy-ca/ pki pki_e0a64032 kubernetes-front-proxy-ca intermediate PKI engine for cluster 12-345 26 cubbyhole/ cubbyhole cubbyhole_1561fda0 per-token private secret storage 27 database/ database database_32607d87 MySQL Database secret engine. 28 identity/ identity identity_0f057f0d identity store 29 secret/ kv kv_df66d275 key/value secret storage 30 sys/ system system_0e6bb73f system endpoints used for control, policy and debugging

Kubeadm Singe Root CA setup 🔗︎

By default, kubeadm uses the /etc/kubernetes/pki directory to store its own, on-the-fly, self-signed certificates. Fortunately, kubeadm is intelligent enough to use preexisting CA certificates, and will skip the generation step if it does. All we need to do is to place our intermediate certificates into the given structure by calling Pipeline’s Secret API.

1 #### Gathering generated intermediate CA certificates from Pipeline 2 3 $ mkdir -p /etc/kubernetes/pki/etcd/ 4 $ cd /etc/kubernetes/pki/ 5 $ certs = ` curl -H "Authorization: Bearer $TOKEN " \ 6 -H "Content-Type: application/json" \ 7 http://localhost:9090/pipeline/api/v1/orgs/1/secrets/26a8dab0bff77fe1821477c0e0b7c09883419b136ae6a745e85a01af664d2db3 ` 8 $ echo " $certs " | jq -r .values.kubernetesCaCert > ca.crt 9 $ echo " $certs " | jq -r .values.kubernetesCaKey > ca.key 10 $ echo " $certs " | jq -r .values.frontProxyCaCert > front-proxy-ca.crt 11 $ echo " $certs " | jq -r .values.frontProxyCaKey > front-proxy-ca.key 12 $ echo " $certs " | jq -r .values.etcdCaCert > etcd/ca.crt 13 $ echo " $certs " | jq -r .values.etcdCaKey > etcd/ca.key 14 $ unset certs 15 16 #### This will result in the following directory structure: 17 18 $ tree /etc/kubernetes/pki/ 19 /etc/kubernetes/pki/ 20 |-- ca.crt 21 |-- ca.key 22 |-- etcd 23 | |-- ca.crt 24 | ` -- ca.key 25 |-- front-proxy-ca.crt 26 ` -- front-proxy-ca.key

It is time to call kubeadm init and continue with the normal init workflow to setup a Kubernetes control plane on the first Kubernetes master node.

Future work 🔗︎

If you would like to have more control over your PKI infrastructure during the kubeadm bootstrap process, it’s advisable that you ‘bring your own’, which can be a difficult to generate and tedious maintain. However, with Pipeline this process is automated so that all of your kubeadm orchestrated clusters will have their own isolated PKI infrastructure, allowing you full control. In forthcoming blogposts we will describe how our PKE Kubernetes distribution is further configured during the provisioning process as managed by Pipeline.

About Banzai Cloud Pipeline 🔗︎

Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.