Automating Highly Available Kubernetes and external ETCD cluster setup with terraform and kubeadm on AWS.

Today I am going to show how you can fully automate the advanced process of setting up the highly available k8s cluster in the cloud. We will go through a set of terraform and bash scripts which should be sufficient enough for you to literally just run terraform plan/apply to get your HA etcd and k8s cluster up and running without any hassle around.

Part 0 – Intro.

Part 1 – Setting up HA ETCD cluster.

Part 2 – The PKI infra

Part 3 – Setting up k8s cluster.

Part 0 – Intro.

If you do a short research on how to setup k8s cluster you may find quite a lot of ways this could be achieved.

But in general, all this ways could be grouped into 3 types:

1) No setup

2) Easy Set up

3) Advanced Set up

4) Hard way

By No setup I simply mean something like EKS, it is a managed service, you don’t need to maintain or care about details while AWS will do all for you. Never used it can’t say much on that one.

Easy setup, tools like kops and alike make it quite easy – couple commands run kinda setup:

kops ~]$ kops create cluster \ --name=k8s.ifritltd.net --state=s3://kayan-kops-state \ --zones="eu-west-2a" --node-count=2 --node-size=t2.micro --master-size=t2.micro --dns-zone=k8s.ifritltd.net --cloud aws

All you need is setup s3 bucket and dns records and run the command above which I described two years ago in this article

The downside is first of all it is mainly only for AWS, and generates all AWS resources as it wants, so lets say it would generate security groups, asg, etc in it’s own way which means

if you already have terraform managed infra with your own rules, strategies and framework, it won’t feet into that model but just added as some kind of alien infra. Long story short if you want fine grained control over how your infra should be managed from single centralised terraform, it isn’t best solution, yet still easy and balanced tool.

Before I start explaining how to use Advanced Set up, I am just going to mention that 4th, The Hard way is probably only good if you want to learn how k8s works, how all components interact with each other, and as it doesn’t use any external tool to set up components, you do everything manually, you literally know all the guts of the system. Obviously it could become a nightmare to support such system in production unless all members of the ops team are k8s experts or there are some requirements not supported by other bootstrapping tools.

Finally the Advanced Set up.

While we are going to use kubeadm to setup HA k8s cluster, there are also two ways of doing that:

1) The Stacked etcd topology, where the etcd cluster is just part of k8s cluster and

2) External etcd topology:

So what is etcd – think of it like database, the data layer of k8s cluster. Obviously external ectd is more secure due to separation of data and service layers, can be even managed by other team, if needed.

If one day your cluster is broken, you literally can bring up a new one from the scratch and all the data of the api server will just be restored from etcd, hence this is the most preferred option. Whereas in the stacked version etcd is just a docker container running along with other components as part of control pane.

Before we start setting up our cluster, we need to set up etcd cluster in HA way, so our k8s cluster can use it as external database.

Part 1 – Setting up HA ETCD cluster.

As I have already mentioned I am going to use terraform, our cluster will need the next resources:

DNS

Ubuntu server running etcd server and PKI infra installed.

Disk volume to store the data permanently

So straight to the etc module code:

tail -n 1000 *.tf ==> data.tf <== data "aws_ami" "ubuntu_1604" { most_recent = true name_regex = "ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-[0-9]*" } data "aws_route53_zone" "zone" { name = "k8s.ifritltd.co.uk." } ==> ebs.tf <== resource "aws_ebs_volume" "ebs-volume" { availability_zone = "${var.availability_zone}" size = "2" tags { Name = "ebs_etcd_${var.zone_suffix}" } lifecycle { prevent_destroy = false //todo enable back when finished } } ==> main.tf <== resource "aws_instance" "etcd" { ami = "${data.aws_ami.ubuntu_1604.id}" instance_type = "t2.micro" key_name = "terra" user_data = "${file("etcd.sh")}" availability_zone = "${var.availability_zone}" vpc_security_group_ids = [ "${var.sg_id}", ] iam_instance_profile = "${var.iam_instance_profile}" tags { Name = "example etcd ${var.zone_suffix}" } } resource "aws_route53_record" "etcd" { zone_id = "${var.zone_id}" name = "etcd-${var.zone_suffix}" type = "A" ttl = "300" records = ["${aws_instance.etcd.private_ip}"] } ==> output.tf <== output "public_ip" { value = "${aws_instance.etcd.public_ip}" }

As you see from the code above we define an instance running Ubuntu 16.04 backed with ebs and register it’s ip with dns record.

Now we can use the module to create resources in all 3 AZs in London region:

➜ etcd git:(master) ✗ tail -n 1000 *.tf ==> main.tf <== module "etcd-a" { source = "module/etcd" availability_zone = "eu-west-2a" zone_suffix = "a" iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}" sg_id = "${aws_security_group.etcd.id}" zone_id = "${aws_route53_zone.k8s_private_zone.zone_id}" } module "etcd-b" { source = "module/etcd" availability_zone = "eu-west-2b" zone_suffix = "b" iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}" sg_id = "${aws_security_group.etcd.id}" zone_id = "${aws_route53_zone.k8s_private_zone.zone_id}" } module "etcd-c" { source = "module/etcd" availability_zone = "eu-west-2c" zone_suffix = "c" iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}" sg_id = "${aws_security_group.etcd.id}" zone_id = "${aws_route53_zone.k8s_private_zone.zone_id}" } resource "aws_route53_zone" "k8s_private_zone" { name = "k8s.ifritltd.co.uk" vpc { vpc_id = "${data.aws_vpc.default.id}" } } ==> security-groups.tf <== resource "aws_security_group_rule" "all-internal" { type = "ingress" security_group_id = "${aws_security_group.etcd.id}" self = true from_port = 2379 to_port = 2380 protocol = "tcp" }

As you can see we created 3 instances in private dns zone k8s.ifritltd.co.uk and the nodes exposed as https://etcd-a.k8s.ifritltd.co.uk:2379 in each zone.

You also need to grant some permission to your instances in the IAM:

"ec2:DescribeVolumes", "ec2:AttachVolume", "ssm:PutParameter", "ssm:GetParametersByPath", "ssm:GetParameters", "ssm:GetParameter"

With this, ec2 can get ebs attached and use ssm for storing pki. The cloud init is probably most important part once infra is set up:

#!/bin/bash set -euxo pipefail sleep 30 AVAILABILITY_ZONE=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | cut -b 10) apt-get update apt-get -y install wget python-pip locale-gen en_GB.UTF-8 pip install --no-cache-dir awscli VOLUME_ID=$(aws ec2 describe-volumes --filters "Name=status,Values=available" Name=tag:Name,Values=ebs_etcd_$AVAILABILITY_ZONE --query "Volumes[].VolumeId" --output text --region eu-west-2) INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id) aws ec2 attach-volume --region eu-west-2 \ --volume-id "${VOLUME_ID}" \ --instance-id "${INSTANCE_ID}" \ --device "/dev/xvdf" while [ -z $(aws ec2 describe-volumes --filters "Name=status,Values=in-use" Name=tag:Name,Values=ebs_etcd_$AVAILABILITY_ZONE --query "Volumes[].VolumeId" --output text --region eu-west-2) ] ; do sleep 10; echo "ebs not ready"; done sleep 5 if [[ -z $(blkid /dev/xvdf) ]]; then mkfs -t ext4 /dev/xvdf fi mkdir -p /opt/etcd mount /dev/xvdf /opt/etcd ETCD_VERSION="v3.3.8" ETCD_URL="https://github.com/coreos/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz" ETCD_CONFIG=/etc/etcd apt-get update apt-get -y install wget python-pip pip install --no-cache-dir awscli useradd etcd wget ${ETCD_URL} -O /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz tar -xzf /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz -C /tmp install --owner root --group root --mode 0755 /tmp/etcd-${ETCD_VERSION}-linux-amd64/etcd /usr/bin/etcd install --owner root --group root --mode 0755 /tmp/etcd-${ETCD_VERSION}-linux-amd64/etcdctl /usr/bin/etcdctl install -d --owner root --group root --mode 0755 ${ETCD_CONFIG} cat > /etc/systemd/system/etcd.service <<EOF [Unit] Description=etcd key-value store [Service] User=etcd Type=notify ExecStart=/usr/bin/etcd --config-file=/etc/etcd/etcd.conf Restart=always RestartSec=10s LimitNOFILE=40000 [Install] WantedBy=ready.target EOF chmod 0644 /etc/systemd/system/etcd.service mkdir -p /opt/etcd/data chown -R etcd:etcd /opt/etcd cat > /etc/etcd/etcd.conf <<EOF name: 'etcd-AZONE.k8s.ifritltd.co.uk' data-dir: /opt/etcd/data listen-peer-urls: https://0.0.0.0:2380 listen-client-urls: https://0.0.0.0:2379 .. .... ..... initial-advertise-peer-urls: https://etcd-AZONE.k8s.ifritltd.co.uk:2380 advertise-client-urls: https://etcd-AZONE.k8s.ifritltd.co.uk:2379 discovery-fallback: 'proxy' initial-cluster: 'etcd-a.k8s.ifritltd.co.uk=https://etcd-a.k8s.ifritltd.co.uk:2380,etcd-b.k8s.ifritltd.co.uk=https://etcd-b.k8s.ifritltd.co.uk:2380,etcd-c.k8s.ifritltd.co.uk=https://etcd-c.k8s.ifritltd.co.uk:2380' ... .... ... client-transport-security: cert-file: /etc/ssl/server.pem key-file: /etc/ssl/server-key.pem client-cert-auth: false trusted-ca-file: /etc/ssl/certs/ca.pem auto-tls: false peer-transport-security: cert-file: /etc/ssl/server.pem key-file: /etc/ssl/server-key.pem peer-client-cert-auth: false trusted-ca-file: /etc/ssl/certs/ca.pem auto-tls: false .... ...... EOF sed -i s~AZONE~$AVAILABILITY_ZONE~g /etc/etcd/etcd.conf aws ssm get-parameters --names "etcd-ca" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/ssl/certs/ca.pem aws ssm get-parameters --names "etcd-server" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/ssl/server.pem aws ssm get-parameters --names "etcd-server-key" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/ssl/server-key.pem chmod 0600 /etc/ssl/server-key.pem chmod 0644 /etc/ssl/server.pem chown etcd:etcd /etc/ssl/server-key.pem chown etcd:etcd /etc/ssl/server.pem systemctl enable etcd systemctl start etcd

So what is happening here. First of all, we install some tools like python, pip, awscli. Then we attach EBS volume, create filesystem on it then mount it to /opt/etcd, same location we later specify in etcd config as data folder. We also advertise peers in the config by replacing AZONE with AZ where instance is running, and finally, we pull PKI infra, which we need to preinstall prior to spinning up our instances. The less important steps in etcd config are skipped but available in the source code

Part 2 – The PKI infra

The whole process is described in here

https://coreos.com/os/docs/latest/generate-self-signed-certificates.html

https://coreos.com/etcd/docs/latest/getting-started-with-etcd.html

Here are the steps to reproduce it:

brew install cfssl cfssl print-defaults config > ca-config.json cfssl print-defaults csr > ca-csr.json cfssl gencert -initca ca-csr.json | cfssljson -bare ca - cfssl print-defaults csr > server.json cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

And the config files used:

tail -n 100 *.json ==> ca-config.json <== { "signing": { "default": { "expiry": "43800h" }, "profiles": { "server": { "expiry": "43800h", "usages": [ "signing", "key encipherment", "server auth" ] }, "client": { "expiry": "43800h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "43800h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } ==> ca-csr.json <== { "CN": "IfritLTD CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "US", "L": "CA", "O": "IfritLTD", "ST": "San Francisco", "OU": "Org Unit 1", "OU": "Org Unit 2" } ] } ==> server.json <== { "CN": "etcd.k8s.ifritltd.co.uk", "hosts": [ "etcd1.k8s.ifritltd.co.uk", "etcd2.k8s.ifritltd.co.uk", "etcd3.k8s.ifritltd.co.uk" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "ST": "CA", "L": "San Francisco" } ] }

If you fancy you can also use openssl for the above, the important thing you need to generate CA, then cert and server key signed by CA and having 3 SANs per node, once you done you will have bunch of files:

ll total 80 -rw-r--r-- 1 kayanazimov staff 832B 25 Nov 17:24 ca-config.json -rw-r--r--@ 1 kayanazimov staff 307B 25 Nov 17:31 ca-csr.json -rw------- 1 kayanazimov staff 1.6K 25 Nov 17:32 ca-key.pem -rw-r--r-- 1 kayanazimov staff 1.0K 25 Nov 17:32 ca.csr -rw-r--r-- 1 kayanazimov staff 1.3K 25 Nov 17:32 ca.pem -rw-r--r-- 1 kayanazimov staff 0B 25 Nov 18:48 cacert.pem -rw-r--r-- 1 kayanazimov staff 2.1K 25 Nov 17:52 cert.crt -rw------- 1 kayanazimov staff 227B 25 Nov 18:16 server-key.pem -rw-r--r-- 1 kayanazimov staff 586B 25 Nov 18:16 server.csr -rw-r--r--@ 1 kayanazimov staff 357B 25 Nov 18:13 server.json -rw-r--r-- 1 kayanazimov staff 1.2K 25 Nov 18:16 server.pem

You only needs some of these files stored by AWS ssm so etcd and later k8s cluster can read them and use.

Right, time to run some terraform and ssh to instance, once on the instance you can test server logs with journalctl and test api as well:

curl -k https://127.0.0.1:2379/v2/keys/message -XPUT -d value="Hello" curl -k https://127.0.0.1:2379/v2/keys/message etcdctl --endpoints=https://etcd-a.k8s.ifritltd.co.uk:2379 get / --prefix --keys-only

Try to write something in one node and access it in the other to test all good.

Part 3 – Setting up k8s cluster

As with etcd, we first need to setup some resources with terraform:

resource "aws_instance" "master_node1" { ami = "${data.aws_ami.ubuntu_1604.id}" instance_type = "t2.micro" key_name = "terra" user_data = "${file("userdata_master_init.sh")}" vpc_security_group_ids = [ "${aws_security_group.kubernetes_sg.id}", ] availability_zone = "eu-west-2a" iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}" tags { Name = "example k8s master1" } } ... resource "aws_instance" "master_node2" { .. resource "aws_instance" "master_node3" { .. ... resource "aws_instance" "slave_node1" { ami = "${data.aws_ami.ubuntu_1604.id}" instance_type = "t2.micro" key_name = "terra" user_data = "${file("userdata_node.sh")}" vpc_security_group_ids = [ "${aws_security_group.kubernetes_sg.id}", ] availability_zone = "eu-west-2a" iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}" tags { Name = "example k8s slave_node1" } } resource "aws_instance" "slave_node2" { ... ..... resource "aws_instance" "slave_node3" { ....

So we spin up 3 masters and 3 nodes. In theory we can spin up as many masters and nodes we want, obviously in real prod environment we will use asg to ensure availability of each master and some cloud watch rules to scale nodes when load increases.

Then we hide the master nodes api server behind load balancer:

resource "aws_elb" "api_load_balancer" { name = "k8s-api" internal = true subnets = ["subnet-4e89bb03", "subnet-52ce2228", "subnet-892a81e0" ] instances = ["${aws_instance.master_node1.id}", "${aws_instance.master_node2.id}", "${aws_instance.master_node3.id}"] listener { instance_port = "${var.listener_port}" instance_protocol = "${var.lb_protocol}" lb_port = "${var.lb_port}" lb_protocol = "${var.lb_protocol}" } health_check { interval = "${var.hc_interval}" healthy_threshold = "${var.hc_healthy_threshold}" unhealthy_threshold = "${var.hc_unhealthy_threshold}" target = "${var.hc_target}" timeout = "${var.hc_timeout}" } security_groups = ["${aws_security_group.kubernetes_sg.id}"] tags = [ { key = "KubernetesCluster" value = "kubernetes" }, ] } data "aws_route53_zone" "k8s_zone" { name = "k8s.ifritltd.co.uk." private_zone = true } resource "aws_route53_record" "kubernetes" { name = "kubernetes" type = "A" zone_id = "${data.aws_route53_zone.k8s_zone.zone_id}" alias { name = "${aws_elb.api_load_balancer.dns_name}" zone_id = "${aws_elb.api_load_balancer.zone_id}" evaluate_target_health = false } }

As we already created our dns zone in etcd terraform we just refer to it here to create a record for api server’s load balancer, which we will use to refer to our cluster from external world.

Don’t pay attention to VPC and subnets, for simplicity sake I just stick to defaults. For security groups please refer to next table

The most important magic happens in cloud init files. Let’s start with masters ones, that is userdata_master_init.sh and userdata_master_join.sh, why two? Because starting from N version of kubeadm init-master config is slightly different from other joiners-masters.

Right, all in order, first of all we install docker! You didn’t know it? Oh yes, k8s uses Docker underneath, not always, so other container run-times are also available:

#!/bin/bash set -euxo pipefail # install docker curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" apt-get update && apt-get install -y docker-ce=18.06.0~ce~3-0~ubuntu # configure docker cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } EOF # setup systemd docker service drop-in directory mkdir -p /etc/systemd/system/docker.service.d systemctl daemon-reload systemctl restart docker .. ....

Once docker is ready, we go on and install kubectl – a tool to control our cluster, kubeadm – a tool to bootstrap the cluster, finally kubelet – part of the cluster, the node daemon, which talks to API server to find containers that should be running on its node:

... ...... systemctl restart docker # K8S SETUP # install required k8s tools curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat >/etc/apt/sources.list.d/kubernetes.list <<EOF deb https://apt.kubernetes.io/ kubernetes-xenial main EOF # all list https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages K8S_VERSION=1.12.3-00 apt-get update apt-get install -y kubelet=${K8S_VERSION} kubeadm:amd64=${K8S_VERSION} kubectl:amd64=${K8S_VERSION} python-pip apt-mark hold kubelet kubeadm kubectl locale-gen en_GB.UTF-8 pip install --no-cache-dir awscli systemctl daemon-reload systemctl restart kubelet

As soon as kubelet is up and running, if you check it’s logs:

journalctl -u kubelet -f

You will see that kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. Before we run kubeadm, we need to configure it:

... .... systemctl restart kubelet mkdir -p /etc/kubernetes/pki/etcd aws ssm get-parameters --names "etcd-ca" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/kubernetes/pki/etcd/ca.crt aws ssm get-parameters --names "etcd-server" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/kubernetes/pki/apiserver-etcd-client.crt aws ssm get-parameters --names "etcd-server-key" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/kubernetes/pki/apiserver-etcd-client.key # for initial master cat > kubeadm-config.yaml <<EOF apiVersion: kubeadm.k8s.io/v1alpha3 kind: ClusterConfiguration kubernetesVersion: stable apiServerCertSANs: - "kubernetes.k8s.ifritltd.co.uk" controlPlaneEndpoint: "kubernetes.k8s.ifritltd.co.uk" etcd: external: endpoints: - https://etcd-a.k8s.ifritltd.co.uk:2379 - https://etcd-b.k8s.ifritltd.co.uk:2379 - https://etcd-c.k8s.ifritltd.co.uk:2379 caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key networking: podSubnet: "192.168.0.0/16" EOF .. .....

First we pull the PKI infra we created in etcd, then create kubeadm config file with reference to etcd nodes and network for our pod subnet.

Once we are ready we can initiate our first master node!

... .... kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=all ..

After this step kubeadm will start configuring our cluster, that is pulling all docker images and telling kubelet what to do.

At this moment we can see docker images pulled by kubeadm:

docker images REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.12.2 15e9da1ca195 13 days ago 96.5MB k8s.gcr.io/kube-apiserver v1.12.2 51a9c329b7c5 13 days ago 194MB k8s.gcr.io/kube-controller-manager v1.12.2 15548c720a70 13 days ago 164MB k8s.gcr.io/kube-scheduler v1.12.2 d6d57c76136c 13 days ago 58.3MB k8s.gcr.io/coredns 1.2.2 367cdc8433a4 2 months ago 39.2MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 10 months ago 742kB root@ip-172-31-19-60:~#

If you were running stacked etcd then it also would appear in this list. Let’s check further:

kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-576cbf47c7-mnd89 1/1 Running 0 19m coredns-576cbf47c7-sh4z8 1/1 Running 0 19m kube-apiserver-ip-172-31-19-60 1/1 Running 0 18m kube-controller-manager-ip-172-31-19-60 1/1 Running 0 18m kube-proxy-d7r49 1/1 Running 0 19m kube-scheduler-ip-172-31-19-60 1/1 Running 0 18m

As you can see our cluster is already running and ready to some extend.

Next is configuring calico for networking security:

# install calico kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Once this steps done, our cluster should be up and running:

kubectl get node NAME STATUS ROLES AGE VERSION ip-172-31-19-60 NotReady master 8m43s v1.12.2 root@ip-172-31-19-60:~#

Just wait until master is ready…

And then next step is to prepare some stuff for next masters:

aws ssm put-parameter --name "k8s-ca" --value "$(cat /etc/kubernetes/pki/ca.crt)" --type "SecureString" --region eu-west-2 --overwrite aws ssm put-parameter --name "k8s-ca-key" --value "$(cat /etc/kubernetes/pki/ca.key)" --type "SecureString" --region eu-west-2 --overwrite aws ssm put-parameter --name "k8s-sa" --value "$(cat /etc/kubernetes/pki/sa.pub)" --type "SecureString" --region eu-west-2 --overwrite aws ssm put-parameter --name "k8s-sa-key" --value "$(cat /etc/kubernetes/pki/sa.key)" --type "SecureString" --region eu-west-2 --overwrite aws ssm put-parameter --name "k8s-front-proxy-ca" --value "$(cat /etc/kubernetes/pki/front-proxy-ca.crt)" --type "SecureString" --region eu-west-2 --overwrite aws ssm put-parameter --name "k8s-front-proxy-ca-key" --value "$(cat /etc/kubernetes/pki/front-proxy-ca.key)" --type "SecureString" --region eu-west-2 --overwrite aws ssm put-parameter --name "k8s-init-token" --value "$(kubeadm token create)" --type "SecureString" --region eu-west-2 --overwrite aws ssm put-parameter --name "k8s-init-token-hash" --value "$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex sed 's/^.* //')" --type "SecureString" --region eu-west-2 --overwrite

Here we store all PKI created by master to AWS ssm so next master can reuse it, we also store token and hash for joiner master.

Now we can spin up another N master:

.. ..... # wait for master node while [ "None" = "$(aws ssm get-parameters --names 'k8s-init-token' --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2)" ];do echo "waiting for init master"; sleep 5;done aws ssm get-parameters --names "k8s-ca" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/kubernetes/pki/ca.crt aws ssm get-parameters --names "k8s-ca-key" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/kubernetes/pki/ca.key aws ssm get-parameters --names "k8s-sa" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/kubernetes/pki/sa.pub aws ssm get-parameters --names "k8s-sa-key" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/kubernetes/pki/sa.key aws ssm get-parameters --names "k8s-front-proxy-ca" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/kubernetes/pki/front-proxy-ca.crt aws ssm get-parameters --names "k8s-front-proxy-ca-key" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2 > /etc/kubernetes/pki/front-proxy-ca.key TOKEN=$(aws ssm get-parameters --names "k8s-init-token" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2) TOKEN_HASH=$(aws ssm get-parameters --names "k8s-init-token-hash" --query '[Parameters[0].Value]' --output text --with-decryption --region eu-west-2) kubeadm join kubernetes.k8s.ifritltd.co.uk:6443 --token $TOKEN --discovery-token-ca-cert-hash sha256:$TOKEN_HASH --experimental-control-plane

I skipped the part which is similar to master-initiator, docker install etc, significant part is joiner master is waiting until token is created by first alpha master, so it can join the existing cluster with kubeadm join command.

Finally when you masters are running, you can join any number of slaves to the cluster:

.... kubeadm join kubernetes.k8s.ifritltd.co.uk:6443 --token $TOKEN --discovery-token-ca-cert-hash sha256:$TOKEN_HASH

The cloud init is just like for master-joiner, but kubeadm init without –experimental-control-plane parameter.

Lets get nodes:

oot@ip-172-31-23-145:~# kubectl get nodes NAME STATUS ROLES AGE VERSION ip-172-31-23-145 Ready master 10m v1.12.2 ip-172-31-23-146 Ready master 17m v1.12.2 ip-172-31-29-77 Ready <none> 7m20s v1.12.2 root@ip-172-31-23-145:~#

You should see something similar, then consequently more nodes will just join the cluster as you spin them up. Once we finish our bootstrapping process, we should have 9 nodes in total, 3 per zone for etcd, 3 for masters and finally 3 for slaves. 9 nodes in total.

While the actual code to bootstrap the cluster may not be quite complete for production use, the approach I used is satisfactory in my opinion for automating the bootstrapping process of HA Kubernetes cluster with external etcd.

Now that you are familiar with bootstrapping process, you can clone and spin up the cluster using my git repo:

HA etcd: https://github.com/kenych/terraform_exs/tree/master/etcd

HA k8s: https://github.com/kenych/terraform_exs/tree/master/k8s_ha

Please note I am using the next versions:

K8S_VERSION=1.12.3-00

ETCD_VERSION=v3.3.8

Normally pinned version means stable code, so I really hope it is just going to work next time you run it. The only manual step is to create and store initial PKI infra which is very straightforward anyway.