Don’t tell me it can’t be automated.

This guide is to create a automated deployment of Kubernetes on an SD card. By the end of this you should have an SD card loaded into your Raspberry Pi cluster to turn on and just “work”. This is a completely automated and near immutable approach to running and maintaining Kubernetes on your home cluster.

The scope is the automated approach to creating a Kubernetes cluster, investigation of the code is left to you.

The main driver behind the development of this stack was to emulate what it is like developing in a cloud environment like GCP, with a pets vs cattle methodology.

Prerequisites

My stack comprises of 15 Raspberry Pis, however the minimum recommended number is 4 to be able to follow the guide.

This could theoretically be executed with 2 Raspberry Pis (1 Master and 1 Worker). However in order to cater for this you are required to adjust the KUBE_MASTER_IP_XX variables used in the Makefile .

Linux Operating System

Kubectl

Minimum 4 Raspberry Pis (3 Masters and 1 Worker)

Access to https://github.com/lucasteligioridis/raspbernetes

List of parts from my stack:

Versions of applications

All of the start up scripts that create this stack have all the application versions pinned to produce consistent expected behaviour.

Network Topology

The design has included a full high availability using a floating VIP (virtual IP) over the three masters in case of an outage, this is used in lieu of a load balancer.

The design also provides some extra flexibility to allow for a power outage on any node without impacting the cluster. The image below describes a high level overview of what the entire stack looks like:

Raspberry Pi Cluster with Full Mesh HA

Hostname and IP configuration

See below for a full list of hostnames and IP addresses that will frequently be used throughout this guide and feel free to create your own to refer back to.

Adjust accordingly depending on the number of Raspberry Pis in your cluster, the following depicts the configuration that I used in my cluster:

rpi-kube-master-01: 192.168.1.101

192.168.1.101 rpi-kube-master-02: 192.168.1.102

192.168.1.102 rpi-kube-master-03: 192.168.1.103

192.168.1.103 rpi-kube-worker-01: 192.168.1.111

192.168.1.111 rpi-kube-worker-02: 192.168.1.112

192.168.1.112 rpi-kube-worker-03: 192.168.1.113

192.168.1.113 rpi-kube-worker-04: 192.168.1.114

192.168.1.114 rpi-kube-worker-05: 192.168.1.115

192.168.1.115 rpi-kube-worker-06: 192.168.1.116

192.168.1.116 rpi-kube-worker-07: 192.168.1.117

192.168.1.117 rpi-kube-worker-08: 192.168.1.118

192.168.1.118 rpi-kube-worker-09: 192.168.1.119

192.168.1.119 rpi-kube-worker-10: 192.168.1.120

192.168.1.120 rpi-kube-worker-11: 192.168.1.121

192.168.1.121 rpi-kube-worker-12: 192.168.1.122

Must Read

Before beginning you should be aware of the Makefile that helps build the SD cards. The default name of the SD device is:

/dev/mmcblk0

Please be mindful, I only tested this on a Debian based operating system and you could erase the wrong device by accident if not careful. Ensure that when you insert your blank SD card ready to write the image of this stack that these are the correct device names. To confirm this, once the SD cards have been inserted into your machine run lsblk in your terminal to confirm and you should get an output like below or similar:

$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

mmcblk0 179:0 0 29.7G 0 disk

├─mmcblk0p1 179:1 0 256M 0 part

└─mmcblk0p2 179:2 0 29.5G 0 part

If you find that your device name is different, you can override the existing default by using the below command in your terminal:

export MNT_DEVICE=/dev/<YOUR_DEVICE_NAME>

Setup

Every step going forward in this guide is assuming that you are running within the same terminal session.

Common

Clone the repository that has been provided and checkout to the release tag that has been listed below to ensure that instructions in the guide are consistent to the time of writing:



cd raspbernetes

git checkout v1.0 git clone git@github.com :lucasteligioridis/raspbernetescd raspbernetesgit checkout v1.0

If wanting a brief description about each of the make targets, just enter the below command into the terminal:

make

Bootstrap the first set of static variables that every permutation or type of node will use (changing to cater for your specific environment):

export RPI_DNS=192.168.1.1

export RPI_NETWORK_TYPE=eth0

export RPI_TIMEZONE=Australia/Melbourne

export KUBE_MASTER_VIP=192.168.1.100

export KUBE_MASTER_IP_01=192.168.1.101

export KUBE_MASTER_IP_02=192.168.1.102

export KUBE_MASTER_IP_03=192.168.1.103

For this setup there is an assumption that the RPI_NETWORK_TYPE is an eth0 (ethernet cables connected from Raspberry Pi to network switch). There is an option to connect to WiFi by setting the below commands (added a space as a suffix to exporting password environment variable to prevent saving to history):

export RPI_NETWORK_TYPE=wlan0

export WIFI_SSID="YourWifiSSID"

export WIFI_PASSWORD="PasswordForWifi"

Master

Insert the SD card into the SD card slot of your machine and run the below commands:

export KUBE_NODE_TYPE=master

export RPI_HOSTNAME=rpi-kube-master-01

export RPI_IP=192.168.1.101 make build

If this is your first time running this, you will be required to download the Raspbian image from the official website. This will only be required once going forward as long as you don’t delete the .img file that gets downloaded into the repositories output/ directory.

You should now see dd formatting the SD card with a brand new copy of Raspbian. After the image has been successfully written onto the SD card, the configuration file will be generated and placed onto the SD card to be sourced at boot time.

The output of the above command should look like the following example:

Formatting SD card with 2019-09-26-raspbian-buster-lite.img

2243952640 bytes (2.2 GB, 2.1 GiB) copied, 7 s, 317 MB/s

536+0 records in

536+0 records out

2248146944 bytes (2.2 GB, 2.1 GiB) copied, 109.321 s, 20.6 MB/s Created a headless Kubernetes SD card with the following properties:

Network:

- Hostname: rpi-kube-master-01

- Static IP: 192.168.1.101

- Gateway address: 192.168.1.1

- Network adapter: eth0

- Timezone: Australia/Melbourne

Kubernetes:

- Node Type: master

- Control Plane Endpoint: 192.168.1.100

- Master IP 01: 192.168.1.101

- Master IP 02: 192.168.1.102

- Master IP 03: 192.168.1.103

Feel free to now eject the SD card from your machine and insert into the Raspberry Pi of your choice.

Make 2 more masters with the above commands, you will need to change the RPI_HOSTNAME and the RPI_IP from the table posted above.

Congratulations, all the masters are now ready to go!

Worker

Insert the SD card exactly like the master setup and refer to the table posted above for hostname and IP references:

export KUBE_NODE_TYPE=worker

export RPI_HOSTNAME=rpi-kube-worker-01

export RPI_IP=192.168.1.111 make build

Once the SD card has finished writing and the make command has completed successfully, continue with writing your remaining SD cards as worker nodes following the details of RPI_HOSTNAME and RPI_IP from the table posted above.

Congratulations, all the workers are now ready to go!

Kubernetes Cluster Time

Now that all the Raspberry Pis are loaded with the correct configuration parameters, you are ready to turn on the cluster. The bootstrap has been designed in a way that ensures that the Pis can be booted in any order and there is no need to wait for the cluster to be initialised first. It has been configured to avoid any sort of race condition or split brain of the cluster.

A break down of what is going to happen at boot time:

All nodes start at the same time. Hostname and IP have been set. All Kubernetes packages have been installed with all required dependencies. Initialisation of cluster begins after Keepalived sets a VIP on one of the master nodes. All other nodes (masters and workers) wait until the first master has initialised the cluster. Joining the cluster can begin, join commands and certificates are retrieved on the master hosting the VIP all in parallel for remaining master and worker nodes.

Keep note of the SSH key that was generated and placed in the output/ssh/ directory that is used for all inter-cluster communication and to remote to the Raspberry Pis. Do not lose this!

Ready to boot

Since I have 15 different nodes and I want to monitor all of the logs, I will be splitting my terminal with tmux , this is optional. If you don’t use tmux I highly recommend it, although you don’t need it for this guide, it is a good utility for navigating efficiently around your terminal.

Power on

You may now power on your Raspberry Pis in any order. This should take about 8 minutes before your cluster has been fully established. After at least 1 minute has elapsed from powering on (waiting for hostname changes to occur), you can tail the logs on your node using the below command (replace <RPI_HOSTNAME> with the node you would like to monitor):

ssh -i output/ssh/id_ed25519 -o StrictHostKeyChecking=no pi@<RPI_HOSTNAME>.local tail -f -n 200 /var/log/syslog | grep -Eo 'kubernetes-bootstrap:(.*)'

If all steps have been followed correctly, your Kubernetes cluster will have deployed successfully! If you tailed the logs on all or any of your nodes, the very last line in the output should be:

kubernetes-bootstrap: Finished booting! Kubernetes successfully running!

This indicates that the node has finished booting and joined/initialised the cluster! Congratulations for making it this far!