Do you have a very specific dream of running a container orchestration tool on top of multiple single-board computers of a RISC architecture?! If you do, this post is perfect for you! If not, stick around maybe you will be inspired. We are talking today about a home-lab setup that I have recently transitioned to not only to save some electricity, but also for a great learning experience. I am talking of course, of Kubernetes running on top of Raspberry Pi 4s.

Kubernetes has been the new hotness for a while now in enterprise deployments and for good reason. Kubernetes is powerful and as you set it up, you realize some of the pains that Docker introduces when trying to use it for services that you want up all the time. Docker is an amazing tool, and Docker Swarm is very simple to use, but every time I set it up I seem to get everything working, and then the whole Swarm falls over for some reason. Getting it back up and running is a pain, and a major drain on the limited project time I already have. Playing system administrator isn't my idea of a good time.

If you don't know what Kubernetes is, I will point you towards https://kubernetes.io/ for the description. The short of it is that it is a tool that when provided with hardware to manage, and descriptions of deployments, will schedule those deployments. This means our services will stay up, move around the cluster, and scale up automagically. Enough of my rambling, on with the show!

Inspiration

The reason this project came into existence was a previous bright idea I had. I bought a full server rack and 3 Proliant DL385 2U rack mounted servers from government auction. This is a great home-lab setup, I had all 3 ESXi hosts with vSphere to manage all of them, remote access with VMWare Workstation. It was awesome. The issue became that I didn't actually have a use for 36 cores and 288 GB of RAM, and running those servers even idle was roughly 100 watts per server. Crunching some quick numbers on power usage, if I did nothing with these servers all year, and they were on 24 hours a day, that is about $300 a year in electricity which isn't awful, but seems expensive to run a webserver. When the servers were loaded they were significantly more expensive to run, up to about 1 kW total for all three when they were actually working, that shoots the figure up to about $1000 per year for electricity. Considering that you can host a website for $3/month on bluehost, my current hardware seemed excessive. Of course, I ran other things on my servers, UNMS for my Ubiquiti gear, various internal services, and a simple QR-code redirection server that I wrote for business cards a while ago, but overall it wasn't enough to justify such beefy hardware.

I looked around for two things, cheaper hardware and a learning experience. In the past I have tried messing around with Rancher, before it was a Kubernetes management software, and it always seems almost ready for real usage. This time, I decided that I was just going to bite the bullet and learn to use Kubernetes. Additionally, recently(ish) the Raspberry Pi 4 was released and it has a beautiful 4 GB RAM option (AARCH64 Processor too!). It. is. awesome. I love embedded devices. I love developing for embedded devices and exotic architectures, and in my personal opinion, 1 GB of RAM was really one of the only major things holding the Pi line of single-board computers from total greatness. I just needed an excuse to buy a few of these boards, and this post was it! So here I have it, a Kubernetes-based Raspberry Pi 4 cluster for home-lab use!

Tutorial

Hardware

The hardware I use for my clustering set up is as follows:

2 Raspberry Pi 4 (4GB) - $110

2 Raspberry Pi PoE Hat - $40

2 Micro SD cards (32GB+) - $15-ish

That is it. Get everything on Amazon or the Pi store and you are off to the races. The most important thing is, don't pay more then $55 for the 4GB variant of the Pi unless it comes with extra stuff you need.

Configuration

Install Etcher

Flash Raspbian-Lite onto the SD cards

Next we are going to configure the default Raspbian installation with raspi-config , we need to:

Enable SSH

Give the Pi a hostname

Set video memory to 16 MB - We have no GUI so we don't need video memory

We want to disable swap on the Pi, this is a requirement of Kubernetes and we can do so pretty easily by just removing the service

sudo systemctl disable dphys-swapfile.service

Go ahead and update all of the packages too

sudo apt update && sudo apt upgrade

Finally, the most important part of set up, is enabling AARCH64. Wait, enabling? Yes, all Raspberry Pi kernels are ARM (32 bits), and the 64 bit kernel is still experimental at this point, but we can opt in easily. This is so we can have access to the ARMv8 containers that we so rightly deserve!

Open /boot/config.txt

Append arm_64bit=1 to the end of the file

to the end of the file Run sudo rpi-update

You are going to have to reboot a few times during that process, but once the dust has settled make sure you have SSH keys (no password authentication) on the Pi for the next step.

Installing k3s

So the next piece of the puzzle once we have our Pi's configured properly is to install Kubernetes to the Pi's. We could do this with k8s (the full version of Kubernetes), but we are somewhat constrained on resources, and we don't really want to spend half of our RAM just running Kubernetes. k3s is a lightweight Kubernetes version that doesn't include the Docker container runtime, which means we can't build docker containers on these hosts, but that isn't as big of a deal as you would think since we can just download them from a registry. While k3s isn't hard to install, there is an even easier way, k3sup. This tool is so amazing, I forgive it for the weird 'ketchup' pronunciation. It makes the installing and clustering absolutely dead simple.

k3sup install --ip $SERVER_IP --user $USER

This gets us our Master node, if you have only a single Pi, this is it, we have Kubernetes running. You will have a kubeconfig file in your current directory. All you need to do is install kubectl (directions of various ways) and you are off to the races.

If we have multiple Pi's its super easy to cluster

k3sup join --ip $AGENT_IP --server-ip $SERVER_IP --user $USER

Now if you run kubectl get nodes you should see a Master and a Worker in your cluster, ready to work!

Deployments

Now that Kubernetes is deployed I won't cover too much on how exactly you interact with Kubernetes since that is a long topic and I am by no means an expert. I will go over some of the things that I noticed setting up my cluster.

My default traefik deployment was very broken. I deleted it as soon as possible, you can actually choose with k3sup to not deploy traefik, since most people end up using nginx as an ingress controller. Personally I like traefik a lot, so I just did some tweaking to get it working.

You are going to need MetalLB. Kubernetes is designed for "the cloud" so you don't actually have a load balancer built in since that is supposed to be provided by the cloud provider. MetalLB is dead simple to set up, and works great. Make sure you install this before trying to fix your traefik installation (no load balancer is part of the reason it won't work)

If you have issues and see exec format error in any use of kubectl logs ... or kubectl describe ... then verify that the image you are trying to download has an ARM64/AARCH64/ARMv8 image that is available. Since ARM support is fairly new, not all of the common images have built in support and automatic deployment of ARM images. I found that images from arm64v8 all seem to work really well!

in any use of or then verify that the image you are trying to download has an ARM64/AARCH64/ARMv8 image that is available. Since ARM support is fairly new, not all of the common images have built in support and automatic deployment of ARM images. I found that images from arm64v8 all seem to work really well! I have not had great luck trying to use Helm. Now there may be a few issues, such as attempting to use Helm 3, not having any experience, or possibly AARCH64, I don't have an exact reason. I did find that using normal Kubernetes deployment yaml's seemed to be easier for me to get everything up and running correctly.

Wrap up

With that, you should have a working home-lab cluster of Raspberry Pi 4's. I have found this set up pretty robust overall, as well as cheap and quiet. The Pi 4 runs anywhere from 3-7 watts from that I have seen. Rounding that up to 10 watts per hour, a Pi costs me about $10 to run 24/7 for a year, while giving me access to 4 cores, ~3.5 GB of RAM (after required setup), and gigabit wired ethernet. The ease of adding new workers means that if I need to scale out the setup, I can, quickly and easily. Overall, I would highly recommend the Pi setup for anyone looking for an affordable home-lab!