Photo by Jonathan Brinkhorst on Unsplash

Overview

While preparing for my CKA & CKAD certification exams, I wanted to build a solution to create ephemeral, ultra low-cost ‘sandbox’ Kubernetes clusters that could be spun-up and dismantled quickly. I did not want to use a solution like Minikube locally since that would only let me create single-node clusters. I required access to a true multi-node cluster environment to practice topics like Kubernetes cluster upgrades & maintenance.

AWS Spot Instances

To keep costs down, I looked at AWS Spot Instances to build my ‘sandbox’ cluster. EC2 Spot Instances are unused EC2 instances which are sometimes available at up to 90% discount compared to On-Demand EC2 instances. The downside of Spot Instances is AWS can terminate Spot Instances with little to no warning. Since resiliency was not a primary driver, I was not very concerned if the Spot instances in my cluster got terminated suddenly; as long I had a way to spin up another one quickly without too much effort. Hence, I resorted to CloudFormation to create a re-usable script that I could run and provision an entire cluster within couple of minutes.

The CloudFormation script k8s-spot.yaml

The entire CloudFormation script is shared at https://github.com/dhiman-halder/k8s-aws-spot

https://raw.githubusercontent.com/dhiman-halder/k8s-aws-spot/master/k8s-spot.yaml

The CloudFormation Script provisions the following AWS resources in us-east-1:

A VPC with 1 public subnet, an Internet Gateway, a Public Route and a Security Group 1 t3a.small EC2 Spot Instance for master node (2 vCPU, 2 GiB memory) running Ubuntu 16.04. While the current On-Demand pricing is $0.0188 per hour, the Spot price is just $0.0056 at the moment- which is close to 70% discount compared to On-Demand price! 1 t3a.micro EC2 Spot Instance for worker node (2 vCPU, 1 GiB memory) running Ubuntu 16.04. The current On-Demand price is $0.0094 per hour and the Spot price is $0.0028- again a 70% discount! Each EC2 Instance is paired with 1 8GB SSD EBS volume. The per hour cost for each volume is $0.0011 IAM role for the Spot Fleet

The Cluster is bootstrapped using Kubeadm through the UserData script as part of the EC2 provisioning process. The shell script does the following:

On the Master node:

Set the hostname to master Turn-off SWAP Install and run Docker as the Container Runtime Engine Install Kubeadm, Kubelet & Kubectl tools Start the Kubelet service Use Kubeadm to initialize the cluster on the master Setup KubeConfig on the master Install Calico as the pod network add-on Create a join.txt file on the user’s root with a Kubeadm command to let the worker node join the Kubernetes cluster

On the Worker node:

Set the hostname to node01 Turn-off SWAP Install and run Docker as the Container Runtime Engine Install Kubeadm, Kubelet & Kubectl tools Start the Kubelet service

The UserData scripts follows the Kubernetes official guide as documented here:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Run the CloudFormation Script

You can run the CloudFormation script either using the AWS CLI or the AWS Management console. You need to supply an EC2 KeyPair for SSH access into the EC2 Instances.

Manual Step

Once the EC2 instances are provisioned and running:

SSH into the master node & copy the Kubeadm join command from within the join.txt file on the user’s root SSH into the worker node and execute the copied command to join the worker node to the cluster.

Testing

From the master node, run kubectl get nodes -this should list the 2 nodes and they should be in Ready state! Run kubectl run nginx --image=nginx --restart=Never followed by kubectl get pods to list the newly created nginx pod

Clean-up!

Delete the created CloudFormation stack to tear down the cluster!

Conclusion

Ignoring any data transfer cost, the current cost of running the 2 EC2 Spot Instances along with their EBS volumes is just $0.0056+$0.0028+$0.0011+$0.0011=$0.0106 which is ~1¢ per hour. Essentially, for $1, the environment could be run for ~4 days! This is of-course assuming the average spot price during the period do not deviate substantially.

Thanks for Reading!

Found this article useful? Hit that clap button. Really like it? Hold the clap, give it two, or fifty!