Kubernetes is the new black. It’s the leading technology for handling containers on multiple nodes. But there is a problem. It’s not very easy to install in a vast scope on your own, especially without having cold feet about the security layer. I will not speak about container security, only how Kubernetes can add some proper structure for them.

Interested in container security? check: ce truc, ce machin, ici et là

This complexity explains why many cloud providers are in place, like AWS, Google, etc…

Generally speaking, it’s pretty easy to create a simple tainted control-plane (a tainted control plane can host pods as well as workers). Lots of documentation will help you achieve this goal. But when we want to step up to a more substantial architecture, some trouble can appear.

The primary purpose of this article is to give you the proper way to install a secured and stable Kubernetes cluster, on your server, without using any black-boxed tools. I don’t say these tools are wrong, but in a world where we used to click somewhere to have something, it can be useful to break this habitude and try to understand what we are doing. (especially when something goes wrong)

This article will give you the direction to follow to understand a Kubernetes installation and the proper way. In the last part, we will use the CNCF guideline to improve our installation step by step (part 5), There are, obviously, other ways but I present you, in all humility, mine.

Sources used for part 4–5 :

#clic!

#boom 500!

By the way, I’m open to comment, “improve oriented,” do not hesitate to poke ma boite aux lettres ;)

Yes, hyperlinks link looks strange, be reassured, it’s the only French part of this article… not to mention my poor English synthase! So, to be clear, I split the article into 5 “short” parts :

Part 1 — Context :

Short explanations

Targeted Architecture

Pre-requirements

Part 2 — Tools :

External Secured ETCD

Install Docker

Install Kube*

Part 3 — Install/configure Kubernetes :

Install First Control-Plane

Main url

ETCD conf

Calico CNI

Install second Control-Plane

Install Node

Part 4 — Basic Security :

SECURITY: Encrypt Secrets

SECURITY: Admission Control

SECURITY: Default ServiceAccount

SECURITY: RBAC

Part 5 — Installation Conformities: CNCF/CIS :

Sonobuoy

Installing

Launch tests

Fetch Results

Benchmark — CIS Conformities

1 — — 1.2.6 Ensure that the — kubelet-certificate-authority

2 — — 1.2.14 Ensure that the admission control plugin ServiceAccount is set

3 — — 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Scored)

4 — — 1.2.21 to 1.2.25 Audit-Profiling Issues

5 — — 1.3.1 Ensure that the — terminated-pod-gc-threshold argument is set as appropriate (Scored)

6 — — 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)

7 — — 1.3.2 Ensure that the — profiling argument is set to false (Scored)**

Maintenance

Part 1 — Context

Short explanations

In terms of infrastructure, Kubernetes is divided in 3 mains parts (highly summarised):

ETCD (high performances key/value database) (Par là)

Control-Plane, It’s the master “Node,” it includes the kube-api, Scheduler, network management…

Nodes/Workers, where your containers will be install/running

Tips: Your control-plane can be configured to run your containers as well. In our architecture, it will be the case.

In order to improve your stack availability, the Control-Plane (CP) can be clustered as well ;)

Targeted Architecture

Okay, let’s have a look at our schema :

As you can see, I’ve tried to keep the things simple.

We have two Control-Pane, Loadbalanced by DNS (Use a real load balancer if you can (HAProxy, cloud LB…)), attached to these CPs an External ETCD Cluster. Finally, we add one k8s node.

Pre-requirements

Servers

They respect the k8s requirements (found here)

In order to use your container efficiently, switch off your swap ;)

swappoff -a

Ports !

For the good of your cluster, and your nerves, some ports must be open.

I use Iptables, but even if you use another firewall, it’s essential to use the correct broad used strategy: Limiting the Attack Surface. You have to drop everything, then open the selected port. Moreover, theses selected ports must be open just for some well-known IPs.

Example :

It’s an example ! open you ssh and some mandatory ports as weel :D

Recap needed ports :

Master server (Control-plane node(s))

Worker Node(s)

Okay, we have the decors, let’s put some actors in it! let’s go through the Part 2 : Installing Tools