Last Updated on August 20, 2019

Reading Time: 5 minutes

The idea of a stripped down operating system that only includes enough to start a container platform was pioneered by CoreOS a long time ago.

Recently there has been a bit of a resurgence in this area with k3os and Talos pushing the boundaries of what a host operating system should include in a very Kubernetes specific context.

Today we’ll take a look at some of the features of Container Linux (formerly CoreOS), RancherOS, Talos, k3os and LinuxKit and discuss if there is any benefit to using these versus installing Kubernetes on a standard Linux distribution.

For the impatient here’s the raw data and link to the Google sheet. A more subjective discussion and interpretation will follow below.

Link to the Google sheet.

Before we begin let’s take a brief detour into what an operating system really is. A blog entitled “Sorry, Linux. Kubernetes is now the OS that matters” rustled a few jimmies not so long ago. This prompted some fascinating discussion.

A traditional view of an operating system is something that abstracts hardware and allows us to start and interact with applications. In a Kubernetes centric world it’s easy to see why there has been a lively debate on this subject.

In fact, if I was merely looking for clicks I’d come out with claims like..

“Kubernetes is the new Linux”

“Kubelet is the new Kernel”

“kubectl is the new GNU base system”

“Helm is the new Yum”

“Operators are the new Config Management tools”

There is some partial truth to these outrageous claims but it’s clearly not the full story. At the heart of all of these Kubernetes centric operating systems there’s still a Linux kernel.

Where things get ambiguous is what else is actually then included in the base operating system.

Does it need a full init system? Possibly not.

Does it need a package management system? Not if we only run containers.

What about user-land binaries? Do we even need SSH? In the case of Talos you don’t even log into the OS any more as everything is managed over a GRPC API.

By reducing what’s contained in the base operating system we shift that complexity up a layer into Kubernetes. Does this improve security and reduce operational cost? I don’t know and I’m not sure anyone can prove it does or doesn’t.

It does mean however that we no longer need to know about Yum and Apt. Instead we learn how to package and manage applications on Kubernetes instead. The benefit being that we manage everything in a uniform and consistent way.

For those of us who have fully drank the Kubernetes kool-aid it’s certainly an attractive proposition. A shift towards an immutable operating system with a read-only root filesystem that boots our container platform and lets us manage our applications and dependencies via a single API.

I guess that explains why people are working on these projects.

Now we’ve dealt with the value, or perhaps the perceived value, let’s have a look at some of these projects and their current status.

Container Linux (formerly CoreOS)

I got quite excited by this a few years ago and in all honesty I’m happy we never adopted it. There was always a concern that the company would go bust or get acquired, that the project would get ruined, and we’d be left on a dead-end platform.

Although nothing as extreme as this has happened I think post-Redhat-acquisition CoreOS has dramatically changed. Their focus, rightly so, is incorporating all the technology into OpenShift and navigating the project Atomic merger.

It’s not very clear right now exactly how you should approach running Kubernetes on Container Linux in production. You can still use Tectonic until August 2019 when it goes EOL but it deploys an ancient version of Kubernetes (1.9 at the time of writing when 1.13 is currently stable).

What about OKD, the community edition of OpenShift? I have no idea if that even uses Container Linux, I don’t think it does. The docs show it supports RHEL or Atomic Host which itself was supposed to be deprecated in favour of the CoreOS tech and the promise of the best of both which has never materialised.

Given there’s no clear direction from Redhat on any of this my suggestion is to simply avoid Container Linux altogether.

RancherOS

If you’re using Rancher then this is probably a no-brainer. RancherOS is production ready and includes a way to keep Kubernetes upgraded to a modern version using RKE. All of this is free and open source and there’s paid support if you want it.

Using Docker as PID1 has slowly turned to becoming a bit of a negative as the Kubernetes CRI landscape has evolved. I suspect Rancher realise this and will replace it in future versions.

Talos

Talos has taken a unique approach to what an operating system should include. Their ideas around managing nodes via a GRPC API are very interesting. It’s a new project that is under extremely active development.

I’d personally like to see this project evolve to something that could be run in production on the major cloud platforms and bare metal hosts.

The team is currently working on a way to upgrade the base operating system and it would be good if Talos included an Operator that supported upgrading Kubernetes packages.

A strong focus on security and the fact that Talos is not tied to any particular ecosystem makes it a prime contender for becoming a very popular Kubernetes specific OS in the future.

k3os

k3os is limited by the scope of k3s itself which is focussed on being a small single binary Kubernetes for running on edge devices or inside CI pipelines. I personally want to try using this as a private Gitlab runner using the Gitlab Helm chart. Similarly, if I wanted to install Kubernetes on a Raspberry Pi or edge device this is the option I’d go with.

In terms of running my production applications in the cloud I’d go with RancherOS instead as that includes a full version of Kubernetes that’s easy to upgrade.

LinuxKit

LinuxKit lets you build operating system images with any customisations you want. There’s a Kubernetes specific project that I’ve referenced in the table above that demonstrates how to create an image with Kubernetes 1.10 baked inside.

I’m not convinced many people want to actually build their own operating system although I may be wrong. It’s a cool proof of concept and I’m sure there are valid reasons why you’d want to bake immutable images but I suspect many would prefer the convenience of using a distribution from an actively maintained project.

Conclusion

Right now none of these projects are compelling enough for me to switch out the standard Linux operating system I’m using on my Kubernetes clusters on AWS at work.

I’d recommend RancherOS if you want to run Kubernetes on your Raspberry Pi or edge devices, or if you’re already using RKE.

Consider looking at k3os for private Gitlab CI runners and any other edge type stuff you may be playing with.

Talos is a project I’ll keep my eye on for having the potential to displace Centos / RHEL / Ubuntu for running my production Kubernetes workloads in the cloud.