Introduction

Kubernetes is designed to run stateless workloads. These workloads typically come in the form of a microservices architecture, are lightweight, scale well horizontally, adhere to the 12-factor app principles, and can deal with circuit breakers and chaos monkeys.

Kafka on the other side is essentially a distributed database. This means you have to deal with state and it is much more heavyweight than a microservice. Kubernetes supports stateful workloads but you have to treat it with caution as Kelsey Hightower points out in two recent tweets:

Now should you be running Kafka on Kubernetes? My counterquestion is: Does Kafka run better without it? That’s why I’m going to point out how Kafka and Kubernetes might complement each other and what pitfalls you might encounter.

Runtime

Let’s have a look at the basic stuff first — the runtime itself.

Process

Kafka brokers are CPU-friendly. TLS might introduce some overhead. And Kafka clients need more CPU if they are using encryption, but this does not impact the brokers.

Memory

Kafka brokers are memory-eaters. The JVM heap can be usually limited to 4–5 GB, but you also need enough system memory because Kafka makes heavy use of the page cache. In Kubernetes, set the container resource limits and requests accordingly.

Storage

Storage in containers is ephemeral — data will be lost after a restart. You could also use an emptyDir volume for the Kafka data which would have the same effect: your broker data will be lost after termination. Your messages might still be available as replicas on other brokers. So after a restart, the failing broker first has to replicate all the data which might be a time-consuming process.

That’s why you should be using persistent storage. Use non-local persistent block storage with XFS or ext4 to be more precise. Don’t use NFS. I warned you. Neither NFS v3 nor v4 will work. In short, a Kafka broker will terminate itself if it’s not able to delete a data directory due to the NFS “silly rename” problem. If you still don’t believe me then read this blog post very carefully. The storage has to be non-local so that Kubernetes is more flexible in choosing another node after restarts or relocations.

Network

As with most distributed systems, Kafka performance heavily depends on low network latency and high bandwidth. Don’t be tempted to put all brokers on the same node as this would reduce availability. If a Kubernetes node goes down then the whole Kafka cluster goes down. Don’t stretch a Kafka cluster across data centers either. The same applies to Kubernetes clusters. Different availability zones are a good trade-off.

Configuration

Plain Manifests

The Kubernetes website contains a very good tutorial on how to set up ZooKeeper using manifests. As ZooKeeper is part of Kafka this is a good starting point to learn which Kubernetes concepts are being applied. Once understood, you can use the same concepts for a Kafka cluster, too.

Pod: A pod is the smallest deployable unit in Kubernetes. It contains your workload and it represents a process in your cluster. A pod contains one or more containers. Each ZooKeeper server in the ensemble and each Kafka broker in the Kafka cluster will run in a separate pod.

StatefulSet: A StatefulSet is a Kubernetes object that deals with multiple stateful workloads which need coordination. StatefulSets provide guarantees about the ordering and uniqueness of the pods.

Headless Services: Services decouple the pods from clients through a logical name. Kubernetes takes care of load-balancing. However, with stateful workloads such as ZooKeeper and Kafka, clients have to communicate with a specific instance. This is where headless services come into play: as a client you still get a logical name but you don’t have to access the pod directly.

Persistent Volume: They are needed to configure non-local persistent block storage as mentioned above.

Yolean provides a comprehensive set of manifests to get you started with Kafka on Kubernetes.

Helm Charts

Helm is a package manager for Kubernetes comparable to OS package managers like yum, apt, Homebrew or Chocolatey. It allows you to install predefined software packages described in Helm charts. A well curated Helm chart simplifies the complex task of properly configuring all the parameters to run Kafka on Kubernetes. Several charts for Kafka are available: An official one in incubator state, one from Confluent and another one from Bitnami, to name a few.

Operators

Due to some limitations of Helm another tool is becoming quite popular: Kubernetes operators. An operator not only packages a software for Kubernetes, it also allows you to deploy and manage a piece of software for Kubernetes.

The list of awesome operators mentions two operators for Kafka — one of them being Strimzi. Strimzi makes it really easy to spin up a Kafka cluster in minutes. Almost no configuration is needed and it adds some nifty features like inter-cluster point-to-point TLS encryption. Confluent also announces an operator which will be coming soon.

Performance

Running performance tests to benchmark your Kafka installation is very important. It gives you insights about possible bottlenecks before you run into trouble. Luckily, Kafka already ships with two performance test tools: kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh. Use them intensively. For reference, you can use the results from this blog by Jay Kreps or this review of Amazon MSK by Stéphane Maarek.

Operations

Monitoring

Visibility is very important otherwise you won’t know what’s going on. Nowadays there is decent tooling to do monitoring with metrics in a cloud native manner. Prometheus and Grafana are two popular tools. Prometheus can collect metrics from all the Java processes (Kafka, Zookeeper, Kafka Connect) with the JMX exporter in a straightforward way. Adding cAdvisor metrics gives you additional insights about Kubernetes resource usage.

Strimzi has a very nice example Grafana dashboard for Kafka. It visualizes key metrics like under-replicated and offline partitions in a very intuitive way. It complements those metrics with resource usage and performance as well stability indicators. So you get basic Kafka cluster monitoring for free!