Hello,

I have discussed the specifics of stateful deployments on kubernetes with friends and colleagues many times. Now is a good time to record these discussions and have them available as reference. There are plenty of posts on how to deploy stateful workloads on kubernetes using helm charts or operators. In this article, I would like to focus on the question whether “it is worth it” or “when it makes sense”.

Why not to move stateful workloads

Every discussion on stateful services starts with the same 3 to 5 arguments, why not to move them to k8s:

It was not designed for stateful services

While this was originally true, kubernetes now have stateful sets as well as operators and frameworks around them. At this point you probably find operators for most of the popular databases on https://operatorhub.io/, some of them are supported by database vendors themselves https://www.confluent.io/confluent-operator, https://www.elastic.co/products/elastic-cloud-kubernetes, https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html

Kubernetes nodes are not reliable

New versions of Kubernetes run as reliable as any cloud compute instance with high uptimes. I have observed kubernetes nodes having hundreds of days of uptime.

Network attached file system is slow (slower than local hard drive)

While this is generally true, there are a few aspects to consider. Most EC2 instances are running with EBS network attached storage that behaves similar to a PersistentVolumeClaim in kubernetes (slower but persistent). Alternatively you could use kubernetes emptyDir, a disposable directory inside of a container that is located on local disk. This behaves similar to an EC2 instance with NVMe attached storage (faster, but lost with the instance or container)(Upd: consider using Local Persistence Volumes as they can give You more guarantees and flexibility (like remount storage if the pod could be restarted on the same k8s node) then emptyDir or HostPath). If you are looking for high performance, then you are likely using a clustered database such as cassandra or elasticsearch. These systems often have replication configured and can handle losing some nodes depending on cluster configuration, so faster but unreliable storage might be a good option.

Not enough control of the execution environment

People usually make this point in regards to the noisy neighbor problem (some services consuming too many resources, making others unresponsive within one node). Usually bottlenecks are:

There may be other resources that should be controlled. I have seen a situation when a k8s node ran out of filesystem inodes because of a broken logger configuration.

As an advanced user you might consider using affinity and anti-affinity to collocate services requiring low network latency on a single worker or spread services requiring high disk or network resources to different nodes.

In case this was not enough to achieve good isolation, you might combine anti-affinity and label selectors to deploy single stateful service pod per worker node with dedicated hardware for that. In such cases k8s will play the role of an orchestration engine that ensures service liveness and readiness as well as the correct amount of replicas, perform rolling downtime free upgrades. For example you might have 6 kubernetes workers with high volume fast disks dedicated for kafka deployment. You could deploy 5 node kafka cluster there and have 1 node as hot standby. It will automatically start one more kafka instance on standby worker if or when 1 of the primary nodes fails (to keep desired replica count). Other use case could be rolling deployment upgrade when we provision one new node, join to the cluster and then decommission old one.

Extra complexity

This is a valid point if you do not have kubernetes installed yet. If you don’t then most likely it is not worth installing it to just move your stateful workloads there. Otherwise you might simplify your overall deployment and configuration management by using one tech stack for everything. If you could use a managed kubernetes service in the cloud the installation is not complicated.

Arguments in favor of moving stateful workloads

Now let’s focus on arguments and use cases that show advantages of having stateful workloads running on kubernetes.

Radically reduced configuration management tools

As I mentioned before, with kubernetes it is possible to deploy applications using just one methodology. This greatly reduces the complexity of deployments as it is much easier to master and keep track of one tool. It is an extra bonus to be able and create stateful clusters (Kafka, Elasticsearch or others) using operators with little initial knowledge of how to install and operate them. The only remaining question is how to provision k8s itself? You could use Terraform or another cloud management system. This may provoke an interesting shift in your organization. One group of engineers is writing software, provisioning and managing databases for them (true devops) and a separate group ensures a stable kubernetes environment with enough compute. This setup decouples teams resulting in less internal dependencies between teams, an increase in development speed and deployment frequency, especially in microservice environments.

Health and liveness checks and monitoring

Let’s be honest, does your current automation restart a database automatically when it is down or will it wake up an engineer in the middle of the night to do manual intervention instead? Helm charts and operators provide an easy already builtin way to check the health of an application and restart it based on parameters.

Did you have time to build a centralized logging and metrics collection system to monitor systems and provide post-mortem analysis on failures? Once again helm charts and operators come to the rescue. Daemon sets can collect the logs and other data in a centralized storage of your choice.

Portability

This is where kubernetes starts to shine. Besides bare-metal installations every major cloud provider now offers kubernetes clusters:

If you refrain from manual configurations and prepare everything with yaml resource definition that are stored in vcs, setting up a new clean application on a new k8s cluster becomes trivial.

This might be useful in a number of use-cases:

Temporary development environment to perform manual/regression testing of changes to one or multiple services before merging and deploying them to a permanent shared environments (such as staging). This can be helpful to find bugs during the local development lifecycle without affecting a broader group of people.

Additional permanent/shared environment: as an organization grows it may need multiple staging environments for development and QA teams. It may be necessary to provide staging environments per team or dedicated environments for QA to perform continuous deployments of all builds to ensure faster development feedback.

Local environment with the possibility to run either entire or partial deployments. This can be extremely useful for troubleshooting using a debugger.

On-Prem installations for a client: managing environments that are not fully accessible can be very hard. In case you could bootstrap the entire deployment automatically in a test cluster, then you could provide the script to the client to execute that against their cluster (in case they already have k8s and your application does not have requirement for specific hardware or volumes).

Disaster recovery: there might be downtimes in a data center or even an entire AWS regions. In this case you can spin up a new environment in another location, restore data from backups and switch DNS entries within a short time frame.

New regions: some businesses are easily sharded by have client data isolated from each other. You can vertically scale your business by adding new nodes to a cluster and horizontally scale by adding new clusters. This also keeps data closer to the customer and isolates incidents to customers of one region.

Single command deployments can be achieved via the “helm umbrella” (helm chart that specifies the dependency of each required service of a version). I am going to add a separate post about this later.

Some of the above examples do not even have strict requirements for performance or SLA’s but are intended to simplify and speedup development. They might be a good approach to try and validate before fully expanding stateful deployments to kubernetes on production.

Conclusion

While kubernetes is not the default solution for deploying stateful services, it may help to simplify the overall deployment (specifically in-case of auto scaling) with the support of automatic restarts for unhealthy deployments or simple integration with centralized logging and metrics collection. There are also plenty of configurations and options to achieve a target level of performance. However this is not free and requires knowledge and understanding of kubernetes as well as careful consideration of potential benefits and pitfalls.

Thanks Matthias Hofschen for reviewing this article.