Before you use Kustomize

The pros and cons of the popular Kubernetes config management tool Kustomize

Kustomize is a tool used to manage and customize YAML files- particularly Kubernetes config YAML files. It gained popularity quickly over the past two years and has become part of kubectl since v1.14. Therefore, when our organization was considering moving away from our in-house Kubernetes config management tools, Kustomize was on the top of our candidate list. Here is what I found after researching and prototyping on Kustomize.

TD;LR

Kustomize relies on a “select & patch” mechanism to simplify and refactor Kubernetes config YAML files. While this powerful approach offers many benefits, it also has limitations for certain tasks.

The problem

Our team has multiple microservices deployed in many different environments. After reviewing our config files, we found:

commonalities among configurations of different microservices, such as labels and annotations.

commonalities among configurations of the same microservice in different environments, such as DNS, CPU, and memory settings.

Therefore, our ideal tool should leverage those to refactor and avoid repeating ourselves(DRY).

A templateless solution

Inspired by the article “Declarative application management in Kubernetes,” Kustomize despises the use of complex templates and solves the problem by decomposing YAML files into base YAML files and overlay YAML files, then assembling them depending on the situation. In general, Kustomize has three types of files:

Base YAML files: these define the skeletons of Kube YAML files.

these define the skeletons of Kube YAML files. Overlay YAML files : these are small bits of Kube YAML files, such as specs for a volume mount or a container.

: these are small bits of Kube YAML files, such as specs for a volume mount or a container. Kustomization files: these define how overlay files should be applied to base files in order to produce the final YAML files, called variants. They can also contain lightweight overlays such as common labels and annotations, as well as generate config maps and secrets. Typically each environment has its own kustomization file to tweak the configurations for that environment. For example, the dev kustomization file adds a sidecar container to a particular deployment while the production kustomization file does not.

What distinguishes Kustomize from a generic YAML file merger is that Kustomize understands Kube YAML syntax. For example, a feature in Kustomize named Target Selector allows users to apply overlays only to deployment files that contain specific labels or annotations in their Kube metadata.

What’s good

Compared to template solutions, Kustomize forces people to avoid common pitfalls when writing config files. A typical templated config file consists of 3 parts:

the part that doesn’t change after rendering(texts)

the part that changes after rendering(templates)

the control logic that combines them

When the file is filled with templates and if…else statements to handle different scenarios, it becomes extremely unreadable and hard to maintain. In the meantime, the lack of integration with Kube syntax also makes some tasks, such as adding a name prefix to all Kube resources, tedious and error-prone, because users need to go over all files and manually add templates for name prefix.

Kustomize promotes good practice by separating the three parts of templated config files. Since Kustomize can only combine raw templateless Kubernetes YAML files, all control logic such as if…else statements must stay in Kustomization files. With the help of base YAMLs and overlay YAMLs, Kustomize encourages putting the fixed text into base YAMLs and the changing part in overlay YAMLs. Meanwhile, Kustomize has a series of handy transformers that understands Kube syntax. They can be used to avoid human mistakes and apply changes to all Kube resources like common labels.

What’s not so good

Trade-offs are burdens that all software designs have to bear, and Kustomize is no exception. Kustomize can help to refactor and customize configurations for different environments, but it has limitations when it comes to refactoring configurations shared among microservices.

Kustomize lets users create overlay YAMLs and use Kustomization files to decide how to apply them to base YAMLs. It seems like a good idea to leverage that and extract some common attributes of Kube resources into overlay files, such as labels, security contexts, and environment variables. However, that only works if Kustomize can put the extracted parts back to their original places. It might sound very straightforward, but it is not always simple with Kustomize.

There are two ways to let Kustomize know where to apply the overlay:

by paths : the path-based method allows users to embed the paths to override in overlay files’ YAML structure, or explicitly state that info in a Kustomization file with a Json6902 transformer. This approach is not very helpful in refactoring, because it requires a 1-on-1 mapping between overlays and paths in base YAMLs, and no parts can be shared.

: the path-based method allows users to embed the paths to override in overlay files’ YAML structure, or explicitly state that info in a Kustomization file with a Json6902 transformer. This approach is not very helpful in refactoring, because it requires a 1-on-1 mapping between overlays and paths in base YAMLs, and no parts can be shared. by selectors: the selector approach, such as Target Selector, is much more suitable for refactoring. It allows users to specify a selector, for example, all deployments with specific annotations, and apply an overlay to those deployments.

However, not all elements in Kube config YAMLs are “selectable,” because selectors always rely on some attributes, such as the kinds of Kube YAML, labels, or annotations to select the Kube elements. Still, not all of Kube elements have those. For example, a Kube deployment can have a list of containers, but there are no annotation-like attributes on those containers for Kustomize selectors to use. In order to apply a common overlay to containers in multiple deployments, the containers must have the same name. This makes refactor tasks such as common environment variables or container security contexts nearly impossible.

Should I use Kustomize?

The problem mentioned above is not entirely unsolvable, but it is somewhat fundamental. To make every part of Kube YAML “selectable,” it calls for a change or hack on Kube object schema, e.g., adding annotation-like fields to every object, or reusing some existing fields as annotations and then extending Kustomize to support them.

However, the problem is only likely to appear when the complexity of microservices reaches a certain level. Overall, I think Kustomize’s advantages still outweigh its disadvantages, especially if your team hasn’t yet tried a Kube config management solution.