Recently there was a question on Kubernetes subreddit as to what should be used for creating custom controllers/Operators. Should we use Kubebuilder, or the Operator SDK, or prefer a start from scratch implementation? I answered on that thread, but wanted to provide more detailed analysis.

In this post we will see under the hood of the Kubebuilder framework and compare it with the start from scratch approach. In a subsequent post we will do similar exercise with the Operator SDK.

Kubebuilder is a framework for creating Kubernetes custom controllers that work with Custom Resource Definitions (CRDs). It is currently hosted in the ‘kubernetes-sigs’ Github organization and is part of the ‘k8s-sig-api-machinery’ Special Interest Group.

Before we dive into details of Kubebuilder here are some Kubernetes terms that you should get familiar with:

CRD: Custom Resource Definition. Object: An instance of Kubernetes basic type/kind or an instance of a CRD. Object Key: Unique string representing an Object. Informer: Mechanism that periodically queries Kubernetes API Server to monitor changes to Objects of particular type/kind. It retrieves changed Objects and stores them in a local Index and also in a Queue. ‘local’ here means where the custom controller is running. Queue: Data structure where Objects are stored before they are processed. Lister: Mechanism for querying Objects from local Index. Client/ClientSet: Mechanism for querying Objects directly from Kubernetes API Server. Querying Objects from local Index using a Lister is preferred over querying the Kubernetes API Server using a Client as it reduces the load on the API server.

Here is a pictorial representation of kubebuilder and its interaction points with custom controller:

kubebuilder and custom controller interaction

We have divided the picture into two sections: kubebuilder (top section) and custom controller (bottom section). We will start with explaining kubebuilder components first.

Kubebuilder components:

1) Generic Controller: kubebuilder provides a generic controller that acts as a wrapper for our custom controller. It is based on the sample-controller. It defines the queue in which Objects are delivered by Informers using event handlers (not shown). The queue itself is not exposed to our custom controller. Generic controller also defines a variable that will hold reference to the Reconcile function that our custom controller will implement. Generic controller passes Object key for an Object stored in the queue as a parameter when it calls the Reconcile function in our controller.

2) Controller Manager: The ControllerManager manages custom controllers. It maintains two data structures: list of controller references and a map of Objects and their Informers. It provides following functions to populate the two data structures: AddInformerProvider, AddController, RunInformersAndControllers.

3) Injector: An Injector defines two data structures: a list for storing CRDs that need to be registered and a list for storing functions that should be invoked for starting informers and controllers. The CRD list will be populated by the CRDs whose Objects our custom controller will be monitoring. The registration of the CRDs is done in the setup phase. The function list is also populated during the setup phase.

Custom controller components:

1) controller.go: This is where we implement our controller code. A base controller will be generated by kubebuilder containing a Type definition for our controller.

2) Custom Controller: This is the Type that represents our controller. It is generated by kubebuilder. It defines variables for a Lister and a Client for our CRDs. A stubbed Reconcile function is also generated. We should write the reconciliation logic as part of this function. It will be called by the Generic controller and an Object Key will be provided to it. Moreover, a function named ‘ProvideController’ is also generated. It adds a reference to the Reconcile function from our controller to the Generic controller and returns the generic controller instance as return value.

3) zz_generated.kubebuilder.go: This file contains the steps to set up various data structures in the Controller Manager and the Injector. It is generated by kubebuilder.

4) main.go: This file contains the steps to start the Informers and the Controllers. It is also generated by kubebuilder.

Comparison with the start from scratch approach:

Here are some takeaways of comparing the Kubebuilder approach with the start from scratch approach.

Similarities:

Both approaches generate basic go files (types.go, register.go, etc.). In the Kubebuilder approach, the kubebuilder cli takes command-line flags that control the file and directory names and their locations. In the start from scratch approach you have to use the ‘hack/update-codegen.sh’ script to generate these files. Both approaches require you to understand Kubernetes concepts of Lister and Client (explained above).

Advantages of using Kubebuilder over start from scratch approach:

Basic directory structure is generated by kubebuilder. With the start from scratch approach, you have to create the ‘pkg’ directory with placeholder sub-directories for ‘client’ and ‘apis’. The hack/update-codegen.sh script will then generate the files inside these directories. When writing your controller, you are not required to manage per controller queue. This is abstracted by the Generic Controller provided by kubebuilder. With the start from scratch approach, you have to create the work queue and create Informer and event handler functions. You have to then wire them up to receive the changed Objects from the Kubernetes API Server. Moreover, if you want to use a local Index, you have to create an Indexer as well. You don’t have to worry about any of this when using kubebuilder. kubebuilder defines a Type for your Controller. It maintains Lister and Client references for your CRD. This type definition provides a single place to look for a Lister or a Client anytime you need to use them in your code. In the start from scratch approach, you have to manage such references yourself.

Concerns of using Kubebuilder over start from scratch approach:

The main concern of using Kubebuilder over the start from scratch approach are the new abstractions that are introduced by kubebuilder — GenericController, ControllerManager, Injector, InjectorArgs, Installer, Install_strategy, etc. Additionally, there are new functions such as ProvideController, AddInformerProvider, RunInformersAndControllers, etc. You need good grasp of these abstractions and functions to build a mental model of the workings of kubebuilder and how it interfaces with your controller.

Conclusion:

While Kubebuilder is certainly useful, it remains to be seen if its approach of providing just enough abstractions will be useful to developers in complex scenarios, or they would prefer working from first principles following the start from scratch approach.

www.cloudark.io