In previous blog post we looked at under the hood of the Kubebuilder framework. In this post we will look at the Operator SDK.

Operator SDK is part of the recently announced Operator Framework. The goal of Operator SDK is to provide high-level abstractions that simplify creating Kubernetes Operators.

Before we dive into details of Operator SDK here are some Kubernetes terms that you should get familiar with:

CRD: Custom Resource Definition. Object: An instance of Kubernetes basic Type/Kind or an instance of a CRD. Object Key: Unique string representing an Object. It is parsed from the Object. Informer: Mechanism that periodically queries Kubernetes API Server to monitor changes to Objects of particular Type/Kind. It retrieves changed Objects and stores them in a local Index. Here ‘local’ means where the custom Operator is running. SharedIndexInformer: A data structure that manages a shared Index amongst multiple Informers for storing Objects. Work Queue: Data structure where Object Keys are stored before they are processed. Lister: Mechanism for querying Objects from local Index. Client/ClientSet: Mechanism for querying Objects directly from Kubernetes API Server. Querying Objects from local Index using the Lister is preferred over querying the Kubernetes API Server using a Client so as to reduce the load on the API server.

Here is a pictorial representation of Operator SDK and how it interfaces with custom Operator code.

Operator SDK and Custom Operator interactions

We have divided the picture into two sections: Operator SDK (top section) and custom Operator (bottom section).

Operator SDK components:

1) informer: The informer Type holds key data structures used by Operator SDK. They are the SharedIndexInformer and the Work Queue. The SharedIndexInformer periodically queries Kubernetes API Server using List and Watch methods to retrieve any changed Objects. Event handling functions are defined for reacting to add, update, and delete events for an Object. These functions parse an Object’s Key and add it to the Work Queue.

2) informer-sync.go: This file defines the functions that process the Object Keys from the Work Queue. The processNextItem function retrieves the Keys and calls the sync function. This then creates an Event Object from the Key and calls the registered Handle function of custom Operator code.

3) api.go: This file provides the public API (the entry point functions). These are used for registering custom Operator’s Handle function, creating Informers for CRDs and Kubernetes-native types that we want to react to (watch), and starting the Informers.

Custom Operator components:

1) handler.go: This file is where we write custom Operator’s reconciliation code. It is generated by Operator SDK. A stub Handle function is provided in the file. This function is invoked by the Operator SDK inside the sync function in informer-sync.go file. An Event that captures an Object Key from the Work Queue is passed to the Handle function.

2) main.go: This file performs the setup steps for connecting custom Operator with the Operator SDK. It is generated by Operator SDK. The typical pattern which is used in this file consists of following actions: (a) set watches on required types (CRDs or Kubernetes-native Kinds), (b) register the handler, (c) start the informers. Respective functions from api.go are used for performing these actions.

Comparison with the start from scratch approach:

Here are some takeaways after comparing Operator SDK approach with the start from scratch approach.

Similarities:

Both approaches provide tools to generate the base go files. In the start from scratch approach, a script is provided for this purpose (hack/update-codegen.sh). Similar to that the Operator SDK provides a command-line tool.

Differences:

Compared to the start from scratch approach, the directory structure generated by Operator SDK is significantly different. In the start from scratch approach, typed Client and Lister for our CRDs are generated by update-codegen script. Operator SDK does not generate these. Instead, it internally creates resource clients by utilizing discovery and REST mapping functions offered by the client-go library.

Advantages of using Operator SDK over start from scratch approach:

When writing your Operator, you are not required to manage the Work Queue or the SharedIndexInformer. This is abstracted by the informer Type provided by the Operator SDK. With the start from scratch approach, you have to create the work queue and also create Informer and event handler functions. Then you have to wire them up to receive the changed Objects from the Kubernetes API Server. Moreover, if you want to use a local Index, you have to create an Indexer as well. When using the Operator SDK you don’t have to worry about any of this.

Concerns of using Operator SDK over start from scratch approach:

The main concern with the Operator SDK might be its performance. In the sample-controller, which is used as the starting point in the start from scratch approach, Listers are used to cut down on the traffic to the Kubernetes API server. In the design of the Operator SDK we did not find any use of Lister functionality. So this might lead to more load on the API Server (we plan to experimentally evaluate this soon). For now, we have opened an issue on the Operator SDK repository to understand the rationale behind this design decision.

Conclusion:

Operator SDK is a well documented code base making it straightforward to understand. The number of packages are small and the breakdown of functionality into different packages is logical. While some refactoring can help, the current code base is well written. As a user, you will not have to learn many new abstractions. Basic understanding of the Event Type defined by the Operator SDK and how it is handled internally should be sufficient. The main thing that you should carefully consider is its performance impact on the Kubernetes API server as compared to the start from scratch approach.