One of the interesting aspects about Kubernetes is that you can extend its API to build a fit-for-purpose platform leveraging underlying power of Kubernetes. The mechanisms available for implementing API extensions are Custom Resource Definition (CRD) and Aggregated API Server (AA). The APIs built using these mechanisms are accessible directly using kubectl, no other CLI is required to access them.

In this post we look at the API extensions from the point-of-view of basic constructs that are involved in an extended API. Using this lens, we are able to consider various patterns that are possible by combining these basic constructs together. We then present which patterns are possible to implement using Custom Resource Definition vs. which are possible using Aggregated API Server.

Basic Constructs in an Extended Kubernetes API

There are three basic constructs involved in Kubernetes extended API. They are: (a) Custom Kind (b) Custom controller (c) Custom sub-resource.

Custom Kind: A custom Kind is a construct that allows you to define your domain-specific requirements in a declarative format. For example, a custom Kind for Postgres can support declarative model for creating databases and users. A custom Kind is similar to existing native Kubernetes Kinds like Pod, Service. It includes metadata section, spec section, and status section. Custom Controller: A custom controller is a Kubernetes controller that performs reconciliation of cluster’s state by watching create/update/delete events on Kubernetes Kinds in your cluster. These could be native Kinds (Deployment, Pod, Service, etc.) or custom Kinds (such as Postgres). Custom Sub-resource: A custom sub-resource on a Kubernetes Kind (native or custom) allows you to define fine-grained actions on it’s Kind. In absence of a sub-resource, Kubernetes allows only basic CRUD actions on any Kind. An example of a custom sub-resource can be http_requests_total defined on a Pod will allow you to find out the number of HTTP requests received by that Pod.

Not every extended API includes all these constructs. In fact, there are four patterns that we have observed in any extended API. They are as follows:

Custom Kind + Custom Controller: This is the most popular pattern. In this pattern you use a custom Kind to declaratively model your domain requirement. The custom controller includes the logic to reconcile state of the cluster by reacting to different events corresponding to the custom Kind (primarily). The popular name for this pattern is Operator Pattern. This pattern can be implemented using Both CRD and AA mechanisms. Custom Kind + Custom Controller + Custom sub-resource: Like above, this pattern is also Operator Pattern. It also supports custom actions (beyond CRUD) on your custom Kinds through the custom sub-resource. This pattern can be implemented only using AA mechanism. Custom Controller + Custom sub-resource: This pattern does not define a new Kind. The custom controller executes its reconciliation logic on events related to Kinds that are already registered in the cluster. The custom sub-resource is used to retrieve information that is collected by the custom controller. An example of this is a controller that collects and maintains information about dynamic composition of various Kubernetes Objects and the custom sub-resource is used to get this information. This pattern can be implemented only using AA mechanism. Custom sub-resource: In this pattern there is no custom controller. The custom sub-resource is typically used as a way to find out some information about some Kubernetes Object. An example of this a metrics server that pulls information from Prometheus and makes it available through custom sub-resource such as http_requests. This pattern can be implemented only using AA mechanism.

Here is a table summarizing these patterns, the supported API extension mechanisms for implementing them, along with pointers to examples where you can see the pattern and reference tools for development phase.

Custom Kind + Custom Controller: You can use either CRD or AA when implementing this pattern. When using CRD look at the sample-controller first. It is a good example of how a CRD is written, installed, and used. You can also look at our Postgres Operator to see an example CRD modeled on sample-controller. Check out kubebuilder and Operator SDK. These tools reduce the boilerplate code that needs to be written when developing CRD starting from scratch using sample-controller. To get better understanding about inner workings of these tools, check out our analyses of them here (sample-controller), here (kubebuilder), and here (Operator SDK). When using Aggregated API server, sample-apiserver is a good starting point. Custom Kind + Custom Controller + Custom sub-resource: When working with this pattern, apiserver-builder tool can help with creating the starter code for your server. It will help you with creating custom Kinds, custom controller, and custom sub-resources on your custom Kinds. Custom Controller + Custom sub-resource: Our kubediscovery API server is a good example for this case. The custom controller periodically polls Kubernetes’s main API to build dynamic composition tree of various Kubernetes Objects. In the next version of the controller we plan to modify its architecture to use a watch/react model instead of the polling model. The custom sub-resource that we have defined is “/compositions”. This can be used with any Kubernetes Kind, native or custom, to find out dynamic composition information for all or particular instance of a Kubernetes Object. Custom sub-resource: The custom-metrics-apiserver is a good example for this pattern. It defines ‘pods/http_requests’ sub-resource which can be used to find out number of HTTP requests received by a Pod. Internally it obtains this information from Prometheus instance running in the cluster.

The first two patterns support the declarative model of state reconciliation of Kubernetes. The custom Kind available in these patterns is used to define the desired state and the custom controller is used to define the reconciliation logic. Whenever a new custom Kind is introduced, that pattern can also be called as Operator Pattern.

The last two patterns are focused on providing ability to perform custom actions on Kubernetes Objects. Current examples — custom-metrics-apiserver, kubediscovery— show how to perform actions that retrieve custom state information from the cluster. Without custom sub-resource, only CRUD actions are possible.

Conclusion:

Kubernetes’s API extension mechanisms of Custom Resource Definition and Aggregated API server consist of three basic constructs — custom Kind, custom controller, and custom sub-resource. Through analysis of existing examples we have identified four patterns combining these constructs that are seen today. By looking at API extension mechanisms through this lens of extension patterns we hope that you will choose the right mechanism, examples, and tools appropriate when developing your extended API.

www.cloudark.io