Kubernetes Custom Resources are typically introduced by Kubernetes Operators. They extend Kubernetes API to manage third-party software as native Kubernetes objects, e.g. Postgres. Standard Kubernetes interfaces (Kubernetes YAML and kubectl) work with Custom Resources just like any other native Kubernetes resources such as Pod, Deployment etc. However, to meaningfully consume Custom Resources and build application platforms using them, application developers need to know more about Custom Resources such as their composition in terms of underlying native resources, Spec Definition, supported operations, configurables, etc. We have developed a Kubernetes Aggregated API server named Kubediscovery to help in this regard.

Kubediscovery is a component in our KubePlus platform toolkit — a toolkit that delivers Platform-as-Code experience by streamlining the process of deploying, managing and consuming Kubernetes Operators. Here is high-level architecture diagram of KubePlus where Kubediscovery is shown as Discovery API server component within it.

KubePlus Platform Toolkit

Kubediscovery Implementation

Kubediscovery is a Kubernetes Aggregated API server that offers new API endpoints for delivering additional information about Custom Resources.

Kubediscovery

Current implementation supports following endpoints.

composition — Retrieve dynamic composition tree of Custom Resource instance in terms of the underlying resource instances (e.g. which underlying resource instances are part of the composition tree of a Postgres Custom Resource instance.) [/apis/kubeplus.cloudark.io/v1/composition]

explain — Retrieve OpenAPI Spec for registered Custom Resources. [/apis/kubeplus.cloudark.io/v1/explain]

Composition endpoint to retrieve Composition tree

The ‘composition’ endpoint is used for obtaining dynamic composition tree of a Custom Resource instance in terms of its underlying resource instances.

In Kubernetes, certain resources are organized in a hierarchy — a parent resource is composed using its underlying resource/s. For example, a Deployment is composed of a ReplicaSet which in turn is composed of one or more Pods. Similarly, a Custom Resource such as Postgres can be composed of a Deployment and a Service resource.

To build the dynamic composition tree, Kubediscovery needs static information about resource hierarchy of a Custom Resource. For this, Operator Developer needs to follow certain guidelines while developing the Operator. We have detailed these guidelines here. The two key guidelines in the context of Kubediscovery are: a) Define underlying resources that are part of a Custom Resource’s resource hierarchy with an Annotation on CRD registration YAML (guideline #9), and b) Set OwnerReferences for underlying resources owned by your Custom Resource (guideline #5).

Kubediscovery uses this information to follow OwnerReferences of individual resource instances and builds the dynamic composition tree. Here is an example. It shows composition tree for ‘postgres1’ instance of ‘Postgres’ Custom Resource.

Composition tree of Postgres Custom Resource Instance

Explain endpoint to retrieve OpenAPI Spec

The ‘explain’ endpoint is used to get static information about Custom Resources like what one would get from the standard ‘kubectl explain’ call. This information is essentially the OpenAPI Spec for a Custom Resource. In KubePlus, we provide a tool based on kube-openapi library to generate OpenAPI Spec for your custom resources.

Here is how you can use the ‘explain’ endpoint with Custom Resources. In the first example we use ‘explain’ endpoint to find information about ‘Postgres’ Custom Kind registered by our sample Postgres Operator. In the second example we use ‘explain’ to find information about ‘S3StorageProvider’ Custom Kind which is part of Oracle MySQL Operator’s Backup Custom Resource Spec definition.

Explain for Postgres

Explain for S3StorageProvider

Currently work is underway in upstream Kubernetes to extend ‘kubectl explain’ to work with Custom Resources. We plan to deprecate the ‘explain’ endpoint from Kubediscovery when this functionality becomes generally available through the main API server.

Conclusion

Our goal with Kubediscovery is to help application developers learn additional information about Custom Resources available on their clusters to build their application platforms. As described, currently it supports two endpoints — ‘composition’ and ‘explain’. In the future we are considering including following endpoints:

- ‘audit’: this will provide dynamic lineage tracking (provenance) of how a Custom Resource instance has changed over time.

- ‘functions’: this will provide static information about which operations are supported by a Custom Resource e.g. Postgres Custom Resource allows database creation, user addition and password change functions.

- ‘configurables’: this will provide additional information about supported configuration parameters under the Custom Resource that affect behavior of underlying platform element such as Postgres database.

We are looking for inputs from the community regarding what additional endpoints you would like to see in Kubediscovery. Also, any prioritization inputs on above mentioned endpoints are also welcome. Let us know by filing a GitHub Issue.

www.cloudark.io