Using ‘kubectl explain’ for Custom Resources

Goal: Explore if ‘kubectl explain’ can be used to discover static information about Custom Resources

Recently we have been working on building a Kubernetes aggregated API server that helps with discovering information about custom resources in a cluster. For instance, static information about a custom resource can be — what actions can be performed on it, or what configurable parameters of are exposed and how can they be modified. Example of dynamic information about a custom resource can be — which native Kubernetes resources comprise a particular instance of a custom resource and their statuses.

Towards building this server we have been studying the workflow of ‘kubectl explain’ command as it seems a good starting point to investigate if we can use it to support discovery of static information for custom resources. This post first presents how ‘kubectl explain’ works. We then present our analysis on whether it can be used for our use-case.

Source of ‘kubectl explain’ output

kubectl explain command is used to show documentation about Kubernetes resources like pod.

Example

# kubectl explain Pods

The information displayed as output of this command is obtained from the OpenAPI Specification for the Pod. OpenAPI Spec for all the Types is generated by the main API server when it starts up and is maintained by it in memory. Below we explain details of this process.

Behind the scene of kubectl explain

Working of kubectl explain consists of two phases. First phase happens when the main API server starts up. In this phase the OpenAPI Spec is built and maintained in memory to serve kubectl explain calls. The second phase is the actual ‘kubectl explain’ command execution. In this phase, the OpenAPI Spec for the requested resource is fetched from the server, parsed, and displayed.

Phase 1: Main API server start up and building of OpenAPI Spec

This phase is divided into two steps as shown in the following diagram.

The first step consists of walking the registered Kubernetes REST API resource endpoints and generating in-memory representation of OpenAPI Spec for all the resource. It starts at genericapiserver.go’s PrepareRun() function. The call graph includes buildPaths() method in kube-openapi/../builder.go package. This method works with the go-restful webservice endpoints that are registered with the main API server and builds an in-memory representation of the OpenAPI Spec. There is a way to download this in-memory representation of the OpenAPI Spec as JSON files using kubernetes/hack/update-openapi-spec.sh script. This will download the json file of the Spec and save it as swagger.json in kubernetes/api/openapi-spec folder.

The second step consists of getting the server ready for serving the generated OpenAPI Spec at a well-known URL e.g. /openapi/v2, or /swagger.json. This is done by the kube-aggregator.

Phase 2: Execution of ‘kubectl explain’ command to retrieve and parse the OpenAPI Spec

Displaying the information in output of ‘kubectl explain’ consists of first retrieving it from the well-known URL path and then parsing it. The components involved in this process are shown in the following diagram.

The process starts at explain.go’s NewCmdExplain() function. It is passed a reference to the cmd util factory object. The factory_client_access.go contains OpenAPISchema() function that is the starting point of the call graph which retrieves the OpenAPI Spec using client-go. The retrieved spec is then parsed using various functions available in the explain package.

Observations

1) One issue with the output of ‘kubectl explain’ is that it does not show the entire API path of an Object. For instance, ‘kubectl explain Pods’ does not show that Pods are accessible at the path: api/core/v1/. If we can get this entire path, it will be easier to use ‘kubectl get --raw’ or curl command with it.

Such information is available in the OpenAPI Spec but is just not displayed by ‘kubectl explain’.

If you have downloaded the in-memory OpenAPI spec, you can look this up in the definitions section in swagger.json. Pod definition is included under “io.k8s.api.core.v1.Pod”. The information that is shown as output of ‘kubectl explain’ is obtained from the description field of this definition.

2) If you want to update description of a native Kubernetes resource (Type), you can follow these steps.

- Modify description text at appropriate place (for Pod change it here)

- Run kubernetes/hack/update-openapi-spec.sh.

This script starts the main api server, which will generate OpenAPI Spec as outlined in Phase 1 section above. The script will also download the json file of the Spec and save it as swagger.json in kubernetes/api/openapi-spec folder.

Note that if you are using Minikube then your change will not be reflected in ‘kubectl explain’. You will have to run Kubernetes cluster from the source code for the change to take effect. You can follow the steps shown here for running a cluster from the source code.

3) The kube-openapi library contains packages and functions that are used for building the OpenAPI Spec. The library provides an annotation — “+k8s:openapi-gen=true” — which is recommended for generating OpenAPI Spec for a Type. Once you add this annotation on a Type, you can use scripts in kube-openapi package to generate OpenAPI Spec file for that Type. However, this annotation is not present on Kubernetes native Types such as Pod. The OpenAPI Spec for the native types is generated run time by parsing API paths as outlined in Phase 1 section above.

Conclusion

It seems that the only way to use ‘kubectl explain’ to discover static information for a custom resource is if that information can be served through the main API server’s OpenAPI Spec. This Spec is constructed by walking the registered API endpoints when the main API server starts. The Spec is not regenerated afterwards. When a custom resource is registered using Custom Resource Definition, the main API server is already running. It is not restarted, so OpenAPI Spec for the custom resource cannot be generated. Similar logic applies when a custom resource is registered using APIService Object using an aggregated API Server approach. Thus, there seems to be no way whereby ‘kubectl explain’ can be used to discover static information about custom resources.

One option that we are considering is to build a custom subresource named ‘explain’ that can be used to get this information, as follows:

kubectl get --raw /apis/kubediscovery.cloudark.io/v1/namespaces/default/Postgres/

explain

Here we have ‘Postgres’ as a custom resource and we are defining ‘explain’ as its subresource.

Our plan is the following. We will annotate Postgres Type definition with “+k8s:openapi-gen=true” annotation and use scripts in kube-openapi library to generate OpenAPI Spec file for Postgres Type. We will load this file as a ConfigMap and our custom handler for the ‘explain’ subresource will retrieve this file, parse it, and display the required information.

If you have any other ideas/suggestions, we are eager to hear them.

You can track the discussion about this issue on Github here.

www.cloudark.io