Recently we have been studying how the Kubernetes REST API works to figure out answers to questions such as the following:

- How/where is the mapping defined from the REST paths to functions that handle REST calls?

- Where does interactions with etcd happen?

- How does the end-to-end path, from client making a request to an object getting saved in etcd, look like?

This post attempts to answer these questions. Kubernetes’s code is quite involved and so the presentation here is to the best of our current understanding.

Kubernetes REST implementation can be broadly divided into three parts as shown in the following picture.

The client/server functionality is implemented through various libraries in the k8s.io package. The server-side implementation is spread over several packages. The primary package on the server-side is ‘apiserver’. The important packages within it are ‘endpoints’, ‘server’, ‘registry’ and ‘storage’. The client-side is implemented in the client-go package. The primary package within it is ‘rest’.

Before going into details of these packages, here are some key Kubernetes terms/concepts that you should keep in mind.

Group: Kubernetes REST APIs are organized in a hierarchy with ‘/apis’ as the root. A group defines a logical name for a set of REST resources under this root. An example API group is ‘apps’, which is represented in the hierarchy as ‘/apis/apps’. You can check all the available API groups using:

kubectl get --raw /apis | python -mjson.tool

2) Version: Kubernetes REST APIs have versions. Version name is used for defining REST resource endpoints that evolve together within a Group. Typical version names are: v1, v1alpha1, v1beta1, etc. You can find all available versions of an API group using: kubectl get — raw /apis/<group-name> Example:

kubectl get --raw /apis/apps | python -mjson.tool

3) Type: A named entity representing a concept (e.g.: Pod, Deployment, Service, etc.).

4) Kind: JSON/YAML representation of a Kubernetes Type.

5) Resource: Endpoint/path that handles REST requests for a particular Kind. A resource is represented in the api hierarchy as follows: /apis/<group>/<version>/namespaces/<namespace>/<kind-plural> (e.g.: /apis/apps/v1/namespaces/default/deployments)

Server-side:

When studying the server-side we focus on answering following questions: a) Where are Resources installed in the Server? b) Where does interactions with etcd happen?

1) genericapiserver.go: This file defines APIGroupInfo type which is used to hold information about an API group, such as what versions exist and which resources are defined in those versions. It defines GenericAPIServer type which implements the InstallAPIGroup method that installs REST endpoints for an API group. This method internally calls the InstallREST method on a api groupversion instance. To start running the server, the GenericAPIServer contains the Run() method.

2) groupversion.go: This file defines APIGroupVersion type which is used to hold information about an api group’s particular version such as references to the objects that provide actual REST endpoint implementation (store.go). It also defines the InstallREST method which internally calls the Install method on installer to install REST resources for that version.

3) installer.go: This file defines the APIInstaller type which implements the Install method. This method uses REST implementation objects from the GroupVersion instance (see point #2) to install REST path to handler function mappings in the go-restful library. The pattern when using this library is to define a ‘handler’ function to be invoked for a particular REST path. This pattern is used in installer.go for setting up handlers corresponding to different resource endpoints. Example: ws.GET(action.Path).To(handler). Here ‘handler’ is the function obtained from the REST implementation object from the GroupVersion instance. It is defined to be invoked for handling GET requests on ‘action.Path’.

4) master.go: This file defines Master type which holds pointer to an instance of GenericAPIServer. It defines the InstallAPIs method which starts the registration workflow by calling the genericapiserver’s InstallAPIGroup method. The InstallAPIs method is called when a New instance of Master is created.

5) registry/rest/rest.go: This file defines various interfaces that should be implemented by any backend that wants to provide Kubernetes-like REST endpoints. The key interfaces in this file are are: Storage and StandardStorage. The references to REST implementation objects maintained by APIGroupVersion in groupversion.go are of type rest.Storage Interface.

6) registry/generic/registry/store.go: This file defines the Store type which implements the rest.StandardStorage interface. The method implementations make calls on the Storage object that is maintained by the Store type. This object is of type storage.Interface (see below).

7) storage/interfaces.go: This file defines an interface named ‘Interface’ containing methods that should be implemented by any store that we want to use for actual persistence. A key point about methods in this interface is that they only return an ‘error’ and nothing else. If any data needs to be returned by a method, it is returned through an object whose pointer is passed in that method as a parameter. As an example, check the signature of the Get method.

8) storage/etcd3/store.go: This file defines the store type that implements the storage.Interface. It uses the etcd clientv3 library to interact with etcd3.

Client-side:

On the client-side we focus on answering following question — how are REST calls made?

1) request.go: This file defines the Request type that implements methods for making REST calls on a Resource. The pattern to make a call is to first create a NewRequest object and then use fluent style method chaining to invoke a REST method (GET, POST, PUT, DELETE). The name of the resource and the namespace, which are essential for creating the complete endpoint, are defined through the chained method calls.

2) client.go: This file defines an interface named ‘Interface’ which contain REST method wrappers that return a Request object pointer (defined in request.go). It also defines RESTClient type which implements this interface. The pattern of making a REST call is to first create a NewRESTClient and obtain a NewRequest object through one of the REST method wrappers. Once NewRequest object is available the REST call is made using method chaining as explained above.

Observations:

a) Kubernetes code is extensively documented. This helps when figuring out what is happening within a piece of code.

b) There are places where Types implicitly implement methods. It will be good to collect such methods in Interfaces. This will make it readily apparent what methods are implemented by that Type. (registry package does this nicely, but installer.go is an example where this does not happen.)

c) Golang import aliases can help in figuring out which package a method comes from. While the code does use aliases, there is scope for using them more extensively.

Conclusion:

Kubernetes code base is big. In this post we have tried to go under the hood of one aspect of it — the REST API. In future posts we may do similar deep dives into other parts of Kubernetes. If you are interested in any specific parts, feel free to let us know in the responses.

www.cloudark.io