In this post we discuss the unique requirements that Kubernetes Custom Resources bring towards Declarative Application Management and present KubePlus API add-on that addresses them.

Declarative Application Management in Kubernetes

Declarative Application Management in Kubernetes refers to the approach of creating and managing application stacks declaratively. There is a community document describing the notion of Declarative Application Management in quite detail here. It refers to the elements of application stacks as -

Bespoke applications — stateless application servers.

Common off-the-shelf (COTS) components — infrastructure software and stateful systems, such as databases, key-value stores, caches, and messaging systems.

The document further explores requirements for tools that support Declarative Application Management in Kubernetes. The application stacks mentioned here are commonly referred as platform stacks as well, in which the various components are essentially platform stack elements. The foundation for declarative management of these platform stacks on Kubernetes is Kubernetes Resources — built-in Resources (Pod, Service etc.) along with Custom Resources added by the installed Operators in the cluster. When application developers leverages these built-in and Custom Resources to define their platform stacks declaratively, we can simply refer to that representation as Platform-as-Code.

A unique aspect of Kubernetes Custom Resources is that they are not known a priori in a cluster. Different Kubernetes clusters may have different Custom Resources depending on the Operators installed on that cluster. Consequently, when working with Custom Resources, application developers are faced with the following questions:

How to discover the capabilities of various Custom Resources present in a cluster?

the capabilities of various Custom Resources present in a cluster? How to define binding between various Resources (specially between Custom Resources) to realize the dependencies in a stack?

In the community there exist tools such as kubectl, Helm, Kustomize, etc. that address some of the aspects around declarative application management. However, the above listed requirements of Custom Resources are unique and are not completely satisfied by these tools. For instance,

For discovery , kubectl now supports ‘kubectl explain’ on Custom Resources to find out information about their Spec Properties. However, additional information beyond Spec properties is needed when working with Custom Resources, such as what are the supported workflow operations, or what are the assumptions made by the Operator developer etc.

, kubectl now supports ‘kubectl explain’ on Custom Resources to find out information about their Spec Properties. However, additional information beyond Spec properties is needed when working with Custom Resources, such as what are the supported workflow operations, or what are the assumptions made by the Operator developer etc. For binding between Resources, Kubernetes provides ‘labels’, ‘label selectors’ and name-based dns resolution. However, when using Custom Resources from different Operators these built-in mechanisms are not enough. Correct binding may require integrating runtime information across Custom Resources, or orchestrating actions on multiple Custom and/or built-in resources.

In the following table we present the level of support that existing community tools offer towards defining platform stacks declaratively consisting of Custom Resources. The last column presents KubePlus API add-on which fills the identified gaps specific to Custom Resources. KubePlus API add-on is built to augment existing tools like Helm and Kustomize.

Custom Resource Declarative Application Management with KubePlus

KubePlus API add-on

KubePlus API add-on provides new discovery endpoints, binding functions, and an orchestration mechanism to enable application developers to construct platform stacks consisting of Kubernetes Custom Resources. These constructs are implemented using the following components — an Aggregated API Server, a Mutating webhook, and an Operator/CRD.

- Discovery Endpoints

For static and runtime information discovery, KubePlus defines following custom endpoints:

> Man endpoint:

kubectl get — raw “/apis/platform-as-code/v1/man”

Usage example on MysqlCluster Custom Resource

kubectl get — raw “/apis/platform-as-code/v1/man?kind=MysqlCluster”

The man endpoint is used for obtaining static usage information about a Custom Resource. It is a mechanism that an Operator developer can use to expose any assumptions or usage details about the Operator or its Custom Resources that go beyond Custom Resource Spec properties. (For output details of above command, visit https://github.com/cloud-ark/kubeplus)

> Composition endpoint:

kubectl get — raw “/apis/platform-as-code/v1/composition”

Usage example on MysqlCluster Custom Resource

kubectl get — raw “/apis/platform-as-code/v1/composition?kind=MysqlCluster&instance=cluster1”

The composition endpoint is used for obtaining runtime composition tree of Kubernetes Resources that are created as part of handling a Custom Resource instance. (For output details, visit https://github.com/cloud-ark/kubeplus)

- Binding Functions

KubePlus API Add-on enables binding between Custom Resources through binding functions that can be used in YAML definitions of Custom Resources. We currently support the following functions that can be used to glue different Custom Resources together.

1. Fn::ImportValue(<Parameter>)

This function imports value of the specified parameter into the Spec where the function is defined.

Here is an example of using ImportValue function to bind moodle1 (Instance of Moodle Custom Resource) to cluster1 (Instance of MySqlcluster resource). In this case binding needs to happen to a specific Service sub-Resource created by cluster1.

2. Fn::AddLabel(label, <Resource>)

This function adds the specified label to the specified resource.

- Platform-as-Code Annotations

For correct working of discovery endpoints and binding functions, following annotations need to be defined on Custom Resource Definition (CRD) YAMLs of an Operator.

platform-as-code/composition

The ‘composition’ annotation is used to define Kubernetes’s built-in Resources that are created as part of instantiating a Custom Resource instance.

platform-as-code/usage

The ‘usage’ annotation is used to define usage information for a Custom Resource. The value for ‘usage’ annotation is the name of the ConfigMap that stores the usage information.

As an example, annotations on MysqlCluster Custom Resource Definition are shown below:

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

name: mysqlclusters.mysql.presslabs.org

annotations:

helm.sh/hook: crd-install

platform-as-code/composition: StatefulSet, Service, ConfigMap, Secret, PodDisruptionBudget

platform-as-code/usage: mysqlcluster-usage.usage

spec:

These annotations are used by KubePlus to enable discovery and binding functions. If you are a Operator developer, add these platform-as-code annotations on your CRD YAMLs.

Custom Resource Ordering

KubePlus API add-on also consists of a CRD (PlatformStack) that can be used to define order for creating Custom Resources. It enables application developers to define all the stack resources as a unit along with the inter-dependencies between them. The dependency information is used by mutating webhook to prevent out-of-order creation of resources.

KubePlus in the Community

We demoed KubePlus API add-on on August 15th in the Kubernetes Community Meeting. Check out the Kubernetes Community Meeting notes here. Our slide deck used in the meeting is here. And the demo here.

Conclusion

As enterprise platform engineering teams are building their custom platform layers leveraging Kubernetes Operators, they are looking for ways/tools to share with application developers that will simplify consumption of these platform layers for them. As described above, Platform-as-Code approach and KubePlus API add-on is designed to fill the gaps in existing declarative management tools specifically for Custom Resources without introducing any new CLI so that teams can easily build their automation on top of Kubernetes native interfaces. Give it a try (https://github.com/cloud-ark/kubeplus) and let us know your feedback.

www.cloudark.io