TPR Is Dead! Kubernetes 1.7 Turns to CRD

• By Eric Chiang

A new API object type called a Custom Resource Definition (CRD) will replace the existing Third Party Resource (TPR) extension mechanism over the next two Kubernetes releases. In the upcoming Kubernetes version 1.7, CRDs are introduced into the beta API namespace for features designated for official support. By version 1.8, CRDs will be the supported mechanism and TPRs will be entirely deprecated.

The Kubernetes API aims for clean generality, and intentionally avoids becoming a warehouse for application specifics. Instead, the system provides a facility for extending the API with new and specific kinds of resources required to automate management of particular applications.

The existing extension mechanism, originating in the Kubernetes 1.2 series, is known as a Third Party Resource. TPRs offer applications a place to create and access program state governed by the CRUD semantics and authenticated protection mechanisms of Kubernetes. TPRs have been widely adopted by applications leveraging Kubernetes features to facilitate clustering and to empower complex application-specific automation with custom Controllers.

CoreOS originated the Operators pattern, pairing the TPR facility with custom Controllers that encode application-specific management knowledge into software. Operators allow complex, stateful, distributed applications to be managed in terms of, and automated by, Kubernetes. The etcd Operator, for example, creates and maintains a set of resources that describe etcd clusters. The custom Controller component understands these resources and reads and updates them to perform and reflect cluster maintenance, automated scaling, and automatic version upgrades of the etcd cluster running atop Kubernetes.

Kubernetes API overview

The Kubernetes API server is responsible for storing configuration state, with validation, storage semantics, and protection rules built around API objects. Other Kubernetes components modify and monitor that state. The Scheduler, for instance, modifies cluster state to instantiate and replicate cluster work in the form of Pods, Jobs, and other execution objects. Cluster members act on changes to state, such as a node’s kubelet fetching containers and running applications to satisfy the desired state.

Kubernetes exposes a REST API, giving access to operations on a set of resources. These resources, such as Pods, Nodes, Deployments, et al, each have the same set of methods defined: Create, Get, Delete, Watch, List, Update, Patch. These methods loosely translate to HTTP verbs. API clients, like kubectl , can GET a Pod, GET all Nodes, CREATE a Deployment, or DELETE a ConfigMap. In fact, it’s easy to explore the API with an HTTP tool like curl .

For example, to list all service accounts in the default namespace:

$ curl http://localhost:8080/api/v1/namespaces/default/serviceaccounts { "kind": "ServiceAccountList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/namespaces/default/serviceaccounts", "resourceVersion": "4876" [...] }

Deprecated: Third Party Resources

Third Party Resources are a way of describing a new API entity to the Kubernetes API server. The resource is added to the cluster’s API at some given path. The resource described by a TPR supports all the usual API object methods, and access to it is controlled with Kubernetes RBAC authorization. TPRs are described by a manifest, usually in YAML, and created and manipulated with kubectl in these examples showing the description, creation, and population of a Third Party Resource named user-group.auth.coreos.com .

For these examples, the custom resource will look like:

apiVersion: auth.coreos.com/v1 kind: UserGroup metadata: name: engineering namespace: default spec: usernames: - alberto - leah - stephanie

First, create a TPR to indicate that the API should start recognizing the new API group and resource:

apiVersion: extensions/v1beta1 kind: ThirdPartyResource metadata: name: user-group.auth.coreos.com Description: A group of users. versions: - name: v1

$ kubectl create -f group-tpr.yaml thirdpartyresource "user-group.auth.coreos.com" created $ kubectl get thirdpartyresources NAME DESCRIPTION VERSION(S) user-group.auth.coreos.com A group of users. V1

After creating the TPR, clients can start interacting with the resource directly:

$ kubectl create -f groups.yaml usergroup "engineering" created usergroup "sales" created $ kubectl get usergroups NAME KIND engineering UserGroup.v1.auth.coreos.com sales UserGroup.v1.auth.coreos.com $ kubectl get usergroups sales -o yaml apiVersion: auth.coreos.com/v1 kind: UserGroup metadata: name: sales namespace: default spec: usernames: - brenton - lora

To clean up, delete all instances of the new resource, then delete the TPR object:

$ kubectl delete usergroups --all usergroup "engineering" deleted usergroup "sales" deleted $ kubectl delete thirdpartyresources user-group.auth.coreos.com thirdpartyresource "user-group.auth.coreos.com" deleted

The replacement: Custom Resource Definitions

Custom Resource Definitions refine the concept of extensible API resources and address issues and corner cases revealed by TPRs’ success in the field. A few features were remodeled or added, among them changes to the pluralization of resource names, and the ability to create non-namespaced CRDs.

Categorizing CRDs in the beta class and into a new apiextensions API group supports the refactoring of the rather miscellaneous extensions group to which TPRs belong. In Kubernetes v1.7 and up, CRDs provide a stable object for developers to employ when creating custom Controllers and Operators that extend the API with the Custom Resource concept. The example below shows the description, creation, and population of a Custom Resource Definition named usergroups.auth.coreos.com , illustrating how naming is more declarative and clearer without auto-pluralization.

apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: usergroups.auth.coreos.com spec: group: auth.coreos.com version: v1 names: kind: UserGroup plural: usergroups scope: Namespaced # Can also be cluster level using "Cluster"

The flow is identical to the TPR. Create the top level CRD, then start interacting with the individual resource:

$ kubectl create -f group-crd.yaml customresourcedefinition "usergroups.auth.coreos.com" created $ kubectl get customresourcedefinitions NAME KIND usergroups.auth.coreos.com CustomResourceDefinition.v1beta1.apiextensions.k8s.io $ kubectl create -f groups.yaml usergroup "engineering" created usergroup "sales" created

To clean up, delete all the data, then the CRD object:

$ kubectl delete usergroups --all usergroup "engineering" deleted usergroup "sales" deleted $ kubectl delete customresourcedefinitions usergroups.auth.coreos.com customresourcedefinition "usergroups.auth.coreos.com" deleted

Migrating from TPRs to CRDs

Upgrading to Kubernetes version 1.7 will not alter a cluster’s existing TPRs, and both mechanisms are functional in this release. The wire format of an individual resource is identical for TPRs and CRDs, unless the CRD version employs different pluralization or namespacing. Applications can construct CRDs that are identical to their existing TPRs.

The new Custom Resource Definition object teams with a migration “helper” strategy in the 1.7 release to make the process of migrating an application’s TPRs to CRDs fairly straightforward. An existing TPR and its data are untouched by the 1.7 upgrade process. This TPR can be replaced by creating a CRD of the same name and namespace. Deleting the TPR then makes the CRD available instead, at the same API URL.

The helper strategy backs this process by copying TPR data to any new CRD with the same name. The TPR data is retained after this migration, so it is possible to return to the previous TPR-based state by recreating the deleted TPR. A TPR of the same name has precedence over a CRD in the 1.7 release.

$ kubectl create -f group-tpr.yaml thirdpartyresource "user-group.auth.coreos.com" created $ kubectl create -f groups.yaml usergroup "engineering" created usergroup "sales" created $ kubectl create -f group-crd.yaml customresourcedefinition "usergroups.auth.coreos.com" created

When both exist, the TPR is the object of API requests. Deleting the TPR exposes the CRD with the same name, and the data that has been copied there. The usergroups data is available even after deleting the TPR in this example:

$ kubectl delete thirdpartyresource user-group.auth.coreos.com thirdpartyresource "user-group.auth.coreos.com" deleted $ kubectl get thirdpartyresources No resources found. $ kubectl get usergroups NAME KIND engineering UserGroup.v1.auth.coreos.com sales UserGroup.v1.auth.coreos.com

It's important to note that this built-in migration is best effort. If the migration fails internally, the CRD may be missing some data. User should ensure no clients are attempting to write to the custom resource while the migration is being performed, and backup their data before deleting the TPR resource. If things go wrong, apply the resource data from the backup.

More migration details will be available in the upstream TPR docs once 1.7 is released.

Continually refining Kubernetes

Custom Resource Definitions are part of a wider effort to refine and enhance Kubernetes as an extensible application platform, factoring everything but the bare essentials out of “core” Kubernetes in favor of modular and maintainable extensibility mechanisms. Join CoreOS in developing “self-driving” modern infrastructure, either upstream in Kubernetes, or in the open source Tectonic Installer, where we develop Tectonic components and the Terraform-based deployment tools for popular cloud providers and bare metal on-premises clusters.

If you’re just getting started with modern infrastructure like Kubernetes and Tectonic, the free Tectonic license for small clusters is an easy way to bring up a Kubernetes cluster and start learning how the system automates application deployment, lifecycle, and scaling. You can stay up to date with new developments in the distributed systems community by subscribing to the CoreOS newsletter.