(updated May 20, 2019) Kubebuilder, the Operator SDK and Metacontroller make it easier for third-party developers to build upon the Kubernetes platform, using custom controllers, sometimes called operators, and Custom Resource Definitions (CRDs). They may one day converge toward an official platform SDK, but until then, developers have to choose (or start from scratch). To guide my decision and hopefully yours, I have studied all three tools and tried them on a simple yet useful case: an Ambassador shim that creates dummy Services with annotations (the source of Ambassador's configuration) from custom Mapping objects.

UPDATE (2019-05-20) - Ambassador 0.70 adds support for CRDs, including a Mapping CRD. This blog post is therefore purely educational. If you're here because you want to configure Ambassador using custom resources, just upgrade to 0.70.

UPDATE (2018-10-18) - I've updated this blog post for Kubebuilder v1.0.5 and Operator SDK v0.0.7 and published the full (working and reproducible) experiments on GitHub: ambassador-shim-kubebuilder, ambassador-shim-operator-sdk, and ambassador-shim-metacontroller. I've also added a section about validation. Today, Admiralty is also open-sourcing multicluster-controller. Check out our new blog post.

UPDATE (2018-07-19) - Kubebuilder v1.0.0 was released today. Please refer to the differences with v0 and the migration guide. Overall, it relies on controller-runtime and controller-tools, which have been factored out. The API resembles more that of the Operator SDK, and there is no need to regenerate code after the initial scaffolding, thanks to a new dynamic client.

In short, if Go isn't an option, I strongly recommend Metacontroller, because, without it, you'd have to write your own cache and work queues... or wait until your language's Kubernetes client library catches up with client-go. In Go, I'd still consider Metacontroller for simple and supported use cases. If more flexibility is needed, I'd switch to Kubebuilder, unless I was using the whole Operator Framework (SDK, Lifecycle Manager, and Metering). Kubebuilder is arguably more idiomatic and performant at the moment *, and it is backed by the SIG API Machinery. In any case, remember that Kubernetes controller and operator development tools are still a moving target, APIs may break ( Kubebuilder is a "prototype" [EDIT (2018-10-18): not anymore] and the Operator SDK is in "pre-alpha"); but it may take time before the community settles on a standard solution.

* EDIT (2018-10-18): Kubebuilder's API converged toward the Operator SDK's, and the Operator SDK is currently being refactored to use controller-runtime.

Kubebuilder Operator SDK Metacontroller Backed by SIG API Machinery CoreOS (Red Hat) Google Cloud Platform Architecture Injection [EDIT (2018-10-18): Encapsulation] Encapsulation Framework Pros Tests and docs scaffolding;

Multiple resources and controllers in one project;

Great documentation Simple API;

Part of the Operator Framework Any language;

Higher-level abstractions;

JSON (dynamic for fast development);

Declarative;

Great documentation Cons Go only;

Could use more abstractions Go only;

Potential performance impact;

API mutates parameters; *

Single resource and controller in one project;

Reference example doesn't follow best practices JSON (no static typing by default);

Use case has to be compatible

* EDIT (2018-10-18): I wasn't a fan of the sdk.Get function signature ( func(runtime.Object) error ) until I realized that's just how functions are made "generic" in Go. The function accepts an interface as an argument, which is actually the output of the operation (but not a return value). The caller passes a pointer to a struct that satisfies the interface, and the function mutates it. I'm still not a big fan, because sdk.Get uses the argument as an input too: the name and namespace of the object to get are taken from the struct.

The Kubernetes API provides foundational resources for container orchestration: Deployments, Services, Namespaces, RBAC, etc. Some use cases, however, require additional or higher-level abstractions: you might want to provision and manage external services as if they were Kubernetes objects (e.g. a cloud database), or you might want to consider several Kubernetes objects as one (e.g. MicroService = Deployment + Service). Ideally, you'd like your API to leverage Kubernetes features like kubectl support, CRUD/watch, labels, etcd storage, HTTPS, authentication, authorization, RBAC, auditing... so you can focus on your business logic. However, you'd rather not fork Kubernetes, or wait for your proposal to make it into a future release (your use case may not even be general enough).

Custom Resource Definition vs. Aggregated API Server#

Luckily, the Kubernetes API can be extended at runtime, as long as you follow its conventions. To do so, the current recommendation is to register CustomResourceDefinitions (CRDs) and deploy a controller, sometimes called an operator (see below). When you use CRDs, all of the features mentioned above come out-of-the-box. The other approach is to deploy a custom API server and register it as an aggregated API via an APIService object. It comes with more responsibilities, including the storage of your custom resources. Unless you need a different storage layer than the etcd cluster managed by Kubernetes, or a feature that is not yet supported by CRDs, a custom API server is likely not worth the effort.

Here is a sample CRD (without any advanced features, like validation):

Copy apiVersion : apiextensions.k8s.io/v1beta1 kind : CustomResourceDefinition metadata : name : mappings.ambassadorshim.admiralty.io spec : group : ambassadorshim.admiralty.io version : v1alpha1 names : kind : Mapping plural : mappings singular : mapping

The corresponding custom controller typically runs in-cluster, managed by a Deployment.

On the other hand, here is a sample APIService:

Copy apiVersion : apiregistration.k8s.io/v1beta1 kind : APIService metadata : name : v1alpha1.ambassadorshim.admiralty.io spec : version : v1alpha1 group : ambassadorshim.admiralty.io service : name : apiserver namespace : ambassadorshim

The corresponding custom API server typically consists of a Deployment exposed by a Service; here, named apiserver , in the ambassadorshim namespace. Any HTTP request for /apis/ambassadorshim.admiralty.io/v1alpha1/... received by kube-apiserver (the main Kubernetes API server) is forwarded to apiserver.ambassadorshim , which is the only one to know about the custom resources (e.g., Mapping) under the ambassadorshim.admiralty.io/v1alpha1 API-version.

Under the Hood#

apiextensions-apiserver and kube-aggregator handle the apiextensions.k8s.io/v1beta1 and apiregistration.k8s.io/v1beta1 API-versions, respectively; both are implementations of apiserver and are included as delegates in kube-apiserver. To check which API-versions your cluster can handle, you can run:

Copy kubectl api-versions

When you register CRDs or aggregated APIs, a new API-version is added to the list.

Aggregated API servers can perform arbitrary validation checks, because they're in charge of storage. However, when CRDs were first introduced, there was no mechanism for synchronous validation. An invalid object would be saved as it came, and its controller would have to update its status to flag it as invalid. But things have improved, and you can now use the OpenAPI v3 schema to validate objects, and use validating webhooks for corner cases.

The Controller Pattern#

Whether you use CRDs or an aggregated API, the best way to implement the behavior of your custom resources is using the controller pattern, the same way Kubernetes controls its own resources. It is often described by three adjectives: declarative, asynchronous and level-based; most of its mechanics are implemented in Go in client-go/tools/cache and client-go/util/workqueue and documented in several blogs.

The user declares the desired state of an object, e.g. kubectl apply -f mapping.yaml , where mapping.yaml contains the specification (Spec) of a Mapping object. The Kubernetes API responds as soon as the desired state has been stored in etcd, but before the user's intent has been fulfilled--which is to create a dummy Service object, owned by the Mapping object, annotated according to its Spec. Asynchronously, the controller watches for CRUD events on the Mapping and Service resources. For each event, the state of the corresponding object is cached, and its key goes to a work queue; the key is the namespace and name of the Mapping object, obtained directly, or via owner references for Service events. This logic is the work of listers and informers. The work queue is backed by a queue and a set, so that if a key is added multiple times before it is processed, it is only processed once. The processing function gets the latest state of the Mapping and Service objects from cache, updates the Mapping's Status (e.g. Service not yet created, or outdated annotation) and takes action to reconcile it according to the Spec (e.g. create the dummy Service, or update its annotation; other use cases could place calls to external services here). Thus, the control loop is level-based, because it reconciles the desired and observed states based on their latest observations, not on their historical changes.

Development Tools#

You can build a controller from the building blocks in client-go. Months ago, you would have started from the workqueue example or from sample-controller, which provides a project structure where you can:

copy/edit structs to define your custom resources; run a helper script to generate typed clients, informers, listers and deep-copy functions, thanks to the code-generator binary; adapt the syncHandler to implement your business logic.

An aggregated API server has more responsibilities, including storage, out of the scope of this blog post. The apiserver library helps you implement those. A project structure is provided in sample-apiserver, but instead you would most likely use apiserver-builder, which is what I would call an early second-generation tool, because it's a CLI that:

initializes projects (no need to fork or copy samples); generates code, including test scaffolding (no need to run a separate code-generation binary, either directly or via Bash scripts); runs the API server and associated controller manager either locally or in-cluster; and even generates documentation.

Example: the federation-v2 project uses apiserver-builder [EDIT (2018-10-18): it used to, but switched to Kubebuilder, see below].

Recently, several projects have positioned themselves as SDKs or frameworks for CRDs and controllers, making the creation process a lot easier. The main ones are Kubebuilder, the Operator SDK, and Metacontroller. It is now possible to become familiar with one of those tools, its underlying concepts, and put together a working CRD and controller in under half a day.

I have tested the three tools on a simple yet useful case. Ambassador, the "Kubernetes-native API gateway for microservices built on Envoy", currently pulls its configuration (mainly Mappings of URL prefixes to Kubernetes Services) from annotations on Services; I wanted Ambassador to be even more Kubernetes native, so I've created a Mapping CRD and a controller that maintains a dummy Service for each Mapping, annotated according to the Mapping's Spec. [EDIT (2019-05-20): Ambassador 0.70 introduces an official Mapping CRD, so this experiment isn't "useful" anymore, but still educational.]

Here is a sample Mapping object configuring Ambassador to proxy requests to /foo/ to the foo service in the bar namespace:

Copy apiVersion : ambassadorshim.admiralty.io/v1alpha1 kind : Mapping metadata : name : foo spec : prefix : /foo/ service : foo.bar

And here is the corresponding annotated dummy Service required to configure Ambassador (without our shim, Ambassador users would usually place the annotation on the foo service directly, but Ambassador annotations can be placed on any Service):

Copy apiVersion : v1 kind : Service metadata : name : foo - ambassadorshim annotations : getambassador.io/config : | --- apiVersion: ambassador/v0 kind: Mapping name: foo prefix: /foo/ service: foo.bar spec : ports : - name : dummy port : 80

Our shim creates the foo-ambassadorshim Service and keeps it in sync with the foo Mapping.

EDIT 2018-10-18: This section has been updated for Kubebuilder v1.0.5. The full source code of this experiment is available on GitHub as ambassador-shim-kubebuilder.

Following Kubebuilder's Quick Start Guide, I installed the kubebuilder binary. Then, in my project folder, I ran:

Copy kubebuilder init --domain admiralty.io --owner "Admiralty Technologies Inc" kubebuilder create api \ --group ambassadorshim \ --version v1alpha1 \ --kind Mapping

The first command generated a basic project structure and the second generated code for an empty resource and its controller. With Kubebuilder, you can technically create any number of resources and controllers, and not all resources have to have controllers (e.g. if a resource owns another). The next step was to edit the resource's API, by modifying the MappingSpec and MappingStatus structs in pkg/apis/ambassadorshim/v1alpha1/mapping_types.go :

Copy type MappingSpec struct { Prefix string `json:"prefix"` Service string `json:"service"` } type MappingStatus struct { Configured bool `json:"configured"` UpToDate bool `json:"upToDate"` }

Note the use of the "json" key in the field tags to customize (un)marshalling. A full Ambassador shim would include more fields in its MappingSpec . Also, a common pattern that I could have used in MappingStatus is a Conditions array, as in many Kubernetes resource APIs.

Then, I implemented the resource's behavior in pkg/controller/mapping/controller.go :

The controller needs to watch the Services owned by Mappings, to update a Mapping's status when its corresponding dummy Service is created (asynchronously) and to recreate/update the Service if it is deleted/modified out of the control loop. To do so, I modified the second Watch() call in the add() function, to watch corev1.Service{} , rather than the generated example's appsv1.Deployment{} (cf. "Watching Created Resources" in Kubebuilder's documentation).

call in the function, to watch , rather than the generated example's (cf. "Watching Created Resources" in Kubebuilder's documentation). Also, I modified the +kubebuilder:rbac comment-annotation to the Reconcile() method to instruct Kubebuilder to create the RBAC rules needed for running in-cluster.

comment-annotation to the method to instruct Kubebuilder to create the RBAC rules needed for running in-cluster. Finally, I modified the Reconcile() method's body for our use case. Compared to the generated example, I added Mapping status updates. Otherwise, the pattern is very similar.

Copy func add ( mgr manager . Manager , r reconcile . Reconciler ) error { err = c . Watch ( & source . Kind { Type : & corev1 . Service { } } , & handler . EnqueueRequestForOwner { IsController : true , OwnerType : & ambassadorshimv1alpha1 . Mapping { } , } ) if err != nil { return err } return nil } func ( r * ReconcileMapping ) Reconcile ( request reconcile . Request ) ( reconcile . Result , error ) { m := & ambassadorshimv1alpha1 . Mapping { } if err := r . Get ( context . TODO ( ) , request . NamespacedName , m ) ; err != nil { if errors . IsNotFound ( err ) { return reconcile . Result { } , nil } return reconcile . Result { } , err } ds , err := dummyService ( m ) if err != nil { return reconcile . Result { } , err } if err := controllerutil . SetControllerReference ( m , ds , r . scheme ) ; err != nil { return reconcile . Result { } , err } os := & corev1 . Service { } if err := r . Get ( context . TODO ( ) , types . NamespacedName { Name : ds . Name , Namespace : ds . Namespace } , os ) ; err != nil { if errors . IsNotFound ( err ) { m . Status = ambassadorshimv1alpha1 . MappingStatus { Configured : false , UpToDate : false , } if err := r . Status ( ) . Update ( context . TODO ( ) , m ) ; err != nil { return reconcile . Result { } , err } err := r . Create ( context . TODO ( ) , ds ) return reconcile . Result { } , err } return reconcile . Result { } , err } if reflect . DeepEqual ( ds . Annotations , os . Annotations ) { m . Status = ambassadorshimv1alpha1 . MappingStatus { Configured : true , UpToDate : true , } err := r . Status ( ) . Update ( context . TODO ( ) , m ) return reconcile . Result { } , err } m . Status = ambassadorshimv1alpha1 . MappingStatus { Configured : true , UpToDate : false , } if err := r . Status ( ) . Update ( context . TODO ( ) , m ) ; err != nil { return reconcile . Result { } , err } os . Annotations = ds . Annotations err = r . Update ( context . TODO ( ) , os ) return reconcile . Result { } , err } type LegacyMapping struct { APIVersion string `yaml:"apiVersion"` Kind string `yaml:"kind"` Name string `yaml:"name"` Prefix string `yaml:"prefix"` Service string `yaml:"service"` } func dummyService ( m * ambassadorshimv1alpha1 . Mapping ) ( * corev1 . Service , error ) { lm := LegacyMapping { APIVersion : "ambassador/v0" , Kind : "Mapping" , Name : m . Name , Prefix : m . Spec . Prefix , Service : m . Spec . Service , } y , err := yaml . Marshal ( & lm ) if err != nil { return nil , err } s := & corev1 . Service { ObjectMeta : metav1 . ObjectMeta { Name : m . Name + "-ambassadorshim" , Namespace : m . Namespace , Annotations : map [ string ] string { "getambassador.io/config" : string ( y ) , } , } , Spec : corev1 . ServiceSpec { Ports : [ ] corev1 . ServicePort { corev1 . ServicePort { Port : 80 } , } , } , } return s , nil }

Then, I ran:

Copy make

Everytime the MappingSpec and MappingStatus structs change, or +kubebuilder comment-annotations are modified, CRD (the validation part) and RBAC manifests must be regenerated.

See the Quick Start Guide for local and in-cluster deployments, testing and documentation generation instructions.

Here are my comments on this experiment:

There were a few mismatches between the documentation, samples, and the generated code.

In my opinion, there are still a few too many responsibilities left to the developer: checking whether objects exist when your use case doesn't require any action if they don't (thanks to Kubernetes garbage collection); setting owner references.

I did not use the event recorder, finalizers, nor webhooks.

EDIT 2018-10-18: This section has been updated for Operator-SDK v0.0.7. The full source code of this experiment is available on GitHub as ambassador-shim-operator-sdk.

Following the Operator SDK's User Guide, I installed the operator-sdk binary. Then, in my organisation folder, I ran:

Copy operator-sdk new ambassadorshim-operatorsdk \ --api-version = ambassadorshim.admiralty.io/v1alpha1 \ --kind = Mapping cd ambassadorshim-operatorsdk

In one command, I initialized the project and generated code for a resource and its controller, called an "operator" in the context of the Operator SDK. The first difference with Kubebuilder is that one Operator-SDK project only deals with one resource/operator pair (which works well when managing applications).

The next step was to modify the MappingSpec and MappingStatus structs in pkg/apis/ambassadorshim/v1alpha1/types.go , just like with Kubebuilder (see listing above), because both tools use code-generator under the hood. Don't forget to run:

Copy operator-sdk generate k8s

Finally, I implemented the Handle() function in pkg/stub/handler.go , with a logic similar to the one implemented in the Kubebuilder experiment above, but adapted to the Operator SDK's API:

No need to Get() the Mapping object, as it is included in the event argument. It just needs to be type-casted.

the Mapping object, as it is included in the argument. It just needs to be type-casted. There's no helper function to set owner references, so I made my own.

TypeMeta MUST be set in the desired Service object. (Kubebuilder auto-filled it based on the struct type.)

Copy func ( h * Handler ) Handle ( ctx types . Context , event types . Event ) error { switch o := event . Object . ( type ) { case * ambassadorshimv1alpha1 . Mapping : m := o if event . Deleted { return nil } ds , err := dummyService ( m ) if err != nil { return err } addOwnerRefToObject ( ds , asOwner ( m ) ) os := ds . DeepCopy ( ) if err := query . Get ( os ) ; err != nil { if errors . IsNotFound ( err ) { m . Status = ambassadorshimv1alpha1 . MappingStatus { Configured : false , UpToDate : false , } if err := action . Update ( m ) ; err != nil { return err } err := action . Create ( ds ) return err } return err } if reflect . DeepEqual ( ds . Annotations , os . Annotations ) { m . Status = ambassadorshimv1alpha1 . MappingStatus { Configured : true , UpToDate : true , } err := action . Update ( m ) return err } m . Status = ambassadorshimv1alpha1 . MappingStatus { Configured : true , UpToDate : false , } if err := action . Update ( m ) ; err != nil { return err } os . Annotations = ds . Annotations err = action . Update ( os ) return err } return nil } type LegacyMapping struct { } func dummyService ( m * ambassadorshimv1alpha1 . Mapping ) ( * corev1 . Service , error ) { s := & corev1 . Service { TypeMeta : metav1 . TypeMeta { APIVersion : "v1" , Kind : "Service" , } , } return s , nil } func addOwnerRefToObject ( obj metav1 . Object , ownerRef metav1 . OwnerReference ) { obj . SetOwnerReferences ( append ( obj . GetOwnerReferences ( ) , ownerRef ) ) } func asOwner ( m * ambassadorshimv1alpha1 . Mapping ) metav1 . OwnerReference { trueVar := true return metav1 . OwnerReference { APIVersion : m . APIVersion , Kind : m . Kind , Name : m . Name , UID : m . UID , Controller : & trueVar , } }

See the User Guide for local and in-cluster deployment instructions.

Here are my comments on this experiment:

The logic of the reference memcached handler in the User Guide is debatable, e.g. it first tries to Create() a child Deployment, even if it already exists, then tries to Get() it even if it doesn't exist ( Create() is asynchronous): that's one too many call either way. I'm hopeful that post-alpha releases will follow best practices; in the meantime, the User Guide does note: "The provided handler implementation is only meant to demonstrate the use of the SDK APIs and is not representative of the best practices of a reconciliation loop."

a child Deployment, even if it already exists, then tries to it even if it doesn't exist ( is asynchronous): that's one too many call either way. I'm hopeful that post-alpha releases will follow best practices; in the meantime, the User Guide does note: The focus on one resource and one control loop can be limiting out of the scope of application provisioning and management.

EDIT 2018-10-18: This section uses Metacontroller v0.2.0. The full source code of this experiment is available on GitHub as ambassador-shim-metacontroller.

Metacontroller isn't a CLI but a framework that runs in-cluster. I installed it following the User Guide. Metacontroller supports two use cases, which are themselves registered as CRDs (hence the prefix "Meta" in "Metacontroller"):

CompositeController is "designed to facilitate custom controllers whose primary purpose is to manage a set of child objects based on the desired state specified in a parent object. Workload controllers like Deployment and StatefulSet are examples of existing controllers that fit this pattern."

DecoratorController is "designed to facilitate adding new behavior to existing resources. You can define rules for which resources to watch, as well as filters on labels and annotations."

The CompositeController fits our need. It's also used in the User Guide's hello-world example. Here is the manifest of a CompositeController watching our custom resource and associated Services:

Copy apiVersion : metacontroller.k8s.io/v1alpha1 kind : CompositeController metadata : name : ambassadorshim - metacontroller spec : generateSelector : true parentResource : apiVersion : ambassadorshim.admiralty.io/v1alpha1 resource : mappings childResources : - apiVersion : v1 resource : services updateStrategy : method : InPlace hooks : sync : webhook : url : http : //ambassador - mapping - controller/sync resyncPeriodSeconds : 10

Metacontroller uses labels to filter child resources on the server side (Kubebuilder and the Operator SDK filter on the client side using owner references). Note that Metacontroller can generate those labels for you, which is convenient.

The CompositeController object defines a webhook that Metacontroller calls to reconcile the parent status and children's desired state from the parent and children's observed state, declaratively. I implemented the webhook as a simple Python 3 HTTPServer (pyyaml is the only requirement, to marshal the MappingSpec into a dummy Service annotation):

Copy from http . server import BaseHTTPRequestHandler , HTTPServer import json import yaml class Controller ( BaseHTTPRequestHandler ) : def sync ( self , parent , children ) : legacy_mapping = { "apiVersion" : "ambassador/v0" , "kind" : "Mapping" , "name" : parent [ "metadata" ] [ "name" ] , } spec = parent . get ( "spec" , { } ) spec . pop ( "selector" , None ) legacy_mapping . update ( spec ) a = yaml . dump ( legacy_mapping , explicit_start = True ) ds_name = parent [ "metadata" ] [ "name" ] + "-ambassadorshim" ds = { "apiVersion" : "v1" , "kind" : "Service" , "metadata" : { "name" : ds_name , "annotations" : { "getambassador.io/config" : a , } , } , "spec" : { "ports" : [ { "port" : 80 , } ] , } , } configured = ds_name in children [ "Service.v1" ] up_to_date = configured and children [ "Service.v1" ] [ ds_name ] \ . get ( "metadata" , { } ) \ . get ( "annotations" , { } ) \ . get ( "getambassador.io/config" , "" ) == a status = { "configured" : configured , "upToDate" : up_to_date , } return { "status" : status , "children" : [ ds ] } def do_POST ( self ) : observed = json . loads ( self . rfile . read ( int ( self . headers . get ( "content-length" ) ) ) ) desired = self . sync ( observed [ "parent" ] , observed [ "children" ] ) self . send_response ( 200 ) self . send_header ( "Content-type" , "application/json" ) self . end_headers ( ) self . wfile . write ( json . dumps ( desired ) . encode ( ) ) HTTPServer ( ( "" , 80 ) , Controller ) . serve_forever ( )

I packaged the webhook as a very simple Docker container managed by a Deployment and exposed by a Service ( ambassadorshim-metacontroller ).

Here are my comments on this experiment:

If Metacontroller supports your use case via either a CompositeController or a DecoratorController, your responsibility is strictly limited to your business logic: no need to check whether the parent exists, or to set owner references, etc. Just beware that you cannot opt out: e.g., children of a CompositeController are always garbage-collected.

Metacontroller's API is declarative, which makes it easy to reason about. To do so, it provides higher-level abstractions like the update strategy (OnDelete, Recreate, InPlace, RollingRecreate, RollingInPlace).

Because webhooks only need to accept and return JSON, it is possible to use dynamically typed languages like Python and JavaScript, which are great for rapid development. An added benefit in our case is that the Ambassador shim translates all of a Mapping's specification to annotations out-of-the-box, not just the required fields I cared to support in the Go implementations (Prefix and Service). However, the absence of static typing by default could be error-prone in more complex situations, or require additional tooling.

Metacontroller is not designed to watch external APIs, whereas Kubebuilder can watch any Go channel.

Kubebuilder and the Operator SDK are quite similar: both are only intended for Go developers; both rely on code-generator to some extent; both provide a CLI to set up a new project, regenerate code, build binaries, images, manifests, etc.; both implement the controller pattern as a library. Their APIs and implementation details, however, differ. Kubebuilder uses code-generator more extensively, to generate typed clients, informers and listers for custom resources, whereas the Operator SDK relies instead on client-go's discovery and REST mapping features. The Operator SDK encapsulates existing abstractions and provides a new, simple API, whereas Kubebuilder conveniently injects existing abstractions, which is fine if you're already familiar with them (from client-go), but can otherwise be daunting. [EDIT (2018-10-18): Kubebuilder v1 uses controller-runtime, which uses client-go's discovery and REST mapping features, and whose API resembles that of the Operator SDK.] Kubebuilder can also help generate tests and documentation, supports multiple resources and controllers per project, and makes full use of its cache to limit kube-apiserver calls. On the other hand, in the context of application operators, the Operator SDK could potentially develop synergies with the rest of the Operator Framework (Operator Lifecycle Manager and Operator Metering).

The third contender, Metacontroller, takes a radically different approach: it's an actual framework (as opposed to generated code), where controllers are custom resources themselves, controlled by the Metacontroller, which delegates reconciliation to webhooks; "all you need to provide is a webhook that understands JSON, you can use any programming language." Using Metacontroller, my Ambassador Mapping controller consists of 50 lines of Python (for the webhook), 30 lines of Yaml (for the Mapping CRD and CompositeController) and that's it; no generated boilerplate whatsoever. Metacontroller isn't as flexible as Kubebuilder or the Operator SDK, but it does support most use cases.

Developing custom resources and controllers has definitely gotten easier over the past year, thanks to the tools discussed above. However, Kubernetes platform development is still a fragmented and rapidly evolving landscape. There was a proposal to create a new SIG to oversee the development of standard tools. It has been rejected for now by the steering committee, but the SIG API Machinery is taking Kubebuilder under its wing and has started refactoring [EDIT (2018-10-18): refactored] it into a controller-runtime and controller-tools.