Sharing knowledge on building my own kubernetes CRDs. I will explain steps I use to develop CRDs using my Githook example. I explain the reason why I create this CRDs in Build cloud native CI/CD build pipeline from GIT webhook.

Note:

This article is for a beginner who what to learn how to build kubernetes CRD The goal is to understand steps to make a CRD. You might not be able to follow it step-by-step since it has some dependencies such as knative installed

What you will learn:

Kubernetes CRDs concept

How to build kubernetes CRD using kubebuilder and golang

Why do I need a CRD?

CRD stands for Custom Resource Definition. It is a way to create your own kubernetes resource that work the same way as Kubernetes resource such as Pod or Deployment. With custom resource, you can put the knowledge on how to manage your resource into kubernetes cluster and that is so powerful.

For example, Strimzi is a kubernetes CRD to manage and running kafka cluster on kubernetes. It eliminates a lot of problem running your own kafka cluster by putting the knowledge to manage kafka cluster into the CRD controller itself.

CRDs controller reconciliation loop

You need a controller (in a container running on cluster) to manage your CRD resource. The controller will run in loop to check if what described in the CRD (desired) is match the actual state in the real world. For example, you want container A and B to run on the cluster. If either A or B does not run, controller will create it for you.

This is a robust way to manage what you want in a resilient and flexible like distributed environment.

Githook example CRD

This CRD is called “GitHook”. It defines git webhook events and a build pipeline. GitHook controller will subscribe webhook events to git repo and when the events happen, it will run a build pipeline as defined in the CRD.

How “GitHook” CRD works

GitHook CRD controller’s job is:

make sure that webhook is registered to git repo with incorrect information. make sure that there is a service running and wait for the webhook events. We use Knative service to receive that webhook since it is easy to implement and can scale to 0 when it is not in used.

Building CRD using kubebuilder

1. Setup

Install kubebuilder and kustomize

kubebuilder is a tool to help you get started kubernetes CRDs very fast. You just need to define your custom type and controller logic. Read more.

kustomize is a tool kubebuilder use to produce a yaml that your CRD needs in order to work on kubernetes

Please follow installation step here.

Initial project

We will create a go project named “githook-tutorial” with module support so it requires go version 1.11+ and the folder must be outside GOPATH.

mkdir githook-tutorial

cd githook-tutorial go mod init gitlab.com/pongsatt/githook kubebuilder init --domain my.domain

“gitlab.com/pongsatt/githook” is go module name. If you change to something else, you need to replace import path in the sample code.

“my.domain” will be your resource’s API path. You can change it as needed

Create your custom resource

This step is to tell kubebuilder the kind of resource you need. We will create a resource named “GitHook” in group called “tools”.

kubebuilder create api --group tools --version v1alpha1 --kind GitHook

When it is done, you should see.

kubebuilder generated files and folders

Interesting paths:

api/ : contains your resource information in go. you will edit your resource type here

config/ : contains all kubernetes yaml files needed to run your controller

Makefile: contains all scripts needed to build and deploy your CRD

2. Define resource type

Edit “api/v1alpha1/githook_types.go” to include our custom type.

See githook_types.go for complete code.

Key notes:

If we need our type to be validated when creating our resource via yaml, we need to put comment on that type. For example, we need your gitProvider to be gitlab, github or gogs.

// +kubebuilder:validation:Enum=gitlab;github;gogs



// GitProvider providers name of git provider

type GitProvider string

Next step is to generate utility function such as deepcopy by running command:

make generate

3. Implement controller

At this point, we have our custom type ready to be consumed by our controller.

How controller works:

Our controller named “GitHookReconciler” has a method “Reconcile” which will be called every time “GitHook” resource has been created, updated or deleted. It will process each event one by one in order using queue. When an error occurred, we will return error state so current event will be requeue and wait to be processed later.

Our controller will control 2 other resources:

Knative service (to be able to process git events when occurred). Git webhook (to be able to receive git events).

Create/Update Logic:

Get GitHook information by name using provided k8s client.

// Reconcile main reconcile logic func (r *GitHookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) { ...

err := r.Get(context.Background(), req.NamespacedName, sourceOrg)

...

Get Knative service that created by this resource. If not exist, create new. If exist but information update, updates it.

func (r *GitHookReconciler) reconcile(source *v1alpha1.GitHook) error {

... ksvc, err := r.reconcileWebhookService(source) ...

Register git webhook using Knative service URL given from previous step and keep returned ID in GitHook resource.

func (r *GitHookReconciler) reconcile(source *v1alpha1.GitHook) error { ... hookID, err := r.reconcileWebhook(source, hookOptions) ...

Update current GitHook resource with latest information

// Reconcile main reconcile logic func (r *GitHookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) { ... if err := r.Update(context.Background(), source); err != nil { .... } ...

Delete Logic:

When GitHook resource get deleted, we detect this by checking field “DeletionTimestamp” is not nil. Then, we will delete our dependent resources which, in this case, knative service and git webhook by calling finalize method.

// Reconcile main reconcile logic func (r *GitHookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) { ... if sourceOrg.ObjectMeta.DeletionTimestamp == nil { ... } else { if r.hasFinalizer(source.(*v1alpha1.GitHook).Finalizers) { reconcileErr = r.finalize(source.(*v1alpha1.GitHook)) } } ...

In case that we cannot delete knative service or git webhook, we will not allow our GitHook to be deleted by not deleting finalizer information from GitHook resource.

func (r *GitHookReconciler) finalize(source *v1alpha1.GitHook) error { ... r.removeFinalizer(source)

}

Code:

Edit file “controllers/githook_controller.go”

Please see complete code githook_controller.go.

Key Notes:

Our controller needs permission to get, list, create and update another kubernetes resource. We need to add comment annotations for kubebuilder to generate permission yaml for us.

// +kubebuilder:rbac:groups=tools.pongzt.com,resources=githooks,verbs=get;list;watch;create;update;patch;delete ... // Reconcile main reconcile logic

func (r *GitHookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) { ...

4. Update manager

Since our controller uses other resource apis such as knative, we need to register those types in “main.go”.

func init() {

githookv1alpha1.AddToScheme(scheme)

servingv1alpha1.AddToScheme(scheme)

servingv1beta1.AddToScheme(scheme)

tektonv1alpha1.AddToScheme(scheme)

corev1.AddToScheme(scheme)

// +kubebuilder:scaffold:scheme }

5. Testing controller

You can test or debug your controller on local machine by running command:

make install make run

make install : will generate CRDs yaml files and apply to your kubernetes cluster

make run : will run test and run “main.go”

Your controller will start monitoring your resource and do the reconciliation loop.

6. Deploy CRDs and controller

It is the time you need your CRD to be usable on your cluster. You need to run command below to deploy:

make docker-build IMG=<your image registry>/controller

make docker-push IMG=<your image registry>/controller

make deploy

make docker-build will build docker image as your IMG variable

make docker-push will push docker image to registry

make deploy will produce all the yaml file and deploy to your cluster

Optionally, you can release your yaml file for other to install your CRD easily using command: