Getting Started

First of all let’s download the code for the tutorial:

➜ git clone https://github.com/harbur/kubernetes-project-initializer-tutorial.git

➜ cd kubernetes-project-initializer-tutorial

Create Kubernetes Cluster

We’ll need a Kubernetes Cluster to deploy to. The Cluster needs to support Initializers which is still an alpha feature.

If you want to use a pre-existing cluster make sure you have Initializers activated:

➜ kubectl get initializerconfiguration

No resources found.

If you want to use minikube to launch locally Kubernetes use the following command:

➜ minikube start --extra-config=apiserver.runtime-config=admissionregistration.k8s.io/v1alpha1

If you want to launch a Kubernetes cluster in Google Kubernetes Engine make sure you have alpha features enabled:

➜ gcloud container clusters create k1 \

--enable-kubernetes-alpha \

--no-enable-autorepair \

--no-enable-autoupgrade

Once the cluster is created in GKE, make sure your user has sufficient privileges to manage RBAC inside the Cluster with the following command:

➜ kubectl create clusterrolebinding cluster-admin-binding \

--clusterrole cluster-admin --user $(gcloud config get-value account)

Configure RBAC

A new ClusterRole project-admin is defined which has aggregated privileges of edit role and can also manage projects . Aggregated ClusterRoles are used to define the RBAC.

➜ kubectl create -f rbac/

clusterrole.rbac.authorization.k8s.io/aggregate:projects created

clusterrole.rbac.authorization.k8s.io/project-admin created

Once created you can see the ClusterRoles defined:

➜ kubectl get clusterrole -l app=project-initializer

NAME AGE

aggregate:projects 1m

project-admin 1m

➜ kubectl describe clusterrole project-admin

...

Resources Non-Resource URLs Resource Names Verbs

--------- ----------------- -------------- -----

projects.tutorial.harbur.io [] [] [*]

...

Create Projects Custom Resource

Let’s create the Project Custom Resource Definition.

Project contains spec.owner field that is used to grant access of the namespaces to that user.

➜ kubectl create -f crd/projects-crd.yaml

customresourcedefinition.apiextensions.k8s.io/projects.tutorial.harbur.io created

Once created you can see the created CRD:

➜ kubectl get crd

NAME CREATED AT

projects.tutorial.harbur.io 2018-09-28T10:11:05Z

Deploy The Project Initializer

The Project Initializer is a Kubernetes Initializer that automates the creation of the Project namespaces and grants privileges to the appropriate Users.

To build the Project Initializer controller:

./project-initializer/build-container

This will build and push the container image to Docker Hub.

The Image is already built and pushed so you can skip this step.

Deploy the project-initializer controller:

➜ kubectl apply -f project-initializer-k8s/

clusterrolebinding.rbac.authorization.k8s.io/add-on-cluster-admin created

serviceaccount/project-initializer created

deployment.apps/project-initializer created

A ServiceAccount project-initializer was created on kube-system namespace and was given cluster-admin privileges:

➜ kubectl get clusterrolebinding -l app=project-initializer

NAME AGE

add-on-cluster-admin 1m

➜ kubectl describe clusterrolebinding -l app=project-initializer

Name: add-on-cluster-admin

Labels: app=project-initializer

Annotations: ...

Role:

Kind: ClusterRole

Name: cluster-admin

Subjects:

Kind Name Namespace

---- ---- ---------

ServiceAccount project-initializer kube-system

Also a deployment project-initializer was created on kube-system namespace:

➜ kubectl logs -n kube-system -l app=project-initializer

2018/09/28 14:23:09 using in-cluster configuration

2018/09/28 14:23:09 Starting the Kubernetes project initializer...

2018/09/28 14:23:09 Initializer name set to: project.initializer.kubernetes.io

Initializing Projects

In this section you will create an InitializerConfiguration that initializes Projects:

➜ kubectl create -f initializer-configurations/project.yaml

initializerconfiguration.admissionregistration.k8s.io/project created

At this point the Project Initializer is ready to initialize new Projects.

We created a project InitializerConfiguration:

➜ kubectl get initializerconfiguration

NAME CREATED AT

project 2018-09-28T14:25:37Z

Demo

Now that we have the Project Initializer ready let’s see how it works.

Create Team Namespaces

We have two teams: blueteam and greenteam . For each team we'll use a separate namespaces to manage their projects:

➜ kubectl create ns blueteam

namespace/blueteam created ➜ kubectl create ns greenteam

namespace/greenteam created

bob User is part of team blueteam

User is part of team alice User is part of team greenteam

Grant project-admin RoleBindings

Let’s grant project-admin privileges to bob and alice on the namespace of their team.

➜ kubectl create rolebinding bob-project-admin \

--clusterrole=project-admin \

--user bob \

-n blueteam

rolebinding.rbac.authorization.k8s.io/bob-project-admin created ➜ kubectl create rolebinding alice-project-admin \

--clusterrole=project-admin \

--user alice \

-n greenteam

rolebinding.rbac.authorization.k8s.io/alice-project-admin created

Now bob User can manage Projects inside blueteam Namespace:

➜ kubectl get projects -n blueteam --as bob

No resources found.

While alice User cannot see Projects in the blueteam Namespace ( alice belongs to greenteam )

➜ kubectl get projects -n blueteam --as alice

Error from server (Forbidden): projects.tutorial.harbur.io is forbidden: User "alice" cannot list projects.tutorial.harbur.io in the namespace "blueteam": Unknown user "alice"

Bob creates Project for Blueteam

Bob can now create a project inside blueteam

➜ kubectl create -f crd/blueteam-b1-project.yaml --as bob

project.tutorial.harbur.io/b1 created

If we check now the logs of the project-initializer operator we can see that it has created the Namespaces and the RoleBindings, and we can also check that Bob can operate inside the new Namespaces:

➜ kubectl logs -n kube-system -l app=project-initializer

2018/10/05 13:20:42 using in-cluster configuration

2018/10/05 13:20:42 Starting the Kubernetes project initializer...

2018/10/05 13:20:42 Initializer name set to: project.initializer.kubernetes.io

2018/10/05 13:31:18 Initializing Project b1 (blueteam)

2018/10/05 13:31:18 - Created Namespace blueteam-b1-dev

2018/10/05 13:31:18 - Created RoleBinding blueteam-admin (blueteam-b1-dev)

2018/10/05 13:31:18 - Created Namespace blueteam-b1-pre

2018/10/05 13:31:18 - Created RoleBinding blueteam-admin (blueteam-b1-pre)

2018/10/05 13:31:18 - Created Namespace blueteam-b1-pro

2018/10/05 13:31:18 - Created RoleBinding blueteam-admin (blueteam-b1-pro)

➜ kubectl get namespaces

NAME STATUS AGE

blueteam Active 13m

blueteam-b1-dev Active 3m

blueteam-b1-pre Active 3m

blueteam-b1-pro Active 3m

...

➜ kubectl get pod -n blueteam-b1-dev --as blueteam

No resources found.

Cleaning Up

To cleanup the demo

kubectl delete ns blueteam-b1-{dev,pre,pro}

kubectl delete project b1 -n blueteam

kubectl delete ns blueteam

kubectl delete -f crd/projects-crd.yaml

kubectl delete -f project-initializer-k8s/

kubectl delete -f initializer-configurations/

kubectl delete -f rbac

If you created a cluster in GKE make sure to destroy it:

➜ gcloud container clusters delete k1

Wrap-up

We’ve used CRD in order to define a new API schema called Project in Kubernetes. We used Kubernetes Operator Pattern to deploy a controller inside Kubernetes that monitors those resources. And finally we used Initializers in order to force initialization of the projects to pass through the controller.

This is just the tip of the iceberg with the possibilities of extending Kubernetes with custom controllers and we didn’t even cover Custom Metrics.