What we will do

Below is the list of steps we will follow:

a brief introduction to GitOps

setup a simple project and manage it within GitLab

integrate a Kubernetes cluster

setup a typical CI/CD pipeline

handle the CD part with GitOps

A brief introduction to GitOps

GitOps is a way to do Continuous Delivery. It works by using Git as a source of truth for both declarative infrastructure and applications. Automated delivery pipelines roll out changes to your infrastructure when changes are made to Git. — Weaveworks

Deploy changes to the cluster : push vs pull

In a typical CI/CD pipeline, the CI tool is in charge of running the tests, building the image, checking the CVEs and of redeploying the new images into the cluster as we can see in the schema below.

Typical CI/CD pipeline (source: Weaveworks)

The GitOps approach is different as the deployment part is not done by the CI tool but by an operator, a process running in a Pod within the cluster (here comes Flux).

CI/CD pipeline including GitOps (source: Weaveworks)

The components involved

The following schema shows the components involved when using GitOps in the context of a Kubernetes cluster.

GitOps’s components within a Kubernetes cluster (source: Weaveworks)

To keep it simple, a Flux daemon is continuously running and checking for new Docker images. When a new image is detected, it calls the API Server to update the running deployments.

In the last part of this post, we will setup Flux and use it to deploy a simple application.

The project

We will consider a simple (very simple) Flask application. The complexity of the project is not important here, the most important being the understanding of the whole CI/CD flow.

Source code

Let’s just consider the following files:

app.py exposes a single HTTP endpoint and returns a string

from flask import Flask

app = Flask(__name__)



@app.route("/")

def hello():

return "Hello World!"



if __name__ == "__main__":

app.run(host='0.0.0.0', port=8000)

requirements.txt defines the dependency, Flask library, needed by app.py

Flask==1.0.2

Dockerfile use to build an image out of the source code

FROM python:3-alpine

COPY . /app

WORKDIR /app

RUN pip install -r requirements.txt

CMD python /app/app.py

Making sure it works

We start by creating a first Docker image for our application

$ docker image build -t hello:1.0 .

Once the image is built, we can run a container using it.



* Serving Flask app “app” (lazy loading)

* Environment: production

WARNING: Do not use the development server in a production environment.

Use a production WSGI server instead.

* Debug mode: off

* Running on $ docker container run -p 8000:8000 hello:1.0* Serving Flask app “app” (lazy loading)* Environment: productionWARNING: Do not use the development server in a production environment.Use a production WSGI server instead.* Debug mode: off* Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)

Our server is listening on port 8000 as we can see below.

GitLab project

We will use GitLab to manage this application, so let’s create a new project named hello :

Creation of a new project in GitLab

We can then initialize git for the application folder and push everything into the GitLab project:

$ git init

$ git remote add origin git@gitlab.com:lucj/hello.git

$ git add .

$ git commit -m "Initial commit"

$ git push -u origin master

A couple of seconds later we can see the 3 files into the project through the GitLab web interface.

First commit of the code

Enters Kubernetes

As we want to deploy our application on a Kubernetes cluster, we will use the Kubernetes integration functionality of GitLab to import the configuration of an external cluster in the project.

Creation of a managed cluster

DOKS (DigitalOcean managed Kubernetes cluster) is my favorite solution, easy to setup and use. It can be created from DigitalOcean web interface or using the dedicated doctl command line interface. In this example, we will setup a 3 worker nodes cluster, the manager nodes being managed by DigitalOcean for us.

Creation of a managed Kubernetes cluster from DigitalOcean web interface

It takes a couple of minutes for the infrastructure to be provisioned and the cluster created. Once it’s done we need to retrieve the kubeconfig file so our we kubectl client can communicate with the cluster’s API Server. We will use the doctl command line and save this configuration in k8s-demo.cfg file:

$ doctl k8s cluster cfg show k8s-demo > k8s-demo.cfg

We then configure kubectl to communicate with our cluster setting the KUBECONFIG environment variable:

$ export KUBECONFIG=$PWD/k8s-demo.cfg

All set. Let’s check the state of our cluster:

$ kubectl get nodes

NAME STATUS ROLES AGE VERSION

k8s-demo-rlf5 Ready <none> 2m10s v1.15.2

k8s-demo-rlfh Ready <none> 2m40s v1.15.2

k8s-demo-rlfk Ready <none> 2m33s v1.15.2

Integration with the GitLab project

From the GitLab’s web interface, it’s very easy to integrate an external Kubernetes cluster to a project. We just need to enter the Operations > Kubernetes and then click on Add Kubernetes cluster:

Integration of a Kubernetes cluster

We then need to select the Add existing cluster tab. From there we need to fill a couple of fields. The firsts of them can easily be retrieved from the configuration file:

Fields to fill in during the Kubernetes cluster integration

Cluster name

URL of the API Server

Cluster’s CA certificate

To provide the cluster CA certificate to GitLab, we need to decode the one specified in the configuration (as it’s encoded in base64).

$ kubectl config view --raw \

-o=jsonpath='{.clusters[0].cluster.certificate-authority-data}' \

| base64 --decode

Service token

The process to get an identification token involves several steps. We first need to create a ServiceAccount and provide it with the cluster-admin role. This can be done with the following command:

$ cat <<EOF | kubectl apply -f -

apiVersion: v1

kind: ServiceAccount

metadata:

name: gitlab-admin

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

name: gitlab-admin

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: cluster-admin

subjects:

- kind: ServiceAccount

name: gitlab-admin

namespace: kube-system

EOF

Once the ServiceAccount is created, we retrieve the associated Secret:

$ SECRET=$(kubectl -n kube-system get secret | grep gitlab-admin | awk '{print $1}')

add extract its JWT token, the one we need to enter in the Service Token field in the GitLab interface:

$ TOKEN=$(kubectl -n kube-system get secret $SECRET -o jsonpath='{.data.token}' | base64 --decode) && echo $TOKEN

Before validating the cluster integration, we uncheck the GitLab-managed-cluster checkbox as we will manage our own namespaces.

Once the cluster is integrated GitLab allows to install several applications in one click through Helm charts. We will not use it in this post though.

Kubernetes cluster integrated with our GitLab’s project

Setting up a typical CI/CD Pipeline

We start by adding a .gitlab-ci.yml file at the root of our project. This one is used to define the actions triggered each time a new code commit is pushed to the repository.

At the top of this file, we define the different stages of the pipeline:

stages:

- package

- test

- push

- deploy

Then for each stage, we define the actions to perform:

the package stage creates a Docker image from the source code and pushes it to the GitLab image repository of the project with a temporary tag (more on that later)

build:

image: docker:stable

stage: package

services:

- docker:dind

script:

- docker build -t $CI_REGISTRY_IMAGE:tmp .

- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY

- docker push $CI_REGISTRY_IMAGE:tmp

only:

- master

the test stage runs a container from the newly created image and makes sure the message returned starts with “Hello”

test:

image: docker:stable

stage: test

services:

- docker:dind

script:

- docker run -d --name hello $CI_REGISTRY_IMAGE:tmp

- sleep 10s

- TEST=$(docker run --link hello lucj/curl -s http://hello:8000)

- $([ "${TEST:0:5}" = "Hello" ])

only:

- master

the push stage add new tags to the image, the first one based on the hash of the git commit, the second one is the name of the current branch (master in this case as we only trigger those actions on the master branch). It then pushes those new tags to the GitLab registry.

push:

image: docker:stable

stage: push

services:

- docker:dind

script:

- docker image pull $CI_REGISTRY_IMAGE:tmp

- docker image tag $CI_REGISTRY_IMAGE:tmp $CI_REGISTRY_IMAGE:$CI_BUILD_REF

- docker image tag $CI_REGISTRY_IMAGE:tmp $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY

- docker push $CI_REGISTRY_IMAGE:$CI_BUILD_REF

- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

only:

- master

the purpose of the deploy stage is to create / update the application within our Kubernetes cluster. We will define 2 manifest files in a k8s folder: a Deployment to manage the Pod of our web server, and a Service to expose it to the outside

We start by defining the following k8s/deploy.tpl template. It will be used during the deploy step to generate the k8s/deploy.yml file specifying the Deployment resource. This template defines a Deployment managing a single replica of a Pod based on the registry.gitlab.com/lucj/hello image.

In this template, we use a placeholder named GIT_COMMIT replaced with the actual hash of the commit as we will see in a bit.

apiVersion: apps/v1

kind: Deployment

metadata:

name: hello

labels:

app: hello

spec:

selector:

matchLabels:

app: hello

template:

metadata:

labels:

app: hello

spec:

containers:

- name: hello

image: registry.gitlab.com/lucj/hello:GIT_COMMIT

We also define a Service resource in k8s/service.yml, to expose the application. This Service is of type LoadBalancer.

apiVersion: v1

kind: Service

metadata:

name: hello

spec:

type: LoadBalancer

ports:

- name: hello

port: 80

targetPort: 8000

protocol: TCP

selector:

app: hello

To actions performed in the deploy step are the following ones:

deploy:

stage: deploy

image: lucj/kubectl:1.15.2

environment: test

script:

- kubectl config set-cluster my-cluster --server=${KUBE_URL} --certificate-authority="${KUBE_CA_PEM_FILE}"

- kubectl config set-credentials admin --token=${KUBE_TOKEN}

- kubectl config set-context my-context --cluster=my-cluster --user=admin --namespace default

- kubectl config use-context my-context

- cat k8s/deploy.tpl | sed 's/GIT_COMMIT/'"$CI_BUILD_REF/" > k8s/deploy.yml

- kubectl apply -f k8s

only:

- master

Several things to note here:

this step is run in the context of an image containing the kubectl client

the cluster information are retrieved from the environment variables automatically set by GitLab. They are used to set a Kubernetes context

a Deployment resource is created out of the template file, the GIT_COMMIT placeholder is replaced with the actual commit available within the $CI_BUILD_REF environment variable

the Service and Deployment resource, respectively in k8s/service.yml and k8s/deploy.yml are created / updated with the usual “kubectl apply” command

Note: this pipeline is quite simple and not optimal but it will be just fine to illustrate the different flows

Testing things out

Let’s push those changes to our GitLab’s project and check the CI/CD pipeline triggered

$ git add k8s

$ git commit -m ‘Add K8s resources’ $ git add .gitlab-ci.yml

$ git commit -m ‘Add GitLab pipeline’ $ git push origin master

This triggers the GitLab pipeline as we can see on the web interface

In the deploy stage of the pipeline (the last one), both Deployment and Service are created as they didn’t exist before. As the Service is of type LoadBalancer, a load balancer resource is created on DigitalOcean infrastructure as we can see below.

Using the external IP address associated to the Load Balancer, we can access the application on port 80 targeting the underlying Pod running our application.

This show both Service and Deployment were correctly created.

Let’s change app.py a little bit so it returns “Hello from Kube” instead of “Hello World!”.

from flask import Flask

app = Flask(__name__)

def hello():

return "Hello from Kube" @app .route("/")def hello():return "Hello from Kube" if __name__ == "__main__":

app.run(host='0.0.0.0', port=8000)

We commit and push the changes

$ git add app.py

$ git commit -m 'change message to Hello from Kube'

$ git push origin master

A new CI/CD pipeline is triggered and we can see the new message if we refresh the browser

We have setup a simple pipeline which, of course, would need some enhancements if used for a real world application. For instance, we could add some more steps like:

additional tests

image scanning, to make sure the image does not contain any CVEs (or at least no critical ones)

If you want to know more about the image scanning part, you may find this article useful https://medium.com/better-programming/adding-cve-scanning-to-a-ci-cd-pipeline-d0f5695a555a

Adding GitOps into the picture

We will now modify our CI/CD pipeline a bit so the CD part is handled with the GitOps approach. The following schema shows the components involved in a GitOps Deployment workflow.

GitOps Deployment Workflow (source: Weaveworks)

Basically, a Flux operator, running in the cluster within a Pod, is in charge of redeploying the application when a new image tag is detected within the image registry.

Flux installation

Flux can be installed either manually from a Deployment or with Helm. In this article we will use the manual approach. The first step is to clone the fluxcd repository:

$ git clone https://github.com/fluxcd/flux && cd flux

Then, within the specification of the Deployment (deploy/flux-deployment.yaml), we will change the following parameters:

--git-url=git@gitlab.com:lucj/hello, this tells Flux which Git repository to watch

--git-path=k8s, within the repository, only the k8s folder will be taken into account (our Kubernetes manifests files are located within this folder)

--git-ci-skip, this option allows to skip the CI pipeline when update to the GitLab’s project repository are done by Flux (tag and Deployment resource)

We can now deploy Flux to the cluster:

$ kubectl apply -f deploy

serviceaccount/flux created

clusterrole.rbac.authorization.k8s.io/flux created

clusterrolebinding.rbac.authorization.k8s.io/flux created

deployment.apps/flux created

secret/flux-git-deploy created

deployment.apps/memcached created

service/memcached created

Several resources were created:

a ServiceAccount, ClusterRole and ClusterRoleBoinding used to provide the Flux Pod the authentication / authorization it requires to operate

the Flux operator

a Service and Deployment for memcached that is used by Flux to cache image metadata

$ kubectl get pods

NAMESPACE NAME READY STATUS RESTARTS AGE

default flux-dcb965db7-pn97k 1/1 Running 0 56s

default memcached-554f994578-t2tss 1/1 Running 0 56s

...

If we look at the logs of the Flux pod, we will see an error message as Flux cannot read the Git repository of the project.

Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

To fix this problem, we can use the fluxctl utility to retrieve the public ssh key generated during the installation.

$ fluxctl identity

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCx4fk4YjcM7cP1FL/AKWtHpN+cg9/Qz1p5dzAlsFLMKilUUy0uCQQmaptXDZQGaZrbvNSyezgT5/yH6qau6W6ICoLYAzBku47PoWlqbUfcbPhMxHSfivjv7s4lSeUE+u3kR2opROxdyHHL+VQMI6n9Xc7qnTq6YC+VJ+RkoUUd0bgBC+Rg/aMURLD9mkAVzmWw6+Y8QAJMVNMzNDgId+8iSHKtOYsHqoxg4GqexdB1R5goE0ChBU9DPsiqLfk8jzuD2I3xuZeGW6or+/JHxa/6vO8lX+of1ZGZGZKr5i3E4OIehSwFUP2A/ypeqXEEI5gmO1s2YrM49jpS+jW4oUMP

and then add this key to our GitLab repository giving it a R/W access as it needs to create/update tags.

Modification of the previous pipeline

As Flux is in charge of deploying the changes done to the project (we will test that in the bit), we need to remove the deploy stage of the .gitlab-ci.yml file we created earlier, everything else can remain the same. .gitlab-ci.yml now looks like the following, no more kubectl to interact with the cluster’s API Server.

stages:

- package

- test

- push build:

image: docker:stable

stage: package

services:

- docker:dind

script:

- docker build -t $CI_REGISTRY_IMAGE:tmp .

- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY

- docker push $CI_REGISTRY_IMAGE:tmp

only:

- master

image: docker:stable

stage: test

services:

- docker:dind

script:

- docker run -d --name hello $CI_REGISTRY_IMAGE:tmp

- sleep 10s

- TEST=$(docker run --link hello lucj/curl -s

- $([ "${TEST:0:5}" = "Hello" ])

only:

- master test:image: docker:stablestage: testservices:- docker:dindscript:- docker run -d --name hello $CI_REGISTRY_IMAGE:tmp- sleep 10s- TEST=$(docker run --link hello lucj/curl -s http://hello:8000 - $([ "${TEST:0:5}" = "Hello" ])only:- master push:

image: docker:stable

stage: push

services:

- docker:dind

script:

- docker image pull $CI_REGISTRY_IMAGE:tmp

- docker image tag $CI_REGISTRY_IMAGE:tmp $CI_REGISTRY_IMAGE:$CI_BUILD_REF

- docker image tag $CI_REGISTRY_IMAGE:tmp $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY

- docker push $CI_REGISTRY_IMAGE:$CI_BUILD_REF

- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

only:

- master

Also, we can remove the template file k8s/deploy.tpl as this one will not be used anymore to update the Deployment’s manifest. Instead, we will use the following Deployment, within k8s/deploy.yml, that Flux will update each time it detect a new image tag.

apiVersion: apps/v1

kind: Deployment

metadata:

name: hello

annotations:

flux.weave.works/automated: "true"

flux.weave.works/tag.hello: regexp:^((?!tmp).)*$

labels:

app: hello

spec:

selector:

matchLabels:

app: hello

template:

metadata:

labels:

app: hello

spec:

containers:

- name: hello

image: registry.gitlab.com/lucj/hello:master

The Flux configuration for this Deployment is done within the annotations key:

flux.weave.works/automated: “true”, activates the automated redeployment for this resource

flux.weave.works/tag.hello: regexp:^((?!tmp).)*$, makes sure the temporary image, the one with the tmp tag is not taken into account

Testing things out

Let’s change the code within app.py so it now returns “Hello from Flux”.

from flask import Flask

app = Flask(__name__)

def hello():

return "Hello from Flux" @app .route("/")def hello():return "Hello from Flux" if __name__ == "__main__":

app.run(host='0.0.0.0', port=8000)

And push this modification to GitLab.

$ git rm k8s/deploy.tpl

$ git add k8s/deploy.yml .gitlab-ci.yml app.py

$ git commit -m 'CD with Flux'

$ git push origin master

Checking the GitLab interface, we could see the pipeline has been triggered several times.

Several pipelines created (some of them were skipped)

One pipeline was triggered by the changes we made, the other ones were triggered by Flux when it updated the Deployment manifest (k8s/deploy.yml) on the master branch, and the tag on the flux-sync branch. The pipelines triggered by those 2 actions are skipped (the associated actions were not performed) as we used the --git-ci-skip option in the Flux configuration (if we haven’t done so, the pipelines would run in a loop).

The new version of the application is then available as we can see by refreshing the browser once again.

Behind the hood, as the Flux operator regularly checks for new image tags, it detected the one created during the CI pipeline resulting of the code change. It then automatically updated the Deployment.