Everyone knows how software testing is important and essential — I believe many readers already do that all the time. Surprisingly, it is not that easy to find a good example of setting up two well-known options for related — I mean, CI/CD — issues: our favorite GitLab and JUnit. Let’s fill this gap!

Background

First, I’ll define the full context:

Since all our applications are running in Kubernetes, I’ll cover testing in the related infrastructure only.

We build and deploy images using werf (such an approach also means that Helm is naturally involved in the pipeline).

I won’t go into details of testing itself: in our case testing is implemented on the customer’s side, we only ensure its proper running (and presence of the corresponding report in the merge request).

Here is the general order of actions concerning our example:

Building an application — we will omit the description of this step. Deploying an application to the separate namespace of the Kubernetes cluster and running tests. Retrieving artifacts and parsing JUnit report via GitLab. Deleting a previously created namespace.

Now let’s move on to the implementation!

Getting started

GitLab CI

We’ll start with the part of .gitlab-ci.yaml describing the deployment of the application and running the tests. The code is rather long, therefore I inserted detailed comments into it:

variables:

# declare the version of werf we are going to use

WERF_VERSION: "1.0 beta"

script:

# create the namespace in K8s if it isn’t there

- kubectl --context="${WERF_KUBE_CONTEXT}" get ns ${CI_ENVIRONMENT_SLUG} || kubectl create ns ${CI_ENVIRONMENT_SLUG}

# load werf and deploy — please check docs for details

# (

- type multiwerf && source <(multiwerf use ${WERF_VERSION})

- werf version

- type werf && source <(werf ci-env gitlab --tagging-strategy tag-or-branch --verbose)

- werf deploy --stages-storage :local

--namespace ${CI_ENVIRONMENT_SLUG}

--set "global.commit_ref_slug=${CI_COMMIT_REF_SLUG:-''}"

# pass the `run_tests` variable

# it will be used during rendering of Helm release

--set "global.run_tests=${RUN_TESTS:-no}"

--set "global.env=${CI_ENVIRONMENT_SLUG}"

# set the timeout (some tests are rather long)

# and pass it to the release

--set "global.ci_timeout=${CI_TIMEOUT:-900}"

--timeout ${CI_TIMEOUT:-900}

dependencies:

- Build .base_deploy: &base_deployscript:# create the namespace in K8s if it isn’t there- kubectl --context="${WERF_KUBE_CONTEXT}" get ns ${CI_ENVIRONMENT_SLUG} || kubectl create ns ${CI_ENVIRONMENT_SLUG}# load werf and deploy — please check docs for details# ( https://werf.io/how_to/gitlab_ci_cd_integration.html#deploy-stage - type multiwerf && source

extends: .base_deploy

before_script:

# create the directory for the coming report

# using $CI_COMMIT_REF_SLUG

- mkdir /mnt/tests/${CI_COMMIT_REF_SLUG} || true

# forced workaround 'cause GitLab requires artifacts

# in its build directory

- mkdir ./tests || true

- ln -s /mnt/tests/${CI_COMMIT_REF_SLUG} ./tests/${CI_COMMIT_REF_SLUG}

after_script:

# delete the release with a Job (and possibly its infrastructure)

# after the tests are finished

- type multiwerf && source <(multiwerf use ${WERF_VERSION})

- werf version

- type werf && source <(werf ci-env gitlab --tagging-strategy tag-or-branch --verbose)

- werf dismiss --namespace ${CI_ENVIRONMENT_SLUG} --with-namespace

# we allow failures to happen, but you can decide otherwise

allow_failure: true

variables:

RUN_TESTS: 'yes'

# set the context in werf

# (

WERF_KUBE_CONTEXT: 'admin@stage-cluster'

tags:

# using runner with the `werf-runner` tag

- werf-runner

artifacts:

# you first have to create an artifact to see it in the pipeline

# and download (e.g. for a more thoughtful study)

paths:

- ./tests/${CI_COMMIT_REF_SLUG}/*

# artifacts older than one week will be deleted

expire_in: 7 day

# note: these lines are responsible for parsing the report by GitLab

reports:

junit: ./tests/${CI_COMMIT_REF_SLUG}/report.xml .test-base: &test-baseextends: .base_deploybefore_script:# create the directory for the coming report# using $CI_COMMIT_REF_SLUG- mkdir /mnt/tests/${CI_COMMIT_REF_SLUG} || true# forced workaround 'cause GitLab requires artifacts# in its build directory- mkdir ./tests || true- ln -s /mnt/tests/${CI_COMMIT_REF_SLUG} ./tests/${CI_COMMIT_REF_SLUG}after_script:# delete the release with a Job (and possibly its infrastructure)# after the tests are finished- type multiwerf && source https://werf.io/how_to/gitlab_ci_cd_integration.html#infrastructure WERF_KUBE_CONTEXT: 'admin@stage-cluster'tags:# using runner with the `werf-runner` tag- werf-runnerartifacts:# you first have to create an artifact to see it in the pipeline# and download (e.g. for a more thoughtful study)paths:- ./tests/${CI_COMMIT_REF_SLUG}/*# artifacts older than one week will be deletedexpire_in: 7 day# note: these lines are responsible for parsing the report by GitLabreports:junit: ./tests/${CI_COMMIT_REF_SLUG}/report.xml # to make it simple, only two stages are shown here

# you will have more of them in real life,

# at least because of deploying

stages:

- build

- tests

stage: build

script:

# build stage - check details in the werf docs:

# (

- type multiwerf && source <(multiwerf use ${WERF_VERSION})

- werf version

- type werf && source <(werf ci-env gitlab --tagging-strategy tag-or-branch --verbose)

- werf build-and-publish --stages-storage :local

tags:

- werf-runner

except:

- schedules build:stage: buildscript:# build stage - check details in the werf docs:# ( https://werf.io/how_to/gitlab_ci_cd_integration.html#build-stage - type multiwerf && source

<<: *test-base

environment:

# the point of naming the namespace

# (

name: tests-${CI_COMMIT_REF_SLUG}

stage: tests

except:

- schedules run tests:< https://docs.gitlab.com/ce/ci/variables/predefined_variables.html name: tests-${CI_COMMIT_REF_SLUG}stage: testsexcept:- schedules

Kubernetes

Now it is time to create a YAML file with the description of a Job ( tests-job.yaml ) and all the necessary Kubernetes resources in the .helm/templates directory. See the explanation below:

{{- if eq .Values.global.run_tests "yes" }}

---

apiVersion: v1

kind: ConfigMap

metadata:

name: tests-script

data:

tests.sh: |

echo "======================"

echo "${APP_NAME} TESTS"

echo "======================" cd /app

npm run test:ci

cp report.xml /app/test_results/${CI_COMMIT_REF_SLUG}/ echo ""

echo ""

echo "" chown -R 999:999 /app/test_results/${CI_COMMIT_REF_SLUG}

---

apiVersion: batch/v1

kind: Job

metadata:

name: {{ .Chart.Name }}-test

annotations:

"helm.sh/hook": post-install,post-upgrade

"helm.sh/hook-weight": "2"

"werf/watch-logs": "true"

spec:

activeDeadlineSeconds: {{ .Values.global.ci_timeout }}

backoffLimit: 1

template:

metadata:

name: {{ .Chart.Name }}-test

spec:

containers:

- name: test

command: ['bash', '-c', '/app/tests.sh']

{{ tuple "application" . | include "werf_container_image" | indent 8 }}

env:

- name: env

value: {{ .Values.global.env }}

- name: CI_COMMIT_REF_SLUG

value: {{ .Values.global.commit_ref_slug }}

- name: APP_NAME

value: {{ .Chart.Name }}

{{ tuple "application" . | include "werf_container_env" | indent 8 }}

volumeMounts:

- mountPath: /app/test_results/

name: data

- mountPath: /app/tests.sh

name: tests-script

subPath: tests.sh

tolerations:

- key: dedicated

operator: Exists

- key: node-role.kubernetes.io/master

operator: Exists

restartPolicy: OnFailure

volumes:

- name: data

persistentVolumeClaim:

claimName: {{ .Chart.Name }}-pvc

- name: tests-script

configMap:

name: tests-script

---

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: {{ .Chart.Name }}-pvc

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 10Mi

storageClassName: {{ .Chart.Name }}-{{ .Values.global.commit_ref_slug }}

volumeName: {{ .Values.global.commit_ref_slug }} ---

apiVersion: v1

kind: PersistentVolume

metadata:

name: {{ .Values.global.commit_ref_slug }}

spec:

accessModes:

- ReadWriteOnce

capacity:

storage: 10Mi

local:

path: /mnt/tests/

nodeAffinity:

required:

nodeSelectorTerms:

- matchExpressions:

- key: kubernetes.io/hostname

operator: In

values:

- kube-master

persistentVolumeReclaimPolicy: Delete

storageClassName: {{ .Chart.Name }}-{{ .Values.global.commit_ref_slug }}

{{- end }}

What resources does this configuration describe? We’re creating a unique namespace for the application during deployment (as stated in the .gitlab-ci.yaml — tests-${CI_COMMIT_REF_SLUG} ) and deploying several components there:

ConfigMap with a test script; A Job with a description of the pod and specified command that actually runs the tests; PV and PVC where the test data will be stored.

Note the initial if statement at the beginning of the manifest. To prevent other YAML files of Helm chart with an application from deploying, you have to insert the following reverse condition:

{{- if ne .Values.global.run_tests "yes" }}

---

Hey, I'm another YAML

{{- end }}

Yet if some tests require additional infrastructure (like Redis, RabbitMQ, Mongo, PostgreSQL…), then you can leave corresponding YAML files enabled and deploy them into the testing environment (of course, feel free to modify them as you see fit).

Final touch

Currently, building and deploying with werf are supported via build server (with gitlab-runner) only. However, the testing pod runs on the master node. In these circumstances, you have to create /mnt/tests folder on the master node and mount it to the runner, e.g. via NFS. A detailed example is available in the K8s docs.

We’ll get the following result:

user@kube-master:~$ cat /etc/exports | grep tests

/mnt/tests IP_gitlab-builder/32(rw,nohide,insecure,no_subtree_check,sync,all_squash,anonuid=999,anongid=998) user@gitlab-runner:~$ cat /etc/fstab | grep tests

IP_kube-master:/mnt/tests /mnt/tests nfs4 _netdev,auto 0 0

The other possibility is to create a shared NFS directory directly on the gitlab-runner and then mount it to pods.

Explanatory note

You may ask, what is the point of creating a Job if you could easily run a test script right in the shell-runner? The answer is obvious:

Some tests require the infrastructure (like MongoDB, RabbitMQ, PostgreSQL, and so on) to check the functionality. Our approach is a unified solution that allows for easy integration of additional instances. As a bonus, we get a standard deployment approach (even if using NFS and extra mounting of directories).

Result

What would be the result of applying the prepared configuration?

The merge request will show a summary of tests executed in its previous pipeline:

Click on the error to get more info:

NB: The attentive reader will notice that we are testing Node.js application, however there is a .NET one on the screenshots. Don’t be surprised: while we have found no issues in the original application (at the moment of writing this article), some of them have been revealed in another one.

Conclusion

It’s so easy, as you can see!

If you already have a working builder for a shell and don’t need Kubernetes, you can supplement it with testing even more effortlessly than described here. There are examples for Ruby, Go, Gradle, Maven, and some other products in the GitLab CI documentation.

This article has been originally written and published in Russian language by Flant’s engineer Ruslan Baimuhametov. More technical stuff from our engineers is to come soon — please don’t forget to follow our blog!