The images are set by their Docker url (OpenShift image streams are not supported here), so the cluster must access those registries. In this example above, we built the image of our app earlier within the same Kubernetes cluster and now pull it from the internal registry: 172.30.1.1 (docker-registry.default.svc). This image is actually our release package that may be deployed to dev, test or prod environment. It’s started with a k8sit application properties profile where the connection urls point to 127.0.0.1.

It’s important to think about memory usage for containers running java processes. Current versions of java (v1.8, v1.9) ignore the container memory limit by default and set a much higher heap size. The v3.9 jenkins-slave images support memory limits via environment variables much better than earlier versions. Setting JNLP_MAX_HEAP_UPPER_BOUND_MB=64 was enough for us to run maven tasks with 512MiB limit.

All containers within the pod have a shared empty dir volume mounted at /home/jenkins (default workingDir). This is used by the Jenkins agent to run pipeline step scripts within the container, and this is where we check out our integration test repository. This is also the current director where the steps are executed unless they are within a dir(‘relative_dir’) block. Here are the pipeline steps for the example above:

The pipeline steps are run on the jnlp container unless they are within a container(‘container_name’) block:

First, we check out the source of the integration project. In this case it’s in the integration-test sub directory within the repo.

We have the sql/setup.sh script to create tables and load test data in the database. It requires the mysql tool, so it must be run in the mariadb container.

Our application (app-users) calls a Rest API. We have no image to start this service, so we use MockServer to bring up the http endpoint. It’s configured by the mockserver/setup.sh.

The integration tests are written in Java with Junit and executed by Maven. It could be anything else — this is simply the stack we’re familiar with.

There are plenty of configuration parameters for podTemplate and containerTemplate following the Kubernetes resource api with a few differences. Environment variables, for example, can be defined at the container as well as at the pod level. Volumes can be added to the pod, but they are mounted on each container at the same mountPath:

podTemplate(...

containers: [...],

volumes:[

configMapVolume(mountPath: '/etc/myconfig',

configMapName: 'my-settings'),

persistentVolumeClaim(mountPath: '/home/jenkins/myvolume',

claimName:'myclaim')

],

envVars: [

envVar(key: 'ENV_NAME', value: 'my-k8sit')

]

)

Sounds easy, but…

Running multiple containers in the same pod is a nice way to attach them, but there is an issue that we can run into if our containers have entry points with different user ids. Docker images used to run processes as root, but it’s not suggested in production environments due to security concerns, so many images switch to a non-root user. Unfortunately, different images may use different uid (USER in Dockerfile) that can cause file permission issues if they use the same volume.

In this case the source of conflict is the Jenkins workspace on the workingDir volume (/home/jenkins/workspace/). This is used for pipeline execution and saving step outputs within each container. If we have steps in a container(…) block and the uid in this image is different (non-root) than in the jnlp container, we’ll get the following error:

touch: cannot touch '/home/jenkins/workspace/k8sit-basic/integration-test@tmp/durable-aa8f5204/jenkins-log.txt': Permission denied

Let’s have a look at the USER in images used in our example:

jenkins-slave-maven: uid 1001, guid 0

mariadb: uid 27, guid 27

amq: uid 185, guid 0

mockserver: uid 0, guid 0 (requires anyuid)

fuse-java-openshift: uid 185, guid 0

The default umask in the jnlp container is 0022, so steps in containers with uid 185 and uid 27 will run into the permission issue. The workaround is to change the default umask in the jnlp container so the workspace is accessible by any uid:

See the whole Jenkinsfile that first builds the app and the Docker image before running the integration test: kubernetes-integration-test/Jenkinsfile

In these examples the integration test in run on the jnlp container because we picked Java and Maven for our test project and the jenkins-slave-maven image can execute that. This is of course not mandatory, we can use the jenkins-slave-base image as jnlp and have a a separate container to execute the test. See an example where we intentionally separate jnlp and use another container for maven: kubernetes-integration-test/Jenkinsfile-jnlp-base

Yaml template

The podTemplate and containerTemplate definitions support many configurations, but they lack a few parameters. For example:

Can’t assign environment variables from ConfigMap, only from Secret.

Can’t set readiness probe for the containers. Without them Kubernetes reports the pod running right after kicking off the containers. Jenkins will start executing the steps before the the processes are actually ready to accept requests. This can lead to failures due to racing conditions. These example pipelines typically work because checkout scm gives enough time for the containers to start. Of course a sleep helps, but defining readiness probes is the proper way.

As an ultimate solution a yaml parameter was added to the podTemplate() in kubernetes-plugin (v1.5+). It supports a complete Kubernetes Pod resource definition, so we can define any configuration for the pod:

Make sure to update the Kubernetes plugin in Jenkins (to v1.5+) otherwise the yaml parameter is silently ignored.

Yaml definition and other podTemplate parameters supposed to be merged in a way, but it’s less error-prone to only use one or the other. Defining the yaml inline in the pipeline may be difficult to read, see this example of loading it from a file: kubernetes-integration-test/Jenkinsfile-yaml

Declarative Pipeline syntax

All the example pipelines above used the Scripted Pipeline syntax which is practically a groovy script with pipeline steps. The Declarative Pipeline syntax is a new approach enforcing more structure on the script by providing less flexibility and allowing no “groovy hacks”. It results in cleaner code, but you may have to switch back to the scripted syntax in case of complex scenarios.

In declarative pipelines the kubernetes-plugin (v1.7+) supports only the yaml definition to define the pod:

It’s also possible to set a different agent for each stage as in:

kubernetes-integration-test/Jenkinsfile-declarative

Try it on Minishift

If you’d like to try the solution described above you’ll need access to a Kubernetes cluster. At Red Hat we use OpenShift, which is an enterprise ready version of k8s. There are several ways to have access to a full scale cluster:

OpenShift Container Platform on your own infrastructure

OpenShift Dedicated cluster hosted on a public cloud

OpenShift Online on-demand public environment

It’s also possible to run a small one-node cluster on your local machine, which is probably the easiest way to try things. Let’s see how to setup Red Hat CDK (or Minikube) to run our tests.

After downloading, prepare the Minishift environment:

Run setup:

minishift setup-cdk

Set the internal Docker registry as insecure:

minishift config set insecure-registry 172.30.0.0/16

This is needed because the kubernetes-plugin is pulling the image directly from the internal registry which is not https.

This is needed because the kubernetes-plugin is pulling the image directly from the internal registry which is not https. Start the Minishift virtual machine (use your free Red Hat account):

minishift --username me@mymail.com --password ... --memory 4GB start

Note the console url or you can get it by:

minishift console --url

Add oc tool to the path:

eval $(minishift oc-env)

Login to OpenShift api (admin/admin):

oc login https://192.168.42.84:8443

Start a Jenkins master within the cluster using the template available:

oc new-app --template=jenkins-persistent -p MEMORY_LIMIT=1024Mi

Once Jenkins is up it should be available via a route created by the template, e.g.( https://jenkins-myproject.192.168.42.84.nip.io). Login is integrated with OpenShift (admin/admin).

Create a new Pipeline project that takes the Pipeline script from SCM pointing to a Git repository (e.g. https://github.com/bszeti/kubernetes-integration-test.git) having the Jenkinsfile to execute. Then simply Build Now.

The first run takes longer as images are being downloaded from the Docker registries. If everything goes well, we can see the test execution on the Jenkins build’s Console Output. The dynamically created pods can be seen on the OpenShift Console under My Project / Pods.

If something goes wrong, try to investigate by looking at:

Jenkins build output

Jenkins master pod log

Jenkins kubernetes-plugin configuration

Events of created pods (maven or integration-test)

Log of created pods

If you’d like to make the additional executions quicker, you can use a volume as local maven repository, so maven doesn’t have to download dependencies every time. Create a PersistentVolumeClaim:

# oc create -f - <<EOF

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

name: mavenlocalrepo

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 10Gi

EOF

Add the volume to the podTemplate (and optionally the maven template in kubernetes-plugin). See kubernetes-integration-test/Jenkinsfile-mavenlocalrepo:

volumes: [

persistentVolumeClaim( mountPath: '/home/jenkins/.m2/repository',

claimName: 'mavenlocalrepo')

]

Note: Maven local repositories claimed to be “non-thread safe” and should not be used by multiple builds at the same time. We use a ReadWriteOnce claim here that will be mounted to one pod only at a time.

The jenkins-2-rhel7:v3.9 image has kubernetes-plugin v1.2 installed. To run the Jenkinsfile-declarative and Jenkinsfile-yaml examples you need to update the plugin in Jenkins to v1.7+.

To completely cleanup after stopping Minishift delete ~/.minishift directory.

Limitations

There are certain aspects of the solution described above that I also want to mention here. Each project is different so it’s important to understand the impact of these in your case:

Using the jenkins-kubernetes-plugin to create the test environment is independent from the integration test itself. The tests can be written using any language and executed with any test framework — which is a great power but also a great responsibility.

The whole test pod is created before the test execution and shut down afterwards. There is no solution provided here to manage the containers during test execution. It’s possible to split up your tests to different stages with different pod templates, but that adds a lot of complexity.

The containers are started before the first pipeline steps are executed. Files from the integration test project are not accessible at that point so we can’t run prepare scripts or provide configuration files for those processes.

All containers belong to the same pod so they must run on the same node. If we need many containers and the pod requires too much resource there may be no node available to run the pod.

The size and scale of the integration test environment should be kept low. Though it’s possible to start up multiple microservices and run end-to-end tests within one pod, the number of required containers can quickly increase. This environment is also not ideal to test high availability and scalability requirements.

The test pod is recreated for each execution, but obviously the state of the containers is still kept during its run. This means that the individual test cases are not independent from each other. It’s the test project’s responsibility to do some cleanup between them if needed.

Summary

Running integration tests in an environment created dynamically from code is relatively easy by using Jenkins pipeline and the kubernetes-plugin. We only need a Kubernetes cluster and some experience with containers. Fortunately, more and more platforms provide official Docker images on one of the public registries. Worst-case scenario we have to build some ourselves. The hustle of preparing the pipeline and integration tests pays back quickly, especially when we want to try different configurations or dependency version upgrades during the life-cycle of our application.

Happy Testing!