Running applications in containers has become a well-accepted practice lately in the enterprise sector as Docker with Kubernetes provides a scalable, manageable application platform. The container-based approach also suits well the microservices architecture that gained significant momentum in the past few years.

Now let’s just focus on one of the most important advantages of having a container application platform: we can dynamically bring up isolated containers with resource limits. Let’s check out how this can change the way we run our CI/CD tasks.

Building and packaging an application needs an environment that can download the source code, access dependencies and has the build tools installed. Running unit and component tests as part of the build may use local ports or require third party applications (databases, message brokers, etc…) running somewhere. At the end, we usually have multiple pre-configured build servers each running a certain type of job. For tests, we maintain dedicated instances of third party apps (or try to run them embedded) and avoid running jobs parallel that could mess up each other’s outcome. The required pre-configuration for such a CI/CD environment can be a hassle, and the required number of servers for different jobs can significantly change by time as teams shift between versions and development platforms.

Once we have access to a container platform (on-site or in the cloud), why not move the resource intensive CI/CD task executions into dynamically created containers? Build environments can be started and configured for each job execution independently. Tests during the build have free rein to use available resources in this isolated box, while we can also bring up a third party application in a side container that only exists for the lifecycle of this job.

It sounds nice… let’s see how it actually works in real life.

This blog is based on a real world solution we put together for a project recently running on a Red Hat OpenShift v3.7 cluster. OpenShift is the enterprise ready version of Kubernetes so these practices work on a k8s cluster as well. To try download the Red Hat CDK and run ‘jenkins-ephemeral’ or ‘jenkins-persistent’ template that creates a pre-configured Jenkins master on OpenShift for you.

Solution overview

The solution to execute CI/CD tasks (builds, tests, etc.) in containers on OpenShift is based on Jenkins distributed builds, which means that:

a Jenkins master is needed — it may run inside the cluster, but it also works with an external master

Jenkins features/plugins are available as usual, so the existing projects can be used

Jenkins GUI is available to configure, run and browse job output

if you prefer code, Jenkins Pipeline is available

From the technical point of view, the dynamic containers to run jobs are Jenkins agent nodes. When a build is kicked off, first a new node is started and “reports for duty” to the Jenkins master via JNLP (port 5000). The build is queued until the agent node comes up and picks up the build. The build output is sent back to the master — just like in case of regular Jenkins agent servers — but the agent container is shutdown once the build is done.

Different kind of builds (e.g., java, nodejs, python, etc.) need different agent nodes. This is nothing new — labels could be used before as well to restrict which agent nodes should run a build. To define the config for these Jenkins agent containers started for each job, you will need to set the following:

The docker image to boot up

Resource limits

Environment variables

Volumes mounted

The core component here is the Jenkins Kubernetes plugin. This plugin interacts with the k8s cluster (by using a ServiceAccount) and starts/stops the agent nodes. Multiple agent types can be defined as “Kubernetes Pod Template” under the plugin’s configuration (refer them by label in projects).

These agent images are provided out-of-the-box (also on centos7):

jenkins-slave-base-rhel7: base image starting the agent that connects to Jenkins master. Java heap set according to container memory.

jenkins-slave-maven-rhel7: image for Maven and Gradle builds (extends base)

jenkins-slave-nodejs-rhel7: image with NodeJS4 tools (extends base)

Please note, the solution described here is not related to OpenShift’s Source-to-image (S2I) build, but that can also be used for certain CI/CD tasks.

Learning material

There are several blogs and good documentation about Jenkins builds on OpenShift. These links are good to start with:

Have a quick look at these resources above to understand the overall solution. Here I only would like to show the different issues we run into while applying those practices.

Build MyApplication

As an example, let’s assume a java project with the following build steps:

Source: pull project source from a Git repository.

Build with Maven: dependencies are coming from an internal repository (let’s use Apache Nexus) mirroring external maven repos.

Deploy artifact: the built jar is uploaded to the repository.

During the CI/CD process, we need to interact with Git and Nexus so the Jenkins jobs need to access those systems somehow. This requires configuration and stored credentials that can be managed at different places:

in Jenkins itself: you can add credentials to Jenkins that the git plugin can use and add files somehow to the project. There’s nothing new just because we use containers.

in OpenShift: use ConfigMap and Secret objects that are added to the Jenkins agent containers as files or environment variables.

in a fully customized Docker image: pre-configured with everything to run a type of job. Extend one of the agent images.

It’s a question of taste which approach is preferred, and your final solution may be a mix. Below we have a look at the second option when the configuration is managed primarily in OpenShift. We customize the maven agent container via Kubernetes plugin configuration by setting environment variables and mounting files.

Adding environment variables through the UI doesn’t work with Kubernetes plugin v1.0 due to a bug. Update plugin or edit the config.xml directly as a workaround and restart Jenkins.

Pull source from Git

Pulling a public git is trivial. For a private git repo authentication is required, and also the client needs to trust the server for a secure connection. Git pull can be done typically via two protocols:

HTTPS: Authentication is with username/password. The server’s SSL certificate must be trusted by the job, which is only tricky if it’s signed by a custom CA.

git clone https://git.mycompany.com:443/myapplication.git

SSH: Authentication is with a private key. The server is trusted when its public key’s fingerprint is found in known_hosts file.

git clone ssh://git@git.mycompany.com:22/myapplication.git

Downloading source through HTTP with username/password is ok when it’s done manually, for automated builds SSH is better.

Git with ssh

For SSH download obviously we need to make sure that ssh connection works between the agent container and the git’s ssh port. First we need a private-public key pair. To generate one run:

ssh keygen -t rsa -b 2048 -f my-git-ssh -N ''

It generates a private key in my-git-ssh (empty passphrase) and the matching public key in my-git-ssh.pub. Add the public key to the user on the git server (preferably a service-account), web UIs usually support upload. To make ssh connection work you need two files on the agent container:

The private key at ~/.ssh/id_rsa

The server’s public key in ~/.ssh/known_hosts

To get this, try ssh git.mycompany.com on your machine, accept the fingerprint, that will create a new line in your known_hosts file. Use that.

Store the private key as id_rsa and server’s public key as known_hosts in an OpenShift secret (or config map).

apiVersion: v1

kind: Secret

metadata:

name: mygit-ssh

stringData:

id_rsa: |-

-----BEGIN RSA PRIVATE KEY-----

...

-----END RSA PRIVATE KEY-----

known_hosts: |-

git.mycompany.com ecdsa-sha2-nistp256 AAA...

Then configure this as a volume in Kubernetes plugin for the maven pod at mount point /home/jenkins/.ssh/. Each item in the secret will be a file matching the key name under the mount directory. You can use the UI (Manage Jenkins / Configure / Cloud / Kubernetes), or edit Jenkins config /var/lib/jenkins/config.xml:

<org.csanchez.jenkins.plugins.kubernetes.PodTemplate>

<name>maven</name>

...

<volumes>

<org.csanchez.jenkins.plugins.kubernetes.volumes.SecretVolume>

<mountPath>/home/jenkins/.ssh</mountPath>

<secretName>mygit-ssh</secretName>

</org.csanchez.jenkins.plugins.kubernetes.volumes.SecretVolume>

</volumes>

Pulling git source through ssh should work in the jobs running on this agent now.

It’s also possible to customize the ssh connection in ~/.ssh/config. For example if we don’t want to bother with knows_hosts or the private key is mounted to a different location:

Host git.mycompany.com

StrictHostKeyChecking no

IdentityFile /home/jenkins/.config/git-secret/ssh-privatekey

Git with http

If HTTP download is preferred add the username/password to a git-credential-store file somewhere:

e.g. /home/jenkins/.config/git-secret/credentials from an OpenShift secret, one site per line:

enable it in git-config expected at /home/jenkins/.config/git/config

[credential]

helper = store --file=/home/jenkins/.config/git-secret/credentials

If the git service has a certificate signed by a custom CA the quickest hack is to set GIT_SSL_NO_VERIFY=true env var for the agent. The proper solution needs two things:

add the custom CA’s public certificate to the agent container from a config map to a path (e.g. /usr/ca/myTrustedCA.pem)

Tell git the path to this cert in an env var GIT_SSL_CAINFO=/usr/ca/myTrustedCA.pem

or in the git-config file mentioned above:

[http "https:// git.mycompany.com " ]

sslCAInfo = /usr/ca/myTrustedCA.pem

In OpenShift v3.7 (and earlier) config map and secret mount points must not overlap, so you can’t map to /home/jenkins and /home/jenkins/dir at the same time. This is why we didn’t use the well-known file locations above. Fix is expected in OpenShift v3.9.

Maven

To make Maven build work there are usually two things to do:

A corporate Maven repository (e.g. Apache Nexus) should be set up acting as a proxy for external repos. Use this as a mirror.

This internal repository may have an https endpoint with a certificate signed by a custom CA.

Having a internal Maven repository is practically essential if builds run in containers because they start with an empty local repository (cache), so maven downloads all the jars every time. Downloading from an internal proxy repo on local network is obviously quicker than directly from internet.

The maven Jenkins agent image supports an environment variable that can be used to set the url for this proxy. Set in Kubernetes plugin container template:

MAVEN_MIRROR_URL=https://nexus.mycompany.com/repository/maven-public

The build artifacts (jars) should also be archived in a repository, which may or may not be the same as the one acting as a mirror for dependencies above. Maven deploy requires the repo url in the pom.xml under distributionManagement (so this has nothing to do with the agent image):

Uploading the artifact may require authentication. In this case username/password must be set in the settings.xml under the server id matching the one in pom.xml. We need to mount a whole settings.xml with url, username and password on the maven Jenkins agent container from an OpenShift secret. We can also use environment variables as below:

Add environment variables from a secret to the container

MAVEN_SERVER_USERNAME=admin

MAVEN_SERVER_PASSWORD=admin123

Mount settings.xml from a config map to /home/jenkins/.m2/settings.xml

<settings ...>

<mirrors>

<mirror>

<mirrorOf>external:*</mirrorOf>

<url>${env.MAVEN_MIRROR_URL}</url>

<id>mirror</id>

</mirror>

</mirrors>

<servers>

<server>

<id>mynexus</id>

<username>${env.MAVEN_SERVER_USERNAME}</username>

<password>${env.MAVEN_SERVER_PASSWORD}</password>

</server>

</servers>

</settings>

Disable interactive mode (use batch mode) to skip download log by using ‘-B’ for maven commands or by adding <interactiveMode>false</interactiveMode > to settings.xml.

If the Maven repository https endpoint uses a certificate signed by a custom CA we need to create a Java KeyStore using keytool containing the CA certificate as trusted. This keystore should be uploaded as a config map in Openshift. Use the oc command to create config map from files:

oc create configmap maven-settings --from-file=settings.xml=settings.xml --from-file=myTruststore.jks=myTruststore.jks

Mount the config map somewhere on the Jenkins agent. In this example we use /home/jenkins/.m2, but only because we have settings.xml in the same config map, the keystore can go under any path.

Then make the Maven java process use this file as truststore by setting java params in the MAVEN_OPTS environment variable for the container:

MAVEN_OPTS=

-Djavax.net.ssl.trustStore=/home/jenkins/.m2/myTruststore.jks

-Djavax.net.ssl.trustStorePassword=changeit

Memory usage

This is probably the most important part because if you don’t set max memory correctly you’ll run into intermittent build failures after everything seems to work.

Running java in a container can cause high memory usage errors if we don’t set the heap in the java command line. The JVM sees the total memory of the host machine instead of the container’s memory limit and sets the default max heap accordingly. This is typically much more than the container’s memory limit and OpenShift simply kills the container when java process allocates more memory for the heap.

Though the jenkins-slave-base image has a built in script to set max heap to half of the container memory (can be modified via env var CONTAINER_HEAP_PERCENT=0.50 ), it only applies to the Jenkins agent java process. In case of a Maven build we have important additional java processes running:

The mvn command itself is a java tool

The maven-surefire-plugin executes the unit tests in a forked JVM by default

At the end of the day we’ll have three java processes running at the same time in the container and it’s important to estimate their memory usage to avoid unexpectedly killed pods. Each process has a different way to set JVM options:

Jenkins agent heap is calculated as mentioned above, but we definitely shouldn’t let the agent have such a big heap. Memory is needed for the other two JVMs. Setting JAVA_OPTS works for the Jenkins agent.

The mvn tool is called by the Jekins job. Set MAVEN_OPTS to customize this java process.

The JVM spawned by surefire for the unit tests can be customized by the argLine maven property. It can be set in the pom.xml, in a profile in settings.xml or simply by adding -DargLine=… to mvn command in MAVEN_OPTS

Here is an example how to set these environment variables for the maven agent container:

JAVA_OPTS=-Xms64m -Xmx64m

MAVEN_OPTS=-Xms128m -Xmx128m -DargLine=${env.SUREFIRE_OPTS}

SUREFIRE_OPTS=-Xms256m -Xmx256m

These numbers worked for us with 1024Mi agent container memory limit building and running unit tests for a SpringBoot app. These are relatively low numbers and bigger heap size and a higher limit may be needed for complex maven projects and unit tests.

The actual memory usage of a Java8 process is something like HeapSize + MetaSpace + OffHeapMemory, this can be significantly more than the max heap size set. With the settings above the three java processes took more than 900Mi memory altogether in our case. See rss memory for processes within container:

ps -e -o pid,user,rss,comm,args

The Jenkins agent images have both JDK 64bit and 32bit installed. For mvn and surefire the 64bit JVM is used by default. To lower memory usage it makes sense to force 32bit JVM as long as -Xmx is less than 1.5 GB:

JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161–0.b14.el7_4.i386

It’s also possible to set java arguments in JAVA_TOOL_OPTIONS env var, which is picked up by any JVM started. The parameters in JAVA_OPTS and MAVEN_OPTS overwrite the ones in JAVA_TOOL_OPTIONS, so we can achieve the same heap configuration for our java processes as above without using argLine:

JAVA_OPTS=-Xms64m -Xmx64m

MAVEN_OPTS=-Xms128m -Xmx128m

JAVA_TOOL_OPTIONS=-Xms256m -Xmx256m

it's still a bit confusing as all JVMs log “Picked up JAVA_TOOL_OPTIONS:”.

Jenkins pipeline

Following the settings above we should have everything prepared by now to run a successful build. We can pull the code, download the dependencies, run the unit tests and upload the artifact to our repository. Let’s create a Jenkins pipeline project that actually does this:

For a real project of course your CI/CD pipeline should do more than the maven build only: deploy to development environment, run integration tests, promote to higher environments, etc… The articles linked above shows examples how to do that.

Multiple containers

One pod can have multiple containers running each having their own resource limits. They share the same network interface, so you can reach started services on localhost, but you need to think about port collisions. Environment variables are set separately, but the volumes mounted are the same for all containers configured in one Kubernetes Pod Template.

Bringing up multiple containers is useful when an external service is required for unit tests and an embedded solution doesn’t work (e.g. database, message broker, etc.). In this case this second container also starts and stops with the Jenkins agent.

See the Jenkins config.xml snippet where we start an httpbin service on the side for our Maven build:

<org.csanchez.jenkins.plugins.kubernetes.PodTemplate>

<name>maven</name>

<volumes>

...

</volumes>

<containers>

<org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>

<name>jnlp</name>

<image>registry.access.redhat.com/openshift3/jenkins-slave-maven-rhel7:v3.7</image>

<resourceLimitCpu>500m</resourceLimitCpu>

<resourceLimitMemory>1024Mi</resourceLimitMemory>

<envVars>

...

</envVars>

...

</org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>

<org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>

<name>httpbin</name>

<image>citizenstig/httpbin</image>

<resourceLimitCpu></resourceLimitCpu>

<resourceLimitMemory>256Mi</resourceLimitMemory>

<envVars/>

...

</org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>

</containers>

<envVars/>

</org.csanchez.jenkins.plugins.kubernetes.PodTemplate>

Summary

As a summary see the created OpenShift resources and the Kubernetes plugin configuration from Jenkins config.xml with all the configuration described above.

One additional config map was created from files:

oc create configmap maven-settings --from-file=settings.xml=settings.xml --from-file=myTruststore.jks=myTruststore.jks

Kubernetes plugin configuration:

Happy Builds!