Dynamic Jenkins agent provisioning in Kubernetes

Introduction

Jenkins is CI/CD tool with long history and keep evolving itself. It’s Master/Agent architecture is great for scalability to do distributed builds. There are many ways to provision Jenkins Agent, from using bare metal machines, Virtual Machines, dynamic EC2 instances, containers from Docker, or Kubernetes clusters.

The integration between Jenkins and Kubernetes cluster is great. With these benefits, I’ve fully migrated CI/CD pipeline from host(VM) based Agent to Pod based Agent.

Dynamic Jenkins agent from Kubernetes, light weighted, provisioned on-demand within a few seconds

Fresh and reproducible Jenkins agent environment for every build

Resource/cost saving brought by Kubernetes

In this article, I’d like to share my recent approach to dynamically provision a Jenkins Agent with simple lines of code in Jenkins pipeline, using the method of Jenkins Shared Library.

Declarative pipeline

<-- invoke library

pipeline {

agent {

kubernetes(k8sagent(name: 'mini+pg')) <- Jenkins Agent provisioned

}

stages {

stage('demo') {

steps {

echo "this is a demo"

script {

container('pg') {

sh 'su - postgres -c \'psql --version\''

}

}

}

}

}

} @Library ("k8sagent@v0.1.0") _pipeline {agent {kubernetes(k8sagent(name: 'mini+pg'))stages {stage('demo') {steps {echo "this is a demo"script {container('pg') {sh 'su - postgres -c \'psql --version\''

Scripted pipeline

<-- invoke library

my_node = k8sagent (name: 'mini+pg' )

podTemplate(my_node) { <--Jenkins Agent provisioned

node(my_node.label) {

sh 'echo hello world'

container('pg') {

sh 'su - postgres -c \'psql --version\''

}

}

} @Library ("k8sagent@v0.1.0") _my_node = k8sagent (name: 'mini+pg' )podTemplate(my_node) {node(my_node.label) {sh 'echo hello world'container('pg') {sh 'su - postgres -c \'psql --version\''

In this one line example, the name ‘mini+pg’ is to request a Jenkins Agent of ‘mini’ size Pod with standard JNLP container plus a ‘postgresql’ container. The type ‘mini’ and ‘pg’ are defined templates in resources files . There are other templates like “small’, ‘large’, ‘privileged’ in the library that you are free to mix/add/change.

Beyond the capability of dynamically request/release a Jenkins Agent to run your CI/CD pipelines, this is a method to dynamically define the required Agent. I describe how to do extension for your usage below.

Source code

How Jenkins works with Kubernetes

There are lots of introduction articles, listing a few( GCP, AWS, Azure, etc). They are all based on the Kubernetes plugin for Jenkins.

There are 2 kinds of integration model.

An existing Jenkins instance connects to new Kubernetes cluster.

When I did the integration for the first time, I already have a production Jenkins running, so this is definitely the straight way.

Note: A link about how to authenticate Jenkins to K8s.

Stand-alone Jenkins + K8s cluster

2. Create a new Jenkins instance inside the Kubernetes cluster.

After putting the first model into production, with the idea to run multiple Jenkins instances for different purposes, I tried the second model and think it’s the better method.

Jenkins in Kubernetes cluster

Both models could work nicely and even within same Kubernetes cluster, in different namespaces.

Official guideline to provision Agent

The Kubernetes plugin for Jenkins has provided documentation and examples for various methods to define simple/complex Agent.

The Jenkins Agent is a Pod provisioned by Kubernetes, with multiple containers which run the specified Docker images. The main container must be from jenkins/jnlp-slave or compatible image. It’s better to build your own image including the Dockerfile content from jenkins/docker-slave.

It’s very flexible to use all kinds of mechanisms provided by Kubernetes, including multiple containers in the Pod, sharing the network among them, sharing the work directory, init containers, resource limit, environments, secrets, etc…

Basically there are 3 methods to provision the Agent,

YAML in declarative pipeline

Configurations in scripted pipeline

Configurations in Jenkins UI

Declarative pipeline, which is to write down the yaml file for the intended Pod.

link pipeline {

agent {

kubernetes {

yaml """

<A long yaml file>

"""

}

}

...

2. Scripted pipeline, which is to config all the parameters.

link podTemplate(cloud: 'kubernetes', containers: [

containerTemplate(

<All kinds of configurations>

]) {

...

}

3. Jenkins Configurations, which is to fill all the configurations for “Pod Template” into the UI.

Pod Templates Configurations in Jenkins UI

Method 3 is the starting point for me, as it’s the same way as other kinds of Agent, defined in the Jenkins master.

I defined multiple “Pod Templates” for various tasks. For example, a general-purpose ‘slave’ with 4 CPU/8GB RAM is suitable for most builds. Some large projects may need 8 CPU/16GB RAM, called ‘large-slave’. Some other may need an additional PostgreSQL container for its testing, called ‘slave-pg’.

Some templates are similar that we can even use the “Template Inheritance” feature to save some effort, which is nice.

In the pipeline, it’s easy to use the label of a Pod Template to choose the Agent.

pipeline {

agent {

label "slave"

}

...

}

There are a few problems observed over time.

Pod Template is too complex to configure from Jenkins UI

A single Pod Template could easily have more than 30 configuration items. This shows the great flexibility of Kubernetes but not good for UI.

Multiple Pod Templates are not manageable

As the need of “various tasks” increases, the number of configured templates increase. With not so many (less than 10) templates, the Jenkins Configuration UI has more than 10 pages for this section. It’s really difficult to find the right place to do the right changes.

Pod Templates configurations across Jenkins instances

As I put more Jenkins instances into production, it becomes a problem to configure these templates across instances. I doubt anybody would like to copy the whole bunch 10 pages of configurations from one Jenkins to another.

Jenkins Configuration as Code(JCasC) is not yet the saver, I believe.

Lack of Determinism

The way for project to use “agent label A” in its pipeline create a dependency on the “Agent A” defined in the Jenkins like a mystery. There is no way to know exactly what is in the Agent A, whether A is same today as yesterday, or A is different among Jenkins instances. It’s a common problem for all kinds of Agent if the Agent is created in changeable way.

Lack of Trace-ability

As a project evolves its requirement for the Agent to build it evolves too. For example project v1.0.0 might only need a standard Agent while v1.5.0 need Agent with PostgreSQL Container for testing. Sometimes docker images used in the Agent need upgrade. It sounds easy to request Jenkins administrator to modify the Agent or create new one. But soon there is no trace-ability for such changes.

Even with some problems, this method is still a must have for those pipelines which must run on various kinds of Agents other than a Pod in Kubernetes.

Pod Template in Pipeline Source code

By looking at the document of Kubernetes plugin for Jenkins, I believe the design idea is to write the Pod Template in the pipeline source code. There are many examples for both declarative and scripted pipelines.

In my observation, as the plugin versions upgrade, the plugin is taking steps to prefer YAML to define the Agent over configurations. I welcome this change as it’s very comfortable to use the same YAML way for Jenkins pipeline and other Kubernetes tasks.

A related topic is about how the pipeline code is integrated with a project, e.g. should it be inside the same source repository or outside of a project. It depends on lots of factors to tell which is better (or suitable). In my case I have to handle both.

I met 2 main problems when trying this approach.

Below is an example pipeline copied from one of the plugin’s examples.

In this simple example, the YAML to define the Pod Template is 20 lines of code, while the build logic is 5 lines.

In my real world experience, an actual useful YAML is easily more than 50 lines. I feel this is too long in the pipeline code. And it’s repetitive code for similar projects.

If you have 100 projects, it’s hard to modify 100 projects to for the Pod Template YAML in the pipeline code.

Different organizations have different development models. There isn’t a single answer who shall be responsible for the project pipeline, specifically for the Pod Template part in the pipeline code.

If it’s individual project team, it’s impossible to perform changes at same time. For example, when there is need to upgrade the docker image for security patch, an immediate action shall be done at the same time for all projects.

If it’s dedicated team, I couldn’t image the effort to modify 100 projects effectively. It’s very likely to miss some.

Move Pod Template to Jenkins Shared Library

When there are common things across Jenkins pipeline code, it’s natural to move them to Jenkins Shared Library.

When I was looking for a solution to use library for Kubernetes Agent provisioning, I’m inspired by this salemove/pipeline-lib project.

It’s a Jenkins Shared Library which provides inPod method. inPod is a thin wrapper around podTemplate and provides flexibility to combine predefined configurations with argument configurations. With inPod as base, some other wrappers define various arguments, for example inDockerAgent, inRubyBuildAgent, to provide different kinds of Agents.

For example, below piece of code provisions an Agent with ‘node:9-alpine’ docker image.

inPod(containers: [interactiveContainer(name: 'node', image: 'node:9-alpine')]) {

checkout(scm)

container('node') {

sh('npm install && npm test')

}

}

The main idea is to collect the parameters from the wrapper layers and put them together for the podTemplate to be used by Kubernetes plugin.

There is a key function addWithoutDuplicates, implemented here, to merge List together, giving precedence to args over default.

// For containers, add the lists together, but remove duplicates by name, giving precedence to the user specified args.

def finalContainers = addWithoutDuplicates((args.containers ?: []), defaultArgs.containers) { it.getArguments().name }

By looking into the project’s introduction, it’s a successful Library already in production. The idea is good and when I wanted to import for my own usage, I found it does not fit for my case.

The Library depends on method 2, providing configurations to Pod Template. While I believe the new trend is YAML.

The Library works for scripted pipelines only. I have both declarative pipelines and scripted pipelines in my work scope. I need one solution for both.

There are many configurations for podTemplate, but those are not all the things supported by Kubernetes. In fact there is a ‘yaml’ configuration to provide raw yaml. The raw yaml will be merged with other configurations.

The Library focus on providing different docker images/volumes to Agent. It’s nice but I need more than that, and many of them are only supported in ‘raw yaml’.

Jenkins Library to get dynamic Agent from Kubernetes

With these considerations, I would like to explore a new solution with these requirements:

Based on method 1, full YAML format

Works for both scripted pipelines and declarative pipelines

Flexibility to provision various kinds of Agent

Easy to use in the pipeline code and hide details in the Library

Easy to develop the Library to support increasing demands of Agent types

After some investigation I believe my idea is feasible, by looking at these 2 examples from Kubernetes plugin: dind.groovy and declarative.groovy.

The common thing between these 2 examples is like below

# scripted pipeline

podTemplate(Map) {

} # declarative pipeline

agent {

kubernetes {

Map

}

}

The target is to generate the Map, with the desired yaml content which defines the Pod spec for the Agent.

The approach is to define small pieces of partial yaml, each for a specific requirement for the Agent, and merge them together to compose a final yaml format spec for the Pod.

For example, Agent may need do maven or gradle build, a resource file named ‘maven’ to include a container with ‘maven’ docker image, another file ‘gradle’ to include ‘gradle’ docker image. Agent may require less or more computing power, mapping to resource files ‘small’/ ‘large’/ ‘fast’ to request some cpu/memory resources. Agent may require ‘privileged’ Pod in the build, a resource file ‘privileged’ to define the ‘securityContext’ and in my case, together with ‘podAntiAffinity’ to make sure one such privildged Pod in one host machine.

To add them together, Agent is defined in format of ‘A+B+C’, for example ‘maven+small+priviledged’ , and the Library will return the merged yaml. Code example for ‘small+pg’ :

# base

spec:

hostAliases:

- ip: "192.168.1.15"

hostnames:

- "jenkins.example.com"

volumes:

- hostPath:

path: /data/jenkins/repo_mirror

type: ""

name: volume-0

containers:

- name: jnlp

image: jenkinsci/jnlp-slave:3.29-1

imagePullPolicy: Always

command:

- /usr/local/bin/jenkins-slave

volumeMounts:

- mountPath: /home/jenkins/repo_cache

name: volume-0 # small

spec:

containers:

- name: jnlp

resources:

limits:

memory: 8Gi

requests:

memory: 4Gi

cpu: 2 # pg

spec:

containers:

- name: pg

image: postgres:9.5.19

tty: true # merged

spec:

hostAliases:

- ip: "192.168.1.15"

hostnames:

- "jenkins.example.com"

volumes:

- hostPath:

path: /data/jenkins/repo_mirror

type: ""

name: volume-0

containers:

- name: jnlp

image: jenkinsci/jnlp-slave:3.29-1

imagePullPolicy: Always

command:

- /usr/local/bin/jenkins-slave

volumeMounts:

- mountPath: /home/jenkins/repo_cache

name: volume-0

resources:

limits:

memory: 8Gi

requests:

memory: 4Gi

cpu: 2

- name: pg

image: postgres:9.5.19

tty: true

For easy development, I started with a similar gradle project which uses TDD method to develop a Jenkins Shared Library. I introduced it in this article and the original source code is github link.

The gradle project has a structure like this:

.

├── build.gradle

├── resources

│ └── podtemplates <- many Yaml files

├── src

│ └── com/github/liejuntao001/jenkins/MyYaml.groovy <- merge Yaml

├── test

│ └── groovy

│ └── K8sAgentTest.groovy <- test

├── testjobs

│ ├── k8sagent_Jenkinsfile.groovy

│ └── simple_Jenkinsfile.groovy <- samples

│ └── simple_scripted.groovy

└── vars

└── k8sagent.groovy <- method

To run the test

./gradlew clean test



> Task :test

K8sAgentTest: testSdk25: SUCCESS

K8sAgentTest: testSmall: SUCCESS

K8sAgentTest: testBase: SUCCESS

K8sAgentTest: testDind: SUCCESS

K8sAgentTest: testFast: SUCCESS

K8sAgentTest: testPg: SUCCESS

K8sAgentTest: testPrivileged: SUCCESS

Tests: 7, Failures: 0, Errors: 0, Skipped: 0

To further develop the Library, simply modify the resource files in resources/podtempltes to fit the new requirements.

Add test cases in test/groovy/K8sAgentTest.groovy to test the merged Yaml againt the expected Yaml content of Agent.

Some notes:

It’s suggested that all the docker images used in the library are self-built private images to be sure about its content

It’s suggested that all the docker images are tagged with version number to get deterministic Agent.

It’s suggest that the whole Jenkins Library is released with version tag. All the pipelines invoke the Library shall use @Library(“k8sagent@v0.1.0”) _

There is a base.yaml in source code which defines the jnlp container and the common things required for all the Agent types. In k8sagent.groovy, base.yaml is included in all Agents. If there are requirements to make a common base impossible, k8sagent.groovy need update.

Yaml merge strategy

To my understanding, there isn’t a ‘One-Fit-All’ Yaml merge method.

For example, Yaml’s main data structure is Map and List. In Kubernetes’s spec, the List is sometimes sequence sensitive, sometimes insensitive.

# sensitive List args

command: ["/bin/sh"]

args: ["-c", "while true; do echo hello; sleep 10;done"] # insensitive List containers

containers:

- name: A

- name: B

In Kubernetes plugin, there is a parameter yamlMergeStrategy: merge() or override() to decide how to merge the Yaml in inheritance.

In my implementation, I reused code from OndraZizka/yaml-merge and modified to ‘just-works’ level to fit my scenarios. It resolves a use case like below:

# a.yaml

containers:

- name: A

some_config # b.yaml

containers:

- name: A

some_other_config # merged yaml

containers:

- name: A

some_config

some_other_config

For more complex scenarios, MyYaml.groovy may need update. But I think most cases could be resolved by define the find grained Yaml files.

References

Kubernetes plugin for Jenkins

Thanks for reading.