Building a reproducible CI system for fun and profit

What’s that you say? Fun and profit with CI? You must be mad!

Well yes, perhaps that’s putting things a bit strongly. But there are a lot of gains to be made by improving your CI setup.

Having a good CI system is an important part of an engineer’s flow. It’s vital that we can have a tool to easily build and test different projects and services, especially in this day and age where microservices are becoming increasingly popular.

And what better system than Jenkins? It’s a battle-hardened, proven system. At Smarkets we use it for its many features and extensive pluggability, allowing us to customise our builds as needed.

Yeah, you wish you were this cool¹

However, it’s not all a pretty picture. As we’ve grown, our CI hasn’t grown with us, making it hard for us to make changes to our everyday builds. But we put some time aside to improve things and we’re in a much better state for it. So how did we do it? We’ll get to that, but first off, a little backstory…

Our previous Jenkins setup, and why it sucked

Well, it didn’t suck, per se. It worked well enough for us to use daily without too many problems. The longstanding issues we had were maintenance and reproducibility: both things which can hit you hard when you least want them to.

While we had backups for the job state on our Jenkins master, a lot of our builds were not version controlled and simply existed as freestyle job created via the Jenkins UI. This meant if the host was lost or destroyed, those builds would be lost forever unless they were caught in the backup. Hardly ideal.

Of course, Jenkinsfiles exist as a way to have jobs as versioned code, but it was hard to enforce this across every team.

Case Study: Upgrades

Meet Bob. Bob is an eager young software engineer, keen to improve our CI system. Bob sets up linting and testing of all services, which is automated by Marge.

Bob’s happy with his setup, until he finds a small issue with how the tests work with Jenkins pipelines. He notices this is a bug fixed in a later plugin version. So what does Bob do? He decides to update the plugin…

Oh dear²

As it turns out, upgrading a plugin often requires another plugin to upgrade. Which in turn, require some more plugins to upgrade. And so on…

Eventually this gets to the point where a plugin may require Jenkins itself to be upgraded(!). Not a great position to be in when reproducibility is hard.

Conclusion: In Jenkins, it’s rather difficult to update a single plugin without adverse side effects.

Dockerising

So how did we fix these problems? Well, we started off by creating a consistent environment, with Docker.

Dockerising the Jenkins master was fairly simple. We used the official Jenkins image, which provides most of the boilerplate, then added some configuration to install our plugins and setup scripts. The end result was something like this: