Introduction

This short story is about how we refactored our CI pipeline trying to take the best of docker images and gitlab-ci runners. As I didn’t find much on Internet about a similar approach I’ll make an attempt at blogging.

Before using docker our average CI pipeline looked like this :

Pipeline before Docker. Pink jobs are running the before_script.

We had a before_script running before linting and tests — a 10-line-long script responsible to pull all the dependencies, build, install and configure the applications. It took between 4 to 10 minutes to run, depending on the dependencies and the build process.

This model was not perfect, but it worked pretty good and provided us what we needed for several years.

One day we decided to switch our microservices from bare-metal to docker containers, so we added docker support in the CI naively by adding a docker_build target before deployment. Our new pipelines looked approximately like this:

Naive pipeline with docker. Pink jobs are running the before_script.

At the same time we added some heavy dependencies in our microservice-skeleton and the build lengths increased drastically. Annnnd we started being annoyed by the CI time, sometimes exceeding 20–30 minutes. This was not acceptable for a CI job, developers would not wait that long before context switching. So the pipeline cost went really high on the process.

Me waiting for the pipeline to finish

Thus, we decided to refactor our pipeline process.

What we wanted (and what we didn’t want)

Long story short, here are the several needs we’ve identified :

Independent builds — two concurrent pipelines should not interact, if the last build failed it shouldn’t affect the next one;

Idempotency and repeatability — the same pipeline should do the same thing when started again;

Reduced CI time;

Resource-efficient — reduce network and cpu load of our poor runners;

Being able to test and deploy quickly on Kubernetes test cluster;

Lightweight images — and no SSH keys remaining in a docker layer (don’t laugh, it happens more frequently than you think);

Simple — to replay, understand, re-use for any developper and on any microservice/project;

Link between image name/commit /code/pipeline — know when I see an image from which commit and which pipeline it has been created.

We had to use our dedicated servers running gitlab-ci runners configured as docker executors and our self-hosted Gitlab configured with a private docker registry.

Trying to Refactor

The gitlab documentation told us we had some environment variables available during a job : the commit and ref-related variables (SHA, branch, tag, …), the pipeline related variables (CI_PIPELINE_ID) and the gitlab-related variables (host, registry, tokens, …).

First thing that came to our mind : can we use those variables outside the gitlab-ci script statement ? The answer is yes :)

As you may have already identified, pulling dependencies for each job and building each time was not a very efficient strategy.

We thought about playing with artifacts in order to move the built project from one job to another, but the artifacts were huge and we were not fully convinced by this solution. Then we thought about running our CI jobs inside a previously built docker with the build already shipped in it. Doing so would permit to reduce again the gap between testing and execution, and it seemed pretty funny.

A simple pipeline building an image and running a test inside looked like this :

image: $CI_REGISTRY_IMAGE/ci:$CI_PIPELINE_ID

[...]

variables:

GIT_STRATEGY: none

# this variable tells the runner not to clone the project

# it is not needed as we already got the code in the docker image build_image:

stage: build

variables:

GIT_STRATEGY: clone

services:

- docker:dind

image: $CI_REGISTRY/docker/base-images/ci/7.1

script:

- make install

- make build

# the above line basically does

# docker build -t $CI_REGISTRY_IMAGE/ci:$CI_PIPELINE_ID . run_test:

stage: test

script:

- make test

We chose to name the image after the pipeline ID as it allows us to start again a new pipeline from an old commit without interacting with the old pipeline.

In order to avoid shipping development dependencies in production we needed to have two images, one for the CI and one for production purposes.

The production-ready image was named after the commit-SHA. Using the docker layer model we extended the production images to add development and test tools needed during the CI. Now ARG is supported in Docker FROM so we don’t have to hack with sed to specify to our ci layer the tag of the production image.

First we build a production image :

docker build -f Dockerfile -t foo.bar/$CI_PIPELINE_ID .

# ./Dockerfile

FROM baseimage

COPY . /var/www/html

RUN pull dependencies please

And then we build a second image from the first :

docker build -f Dockerfile-ci -t foo.bar/ci:$CI_PIPELINE_ID --build-arg PIPELINE="$CI_PIPELINE_ID" .

# ./Dockerfile-ci

ARG PIPELINE

FROM foo.bar:$PIPELINE

RUN pull those crazy heavy test dependencies

We chose to run some jobs only on master so we made two different build jobs, one for master and the other one for feature branches. This part heavily depends on the git workflow, here we’re using a sightly modified version of nvie’s GitFlow (master and develop are the same branch).

The result