Today I would like to discuss continuous integration (CI). We will discuss specifically how we can use GitLab’s continuous integration pipeline to automatically build C++ projects. Let’s start with its definition:

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily — leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” — Martin Fowler

Note that the term continuous integration includes the word continuous, meaning that integrating code into the main code base, also known as the master branch, happens frequently. When we are working on a task that takes longer than a couple of days, it is probably useful to question the size of the task.

A side effect of working on a big task, that takes a couple of weeks to implement, is that integrating it into the master branch could be troublesome. Unless the task is isolated, it is likely that integration conflicts, also known as merge conflicts, will arise due to the changes that may have happened in the master branch while we were working on the big task in a separate branch.

One way to prevent these merge conflicts from happening is to frequently merge the updates from the master branch into the work-in-progress branch:

$ git fetch

$ git merge origin/master

Another way to prevent integration conflicts is to “simply” define tasks that are smaller in size, allowing integrations to occur more frequently.

Defining tasks that are smaller in size is strongly recommended because these can actually be code reviewed.

Unfortunately, code reviews are not able to catch all errors and that is why automated builds and tests are important. Nowadays, it is a widely accepted practice to use continuous integration to automatically build and test code changes before they are merged into the master branch.

An error that is not easily caught during code reviews is code that only works on one machine but does not work on other machines unless someone actually tests the new changes by pulling in the new changes locally. These kinds of errors are easily caught by automatically building the code base in GitLab’s continuous integration pipeline. Especially for C++ code repositories, this is important because we often rely on libraries that are installed using the system’s package manager, like APT on Ubuntu and Homebrew on macOS.

Problem definition

C++ code often relies on libraries that are installed through the system’s package manager. In this example, let’s assume we are building a C++ code base on Ubuntu 18.04, allowing us to install libraries with APT. We want to use GitLab’s continuous integration pipeline to ensure that the C++ code base builds on a fresh Ubuntu installation.

For this article, we assume that the C++ code base relies on Boost and VTK. The Boost library will be installed with the APT package manager and VTK 8.1.1 will be built manually because this version is not available through APT.

Note that this article is not about how to install and configure GitLab to support continuous integration. We assume that GitLab is configured correctly to support continuous integration.

Setting up continuous integration in GitLab

Setting up continuous integration in GitLab is as easy as adding a .gitlab-ci.yml file to the root of the project:

image: ubuntu:18.04 job:

- echo "Hello, World!"

Whenever a commit is pushed to GitLab, it will trigger the continuous integration pipeline and the jobs defined in .gitlab-ci.yml will be run accordingly. In the example above, a new Docker container is created from the Ubuntu 18.04 image and then “Hello, World!” is printed as defined in the job, called job . Note that we can define an unlimited number of jobs and a job must have a unique name and cannot use one of the reserved keywords.

Let’s make it work for the problem that was described earlier to ensure that the C++ code base builds on a fresh Ubuntu installation.

image: ubuntu:18.04 helloworld:

script:

- apt-get update

- apt-get install --yes

wget cmake build-essential

libboost-all-dev

libgl1-mesa-dev libxt-dev

- wget https://gitlab.kitware.com/vtk/vtk/-/archive/v8.1.1/vtk-v8.1.1.tar.gz

- tar xf vtk-v8.1.1.tar.gz

- cmake -Hvtk-v8.1.1 -Bvtk-v8.1.1/build

-DCMAKE_BUILD_TYPE=Release

-DBUILD_SHARED_LIBS=ON

-DBUILD_DOCUMENTATION=OFF

-DBUILD_EXAMPLES=OFF

-DBUILD_TESTING=OFF

- make -j8 -C vtk-v8.1.1/build install

- cmake -H. -Bbuild

- make -j8 -C build

- ./build/bin/helloworld

First, we install the required dependencies using APT, like build tools, the Boost library and VTK dependencies. Afterward, we download the VTK source code, and then build and install VTK 8.1.1. Finally, we build the C++ code base and run the helloworld executable that is produced during the build.

Now we are sure that the code will build and run on every Ubuntu 18.04 machine assuming the correct dependencies are installed. If we wouldn’t have installed libgl1-mesa-dev and libxt-dev in the CI pipeline, building the project would have resulted in an error message indicating that the OpenGL and X11 libraries aren’t installed. In other words, the .gitlab-ci.yml file forces us to explicitly write down the required dependencies for building the C++ code base.

Using Docker to decrease build time

The problem with the previous configuration is that installing C++ build tools, Boost and building VTK 8.1.1 altogether takes around 75 minutes. These dependencies are required in every commit and it is kind of silly to install these dependencies over and over again, while the part that we care about the most — building the project — only takes a short period of time during the whole CI pipeline.

A simple solution to decrease the build time is to build a Docker image that includes Boost and VTK, and push it to the GitLab Docker registry. From the command line:

$ docker build -t registry.gitlab.com/aaronang/gitlab-ci:latest .

$ docker push registry.gitlab.com/aaronang/gitlab-ci:latest

The Dockerfile would have the following content:

FROM ubuntu:18.04 RUN apt-get update \

&& apt-get install --yes --no-install-recommends \

wget cmake build-essential \

libboost-all-dev \

libgl1-mesa-dev libxt-dev \

&& rm -rf /var/lib/apt/lists/* RUN wget https://gitlab.kitware.com/vtk/vtk/-/archive/v8.1.1/vtk-v8.1.1.tar.gz \

&& tar xf vtk-v8.1.1.tar.gz \

&& cmake -Hvtk-v8.1.1 -Bvtk-v8.1.1/build \

-DCMAKE_BUILD_TYPE=Release \

-DBUILD_SHARED_LIBS=ON \

-DBUILD_DOCUMENTATION=OFF \

-DBUILD_EXAMPLES=OFF \

-DBUILD_TESTING=OFF \

&& make -j8 -C vtk-v8.1.1/build install \

&& rm -rf vtk-v8.1.1.tar.gz vtk-v8.1.1

Then, we can change the .gitlab-ci.yml file to the following:

image: registry.gitlab.com/aaronang/gitlab-ci:latest helloworld:

script:

- cmake -H. -Bbuild

- make -j8 -C build

- ./build/bin/helloworld

In this case, on every new commit, the GitLab CI pipeline will start a Docker container based on the Docker image that includes Boost and VTK. These changes will significantly decrease the build time because we no longer need to install Boost and build VTK on every commit, and can immediately start compiling and running our C++ code.

By building a custom Docker image and using it in the .gitlab-ci.yml file, the whole CI pipeline now only takes about a minute! In other words, this is roughly 75 times faster than the initial setup.

Conclusion

We have discussed how to set up GitLab’s continuous integration pipeline that allows us to automatically build a C++ project. Building projects in the CI pipeline guarantees that the project will be able to compile and run on every machine that uses a similar setup as the Docker container defined in the .gitlab-ci.yml file. Also, the .gitlab-ci.yml file forces us to explicitly define the required dependencies for building the C++ repository.

We have seen how we can speed up the CI pipeline by building a custom Docker image that includes the required C++ dependencies. In fact, we were able to reduce the CI pipeline time from roughly 75 minutes to roughly 1 minute!

However, as we all know, there is no such thing as free lunch. Managing and building Docker images comes with a lot of complexity and we should always question whether the advantages outweigh the disadvantages.

Feel free to reach out to me with comments, questions, and feedback. The code used in this article can be found on GitLab.