Over the last few months, we’ve compared build testing of the Hack typeface between the free, open source account plans that are offered by the Travis and Semaphore continuous integration testing services. This testing has been part of an examination of our CI testing approach across the Source Foundry software repositories that include open source Python, Go, C/C++, and UFO typeface source software.

Why Test?

The Semaphore team reports a 50% reduction in test times for the elixir-lang/elixir test suite relative to times that were observed on the Travis CI and Circle CI testing platforms. Here is a screen capture of the data from the semaphore.com landing page:

Intriguing! As a long time user of Travis CI, I decided to put the test services to the test and find out if these benefits apply to our software testing needs. The following analysis compares the two testing services with the same git commits that were pushed to the Hack typeface repository and built simultaneously on both CI testing platforms via GitHub git commit hooks.

The Build

The build process involves the following series of steps:

Installation of Python typeface compile and post-compile font modification software with pip

Compile of the FreeType & Harfbuzz libraries and the ttfautohint executable from C/C++ source with a make-based compile workflow

Compile of four full character set font variants (regular, italic, bold, and bold italic) in desktop (*.ttf) and web font ( *.woff, *.woff2) formats from UFO version 2 typeface source files

Compile of four character subset font variants in web font (*.woff, *.woff2) formats from UFO version 2 typeface source files

Post-compile modification writes to the font binaries with instruction sets (i.e., hints)

Post-compile modification writes to the font binaries with other OpenType table fixes including DSIG table mocking, fstype permission modification

This build process has historically required approximately 4–5 minutes on the Travis platform. We perform parallel shell script linting tests that are not considered in this analysis as they are brief relative to our compile tests and do not impact the total time to test completion. The time-intensive tests for us are those shown in the list above.

The Testing Platforms

The Travis and Semaphore tests were both performed on Ubuntu 14.04 LTS systems. The git versions were 2.13.0 on Travis and 2.14.2 on Semaphore. The gcc versions were 4.8.4 on Travis and 4.9 on Semaphore. The Python versions were 2.7.13 on Travis and 3.6.x on Semaphore.

Test Results

Here are the data on the build execution times required for five recent sequential commits in our repository. These data exclude queue time and represent the software build testing execution time only. The builds were run on both testing services simultaneously via GitHub git commit hooks.

Raw build times by git commit

The following figure demonstrates the raw build time in seconds by git commit to the Hack typeface repository. The git commits are labeled with git commit SHA1 short hash strings on the y-axis.