Testing Python projects on additional OS

So, you’re testing your simple pure-Python package on every important major version of Python. You’re a responsible open-source contributor. Yay for you.

But what if your Python project is not a pure-blood, but a muggle containing some specialized C++ code? Or maybe your code is pure-Python, but it interacts with the operating system (e.g. writing files, juggling threads or processes, etc.) in a non-trivial way, that can differ between operating systems?

If that’s the case, you definitely should test your code (and possibly also build it) on all three major OS, if you want your project to support them.

For myself, the need arose on two projects, both pure Python:

The first is Cachier, a package providing persistent, stale-free, local and cross-machine caching for Python functions. I had to lock files for multi-thread safety when writing to them, and it turned out the built-in solution I had (using fcntl ) broke the first time a Windows user tried to use my package. The second is Skift, which implements scikit-learn wrappers for Python fastText. The implementation required writing and reading files in different encodings, which turned out to behave differently on different operating systems in some cases.

The solution I’ve settled on was expanding the Travis build matrix to include specific combinations of operating systems and major Python versions, each to be run in it’s own job, on a totally separate environment.

Again, when comparing this approach to using tox , I’ll say the main advantages are:

Offloading complexity and responsibility from you to Travis. Getting more accurate representations of real-life environments: pure Python installations of a single version directly at the OS-level, instead of through tox . This is how most users of small, open-source Python projects will install your code. One job == one OS version and one Python version. You can immediately see if a build failed because your tests fail on a specific Python version (e.g. 2.7 on all operating systems), a specific OS (all Python versions on Windows) or on specific combinations. This is extremely visible from the jobs view:

We obviously have a Linux-related build problem

Hopefully I’ve convinced you this is a valid approach to multi-OS testing, so we can move to the specifics. We’ll start with testing on macOS and finish with Windows.

Testing Python projects on macOS

At the time of writing, Python builds are not available on the macOS environment. This doesn’t mean it’s impossible to test Python on macOS with Travis, just that the following naive approach won’t work:

Whatever the version number you assign to the python key, you’ll get a macOS machine with Python 3.6.5 installed. This is because asking for a machine with os: osx spins up a machine using the default Xcode image, which is currently Xcode 9.4.1 for Travis.

The current hack-ish way to get a macOS machine with a specific Python version is to ask for a specific Xcode image, using the osx_image tag, which you know comes preinstalled with the Python version you want to use.

For example, to get a machine with Python 3.7 you can add the entry of osx_image: xcode10.2 (you’ll get Python 3.7.3, specifically). Cool. So how do you know which Xcode image comes with which Python version? Unfortunately, this mapping is not listed anywhere on Travis’ website or documentation.

Luckily for you, however, I did the dirty work and dug this information up. This basically entailed actively searching the Travis blog for posts on Xcode images releases to hunt down the Python versions on each image. The latest releases of major Python versions I have found are:

xcode9.3 — Comes pre-installed with Python 2.7.14_2

— Comes pre-installed with Python 2.7.14_2 xcode9.4 — Comes pre-installed with Python 3.6.5

— Comes pre-installed with Python 3.6.5 xcode10.2 — Comes pre-installed with Python 3.7.3

Unfortunately, I haven’t found a Travis Xcode image to come preinstalled with Python 3.5 (let me know if you do!).

So you got the right Xcode tag. You still, however, need to adapt some of the build commands. For Python 3 versions, for example, we need to explicitly call pip3 and python3 to install and call (respectively) Python 3 code, since macOS comes preinstalled with Python 2 (which is what the python command points to):

Considering this, you would have thought that a Python 2 job would require less custom entries. Unfortunately, because we’re using the OS Python, pip installation commands need to be appended with the --user flags for Python 2. Moreover, as a result their CLI commands won’t be installed, so we’ll again have to call their commands through the python command:

Good, we’re done with testing Python on macOS. Have a cookie.

A cookie

Testing Python projects on Windows

Travis support for Windows builds is in an early access stage. Currently, only Windows Server (version 1803) is supported. This doesn’t come with Python, but does come with Chocolatey, a package manager for Windows, which we’re going to use to install Python.

Since we are using Chocolatey to install Python, we are limited to the versions available through it. For Python 3, these are 3.5.4, 3.6.8 and 3.7.4. For Python 2, version 2.7.16 is currently the one installed by default.

Here’s a simple variation of a job entry to get a Windows-Python job, which includes the Chocolatey install command choco and an environment variable setup: