Python Programming, news on the Voidspace Python Projects and all things techie.

tox: Testing projects with multiple versions of Python

mock is tested on Python 2.4 upwards. Running all the tests is done with unittest2 test discovery, so up until now I've had several scripts for running the tests with different versions of Python. Not difficult, but a pain.

Enter a new project by the prodigious Holger Krekel:

To use tox you create a simple config file tox.ini specifying the test command and Python versions and tox will create a virtualenv for each Python version and run your tests in it.

I can now run all the mock tests for Python 2.4 - 2.7 on both Mac OS X and Windows with a single command (a single command on each platform - although tox does have Hudson support for continuous integration servers).

As well as running tests tox can execute arbitrary commands. This means it can run sphinx commands. For mock I have tox execute two Sphinx commands to build the documentation and run all the doctests (only for Python 2.6 & 2.7 as some of the doctests use the with statement). This way the tox run fails if there are any errors in the documentation, either reStructured Text errors or doctest failures.

A successful run looks something like this (output from Windows shown - with lots of it snipped):

C:\compile\mock > tox _________________________________ [tox sdist] _________________________________ [TOX] ***creating sdist package [TOX] C:\compile\mock$ C:\Python26\python.exe setup.py sdist --formats=zip --dis t-dir .tox\dist >.tox\log\0.log [TOX] ***copying new sdistfile to 'C:\\Users\\michael\\.tox\\distshare\\mock-0.7 .0.zip' _____________________________ [tox testenv:py24] ______________________________ [TOX] ***reusing existing matching virtualenv py24 [TOX] C:\compile\mock\.tox$ py24\Scripts\pip.exe install dist\mock-0.7.0.zip --d ownload-cache=C:\compile\mock\.tox\_download >py24\log\3.log [TOX] C:\compile\mock$ .tox\py24\Scripts\unit2 discover ..............................................................s................. ...s ---------------------------------------------------------------------- Ran 84 tests in 0.109s ... No builder selected, using default: html building [html]: targets for 0 source files that are out of date updating environment: 0 added, 0 changed, 0 removed looking for now-outdated files... none found no targets are out of date. ________________________________ [tox summary] ________________________________ [TOX] py24: commands succeeded [TOX] py25: commands succeeded [TOX] py26: commands succeeded [TOX] py27: commands succeeded [TOX] congratulation :) C:\compile\mock

Notice how the output from tox shows us exactly which commands are being executed. The test execution is done with test discovery using the command .tox\py24\Scripts\unit2 discover .

Here's the tox.ini for the mock module:

[tox] envlist = py24,py25,py26,py27 [testenv] deps=unittest2 commands=unit2 discover [] [testenv:py26] commands= unit2 discover [] sphinx-build -b doctest docs html sphinx-build docs html deps = unittest2 sphinx [testenv:py27] commands= unit2 discover [] sphinx-build -b doctest docs html sphinx-build docs html deps = unittest2 sphinx

This tells tox that when the tox command is run it is to create four different virtual environments with the four different versions of Python (so all these versions of Python need to be installed). For Python 2.6 and 2.7 there are custom command sets and Python 2.4 and 2.5 use the default [testenv] block.

Note Setting the PIP_DOWNLOAD_CACHE environment variable to a valid directory allows pip to reuse downloaded packages when it creates and populates virtual environments. It will reuse virtual environments anyway, but keeping the cache around can be a good way of avoiding unnecessary network traffic for package downloads.

The dependencies for Python 2.4 & 2.5 are just unittest2, for 2.6 & 2.7 Sphinx is also a dependency. The test command is, as we've seen, with unit2 discover . The [] after the command allows extra command line options to be passed through from tox to the test runner. For example we could use this to modify the parameters for test discovery:

tox -- -p test_signal\*

Although tox is in its early days its already useful for me to be able to run tests on four versions of Python, including testing the Sphinx documentation builds correctly and the code examples work, with a single command.

unittest2 0.5.0: setuptools compatible test collector and Python 2.3 distribution

unittest2 0.5.0 has just been released. This version of unittest2 has "feature parity" with the version of unittest in Python 2.7:

If you want to ensure that your tests run identically under unittest2 and unittest in Python 2.7 you should use unittest2 0.5.0. Later versions of unittest2 will include changes in unittest made in Python 3.2 and onwards after the release of Python 2.7.

unittest2 is a backport of the recent enhancements in the Python unittest testing library in Python 2.7.

Improvements in unittest2 over standard unittest in Python 2.6 and earlier include:

automatic test discovery from the command line

failfast, catch and buffer command line options

class and module level setup and teardowns

addCleanup for better resource handling

for better resource handling test skipping and expected failures

improvements to assertRaises and assertAlmostEqual

and many new assert methods, plus better failure messages

load_tests protocol for customizing test loading

protocol for customizing test loading various other API improvements and fixes

unittest2 is tested with Python 2.4 - 2.7. New in 0.5.0 is a distribution for Python 2.3 contributed by Mark Roddy.

Also new in 0.5.0 is a setuptools compatible test collector. If you put test_suite = 'unittest2.collector' in setup.py you can then find and run your tests (using test discovery) with:

python setup.py test

This starts test discovery with the default parameters from the directory containing setup.py, so it is perhaps most useful as an example (see unittest2/collector.py).

All Changes in 0.5.0

Addition of a setuptools compatible test collector (very basic). Specify test_suite = 'unittest2.collector' in your setup.py.

TestSuite.debug() and TestCase.debug() now execute cleanup functions and class and module level setups and teardowns.

No longer monkey-patch os.path.relpath for Python 2.4 / 2.5 so that projects don't accidentally depend on our patching. Contributed by Konrad Delong.

Added a Python version specific unit2 entrypoint. This will, for example, create a unit2-2.6 script if unittest2 is installed with Python 2.6. (Requires setuptools or distribute.)

Python 2.3 compatibility (in the python2.3 branch of the repository), contributed by Mark Roddy.

setuptools console script entry points are created as '.py' scripts on Windows.

Feature parity with the Python 2.7 final release.

Archives