Posted by Gavin Love on Aug 04, 2014 Running PHPUnit tests in parallel with Jenkins and Ant

At MyBuilder we have always believed that having a good test suite is the key to fast development, happy developers and happy customers.

When we finished our migration from Symfony 1 to Symfony 2 in February 2013 our entire test suite was taking about 18 minutes to run, but as each month went on and we added more tests this got longer and longer until october 2013 when our test suite was taking around 38-40 minutes. We spent far to much time waiting for tests to run during development, we got frustrated, our productivity dropped, this was clearly unacceptable.

A faster jenkins server

The first thing I did was order a new server to run our Jenkins builds on, We replaced our old 2 x 4 core @ 2.20GHz, 16GB RAM with a new 2 x 20 core @ 3.00GHz beast with 96GB, the fastest CPU’s they could give us. Unfortunately there was a shortage of these CPU’s as they had only came out 3 weeks before our order so we had to wait for it to be delivered.

PHPUnit accelerator

While waiting for the new server to arrive I did some research into speeding up PHPUnit tests. This lead me to Kris Wallsmith’s faster PHPUnit article which I quickly implemented on our test suite and immediately noticed a 4-5 minutes reduction in the time taken to run the test suite.

Given we had never heard of this technique before I decided we should share this with everyone and we created a small helper and phpunit-accelerator was born.

Setup and Configuration

Assuming you already have composer installed simple run

./composer.phar require mybuilder/phpunit-accelerator:~1.0

Then just add to your phpunit.xml configuration

<listeners> <listener class= "\MyBuilder\PhpunitAccelerator\TestListener" > </listener> </listeners>

This was good, a whole 6 minutes knocked off the time it takes to run our entire tests suite, but we were still taking nearly 30 minutes, clearly that was not good enough.

Our new server arrives

After a few weeks our new Jenkins server finally arrived, however, it only knocked 3-4 minutes of the time it took to run the entire suite.

As we suspected faster cores was not going to be the answer to our problem nor would it allow us to grow our test suit going forward. Since we now had 40 cores the obvious course of action to speed up our test suite was to run the tests in parallel…

GNU Parallel

We initially looked into using GNU Parallel after we discovered this line in Symfony’s travis.yml

ls -d src/Symfony/*/* | parallel --gnu --keep-order 'echo "Running {} tests"; phpunit --exclude-group tty,benchmark {};'

However this did not work for us, we had problems with segfaults and random test failures.

We tried to adjust the number of threads and that seemed to reduce the problems but after lots of digging around we could not figure out why. So we decided to try another way.

Jenkins and Ant

Since we already used ant and jenkins to run our continuous integration we looked into ways to leverage those tools. We discovered ant-contrib and it’s for task which seemed to be exactly what we needed.

To use the ant-contrib tasks you will probably need to tell ant where the contrib file is located by adding the following to the top of your build file.

<taskdef resource= "net/sf/antcontrib/antcontrib.properties" > <classpath> <pathelement location= "/usr/share/java/ant-contrib.jar" /> </classpath> </taskdef> This was on debian using apt-get to install, your location will probably be different.

The first step is to build a list of the directories that contains your test suits. To do this we used the ant dirset which allows you to include and exclude file paths.

We use Symfony and all of our tests within an app are in bundles so this example finds all directories with a Test directory in src, you can customise this to find any directories you want.

The threadCount parameter limits the number of tests Ant will run at the same time.

<target name= "phpunit-parallel" > <for param= "directory" parallel= "true" threadCount= "4" > <dirset dir= "${basedir}" > <include name= "src/**/Tests" /> </dirset> <sequential> <antcall target= "phpunit_run_by_directory" inheritall= "true" > <param name= "testDirectory" value= "@{directory}" /> </antcall> </sequential> </for> </target>

For each directory we execute phpunit_run_by_directory passing in the full path to the directory to be tested.

From the passed in ${testDirectory} we extract last two parts of the directory name into junit.name so we can output the test results into build/logs/${junit.name} so Jenkins can read the test results for each directory.

<target name= "phpunit_run_by_directory" > <propertyregex property= "junit.name" input= "${testDirectory}" regexp= "(\w+)\/(\w+)\/Tests$" select= "\1-\2.xml" casesensitive= "false" override= "true" /> <exec dir= "${basedir}" executable= "${basedir}/bin/phpunit" outputproperty= "testOutput" resultproperty= "exitCode" > <arg line= "-c ${basedir}/app --log-junit build/logs/${junit.name} ${testDirectory}" /> </exec> <echo> TestSuite ${testDirectory} finished: </echo> <echo> ${testOutput} </echo> <fail message= "TestSuite '${testDirectory}' failed." > <condition> <not> <equals arg1= "${exitCode}" arg2= "0" /> </not> </condition> </fail> </target> Our `phpunit.xml` is located in the app folder, you will need to change it to your location.

We also use composer to include phpunit in our projects bin directory

You now need to configure your Jenkins build to read the jUnit files that were written into build/logs . Simply change your publish xUnit pattern to be build/logs/*.xml as shown below.

Once you have gotten this all setup you can tweak the number of threads to get the best performance for your setup.

Results

So after a brand new server, adding our new accelerator and running our tests in parallel we finally got our build down to around 8 minutes in total. For us this is within the acceptable ten minute window but it doesn’t give us much room to grow.

We are now being more careful with our tests, keeping them quick with less functional tests and more unit tests.