February 29, 2016 Javier Eguiluz

Transient tests are those which fail randomly depending on spurious and external circumstances, such as the underlying system load. These tests are very risky because they make your test suite unreliable.

Tests that deal with time-related functions are one of the most common transient tests. Consider for example the following test:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 use Symfony\Component\Stopwatch\Stopwatch ; class MyTest extends \PHPUnit_Framework_TestCase { public function testSomething () { $stopwatch = new Stopwatch (); $stopwatch -> start ( 'event' ); sleep ( 10 ); $duration = $stopwatch -> stop () -> getDuration (); $this -> assertEquals ( 10 , $duration ); } }

This code is so simple that it seems impossible to fail. However, depending on the load of the server, the $duration could be for example 10.00000023 and the test would fail for no apparent reason.

This kind of errors happen frequently when using public continuous integration services like Travis CI. We even have a long-running issue to hunt all these transient tests.