Writing tests with good coverage and that work all dimensions of your code is time consuming, repetitive, tedious and subject to the law of diminishing returns.

When we start out as developers writing tests, we tend to take the approach of one test per case, and aggregate related tests into suites. For example, you might see a suite like this:

class TestMyCode(TestCase):



def test_first_case(self):

assert some_stuff()



def test_second_case(self):

assert some_other_stuff()



def test_third_case(self):

assert yet_more_stuff()

Each test case needs to carry out all its custom prep, execution and assertion stages. If these cases are closely related, this could result in a lot of code duplication (some of that can be mitigated in suite-wide setUp and tearDown methods, but by no means all).

This approach results in very large testing codebases, a huge amount of repetition, and a maintenance headache whenever things change.

There is a better way: parameterised testing.

Parameterised testing asks you to generate a single test which can take a set of arguments to configure the run. Imagine we have a reasonably simple function we want to test; something like:

def compute(a, b):

return (a + b) / (a * b)

To test this we want to hit a number of scenarios and see how the function behaves:

Test some positive numbers

Test some negative numbers

Does it work for long integers as well as regular integers

Does it work for floats?

What happens if either or both numbers is zero?

What happens if either or both arguments are undefined?

What if either or both arguments are strings or some other object?

If we did this the traditional way, we’d have a long list of tests for a slim function. If we parameterise, though, we can write a short test and feed in the arguments.

def test_compute(self, a, b, expected, raises=None):

if raises is not None:

with self.assertRaises(raises):

compute(a, b)

else:

assert compute(a, b) == expected

Here’s a selection of the parameter sets you could pass in. The real list would be much longer, depending on how exhaustive you want to be. In this example I’ve selected a good range for a and then some random pairings for b .

a | b | expected | raises

------------+-----+----------+-------

-2 | 2 | 0 |

0 | 2 | | ZeroDivisionError

2 | 2 | 1 |

0.5 | 0.4 | 4.5 |

None | 2 | | TypeError

"10" | "1" | | TypeError

3000000000L | 2 | 0.5 |

Things start out well here: we expect a ZeroDivisionError which is reasonable enough, and a couple of TypeErrors , plus a bunch of sensible responses. Perhaps, on reflection, we would change the function to do some better input validation, so we don’t have to deal with these error types.

Things get weird when we get to the final line in the table. We imagine that 3000000000 and 2 give us 0.5 (approximately, and we’ll come back to that in a moment), but if we actually plug that into our function, we get a 0 and our test fails. The problem is that since both a and b are integers, in Python division always results in an integer output, and 0.5 becomes 0. We’ve uncovered an actual conceptual problem with our function:

Should it always have a consistent output? Always an integer or always a float?

To what degree of precision do we provide the output? Because that 0.5 we expect is actually going to be 0.5000000003333334 .

How we fix this up isn’t really the point of this post, and if you want to have a go at making this function more coherent, drop your solution in the comments.

What is the focus is that using parameterised testing we were able to create a simple test case and quickly generate and pass in a number of difficult cases for the function to work over. As a result we uncovered a fundamental problem around the purpose and scope of the function in the first place.

While you could, of course, have reached this position with a traditional approach to testing, I would argue that parameterised testing pushes the way that you think about tests away from the code itself and into the argument value space. In this space it is easier to conceive of and combine edge and corner cases for really effective testing.