Testing is a controversial topic. People have strong convictions about testing approaches. Test Driven Development is the most prominent example. Clear empirical evidence is missing which invites strong claims. I advocate for an economic perspective towards testing. Secondarily, I claim that focussing too much on unit tests is not the most economic approach. I coin this testing philosophy Lean Testing.

Photo by Helloquence on Unsplash

The main argument is as follows. Different kinds of tests have different costs and benefits. You have finite resources to distribute into testing. You want to get the most out of your tests, so use the most economic testing approach. For many domains, e.g. GUIs, other kinds of tests than unit tests give you a better bang for the buck.

Confidence and Tests

The article ‘Write tests. Not too many. Mostly integration’ and the related video by Kent C. Dodds express the ideas behind Lean Testing well. He introduces three dimensions with which to measure tests:

Cost (cheap vs. expensive)

(cheap vs. expensive) Speed (fast vs. slow)

(fast vs. slow) Confidence (low vs. high or click doesn’t work vs. checkout doesn’t work)

The following is the ‘Testing Trophy’ suggesting how to distribute your testing resources.

Compared to Fowler’s ‘Testing Pyramid’ confidence as a dimension is added. Another difference is that unit tests do not cover the largest area.

One of Kent C. Dodds’ major insights is that you should actually consider the confidence a test gives you.

Return on Investment of Tests

The Return on investment (ROI) of an end-to-end test is higher than that of a unit test. This is because an end-to-end test covers a much greater area of the code base. Even taking into account higher costs, it provides disproportionally more confidence.

Plus, end-to-end tests test the critical paths that your users actually take. Whereas unit tests may test corner cases that are never or very seldomly encountered in practice. The individual parts may work but the whole might not. The previous points can be found in ‘Unit Test Fetish’ by Martin Sústrik.

Further, Kent C. Dodds claims that integration tests provide the best balance of cost, speed and confidence. I subscribe to that claim. We don’t have empirical evidence showing that this is actually true, unfortunately. Still, my argument goes like this: