It’s really surprising how many people believe that code coverage and test coverage are the same thing. I don’t know where this confusion has stemmed from, but from scouring around the internet, it seems to be a common challenge that people switch between code coverage and test coverage interchangeably, probably subconsciously too. They are not the same. Let me use one of my son’s toys to explain…

My son, Angus, has a push/walker toy that he’s had for a while. It’s really helped him get his balance – he’s confidently running around at speed now at 17 months old no doubt this push toy has helped him with that.



Anyway… The push toy has some shaped holes cut out on the top and sides, and comes with blocks that fit in the holes. He loves this toy, and from watching him playing with this toy over the past few months, I’ve realized that it provides me with a perfect example for explaining the differences and subjectivity between code coverage and test coverage.

The toy

Here’s a picture of Angus’ toy:

Angus’ push toy with block shapes to insert into the holes

From the picture you can see the red rectangular block. That red block fits into the rectangular hole on the side of the toy.

Using this toy as an analogy, if we look at the rectangular hole as relating to code that represents a feature, and the red block representing data that a user can input relating to that specific feature, then pushing the block through the hole would essentially cover that code. It could be seen as 100% code coverage. We’ve pushed the block through the hole, therefore we’ve exercised 100% of that feature’s code.

But, if we look closer at the block, you’ll see that there is actually 16 different ways that you could pass this block through the same hole.

16 different ways to insert the block

The 6 edges of the red block…

The block has 6 sides, which I’ve labelled in the image above. And the block can fit into it’s hole from each side, in multiple ways:

Top side facing up: Edge 1 side being inserted into the hole Edge 2 side being inserted into the hole Edge 3 side being inserted into the hole Edge 4 side being inserted into the hole

Bottom side facing up: Edge 1 side being inserted into the hole Edge 2 side being inserted into the hole

Edge 3 side being inserted into the hole

Edge 4 side being inserted into the hole



But in addition to this, the block can be passed in each of these ways in two directions too:

Outside in (meaning you push the block through the hole so it lands inside the toy)

Inside out (meaning that you put your hand inside the toy and push the block through the hole so it lands outside of the toy on the floor).

This gives us a total of 16 different ways that you can pass the block through the hole that it’s made for.

So in terms of tests, with there being 16 known tests now, that means our single initial test that we mentioned above that potentially gave us 100% code coverage, actually only has a testcoverage of 6.25%.

And here’s another catch, this time with test coverage: test coverage percentages only relates to your known test ideas. There are 16 tests that we have thought of so far, so you might think running all 16 known tests gives us 100% test coverage… But there are inevitably more tests that we haven’t thought of yet.

Take the toy for example. It has other holes that have other shapes. And some of these other shapes also fit through the previous rectangular hole. From multiple sides too…

The cube hole at the top of the toy

See the square hole on the top of the toy? The cubed block that fits in that square hole also fits into the rectangular hole…

That cube block means that there is a further 48 possible ways to pass the second block through the rectangular hole.

You can see clearly that coverage percentage metrics are extremely subjective here, and purely releate to that snapshot in time based on what information is clearly known. They don’t in fact tell you anything about the quality of your software, or the quality of your testing.

If you and I were testing the exact same feature, at the same time, for the same duration of time, in isolation – our test coverage number will more than likely be different because we’ll think about and uncover different test ideas, and we will both have a level of unawareness of other unknown test ideas that we didn’t think about due to our different past experiences, beliefs and biases.

And if by fluke, we did end up with the same coverage number, it definitely doesn’t mean our testing is the same – our testing will be completely different. This situation would simply obfuscate the problem and the subjectivity further, and it nicely highlights again that coverage metrics don’t tell us about the quality of our application or the quality of our testing, or indeed what has been tested.



There is one thing that code coverage is useful for

Code coverage can be useful though – it tells you about areas of the application which are not being covered at all by any assertive tests. And this is a risk and acts as an invitation to investigate those untested areas.

So the percentages might not matter, but the information regarding what code isn’t being executed at all is a useful heuristic.

Do you use code coverage and test coverage metrics? How are you using them? I’d love to hear more about why or why not, and if you find them valuable. Leave a comment below!