When method calls are idempotent (they can be called many times without changing the outcome) it is possible that multiple calls will be made without the caller knowing. Additionally code that executes within a method but which does not significantly alter the outcome of the method may be executed without the caller’s knowledge. Some may claim this is correct and that the called shouldn’t need to know what goes on inside the method, and to a large extent they’d be right but there is critical information being lost here that may lead to bugs of a particularly insidious nature.

Imagine that we have a method that, among other things, updates a row in the database:

public void updateData ( String data , DatabaseWriter writer ) { writer . update ( data ); writer . update ( data ); writer . flush (); }

A traditional black box test for this might look something like this:

String data = "foobar" ; MyClass instance = new MyClass (); DatabaseWriter writer = new DatabaseWriter (); instance . updateData ( data , writer ); DatabaseReader reader = new DatabaseReader (); String dataAfter = reader . readData (); Assert . assertEquals ( data , dataAfter );

In this test we want to make sure that after we call the update method the database has been correctly updated with the new data.

This test passes and we are happy the code is working, but there’s an obvious problem. The “update” call on the DatabaseWriter was executed twice but because this call is idempotent the test does not know. Although the outcome of this is somewhat meaningless from a behavioral standpoint, the actual implementation has a bug which could prove disastrous. If this method is called several times (or millions/billions of times) the the overall performance of the system will suffer and this may be an extremely difficult bug to track down.

Conversely consider the mocked variation:

public void duplicateCheck () { String data = "foobar" ; MyClass instance = new MyClass (); DatabaseWriter writer = Mockito . mock ( DatabaseWriter . class ); instance . updateData ( data , writer ); Mockito . verify ( writer , Mockito . times ( 1 )). update ( data ); }

The first thing to note is that the test has considerably less code which is quite common when using mocks but the more important issue is that not only does this verify that the database “would have” been written to, but crucially that it would only happen once. This is something not typically caught by a more coarse black box test.

Similarly the execution of unexpected code can also be detected. In the previous example the call to “flush()” may not be required by the method but may in fact be a cause of significant performance delay. Again this may not be detected by the black box test as it does not have a meaningful impact on the outcome of the method, but would be detected by the mocked example as a matter of course.

There is of course a natural criticism of this approach in that asserting that (for example) the database “would have been” written to is not the same as asserting that it actually was written to but this is a case of appropriate separation of tests. We assume that the DatabaseWriter class in the above example is tested elsewhere and those tests verify that it does what it is supposed to do. Even in those tests we may not be testing whether the database was actually written to but simply following execution path up to the point that we no longer have control over the outcome. We don’t for example need to test that MySQL knows how to deal with an “UPDATE” SQL statement, we assume this works.