Step 1: Don’t write a test

Why Leaky Test Opinions Are Worth Avoiding

The Straw Man

Assume we’re writing tests for a Redux reducer function. This function will manage the state behind the retrieval of Github users.

The first test could assert that when retrieving users we should flip a flag so that the view knows to show a loading indicator.

Testing the retrieval state

Now let’s make it pass.

And we’ll add tests for when the data comes back successfully..

Testing the success state

..updates to the reducer..

..and the error tests..

Testing the error state

..similar updates to the reducer..

Let’s pause here.

If we continue down this road this approach is going to be more fragile for two reasons.

Firstly, we are essentially writing tests for a state machine, but we are making up those different states as we go rather than thinking about the problem holistically. As a result, we are writing opinions in our tests that aren’t well thought-out and leak into the implementation.

Secondly, the UserStore type being built is capable of representing a handful of future bugs.

What does it mean for activelyRetrieving and alreadyRetrieved to both be true?

What does it mean for users and error to be undefined, but alreadyRetrieved is true?

What does it mean for users to be a populated array while at the same time we also have an error?

While the above buggy states won’t occur now, there is no protection against our future selves or colleagues and we should write quite a few more unit tests to get that protection.

Is this really a straw man?

Do you think about the problem holistically when writing tests first?

Do you remember to refactor after you’ve written a few tests? What about your colleagues?

Would you refactor the three case-statements written above? If so, how?

Step 1: Identify your invariants

An invariant is a rule, or condition, that can be relied upon to be true during the execution of the program. A more succinct definition is:

An invariant is a logical guarantee about your application.

Invariants can be very low level. For example, libraries like seamless-immutable give you runtime guarantees that your data is immutable. Using a type checker like TypeScript gives you compile-time guarantees about types.

Invariants can also represent high level user experiences.

We can also get logical guarantees for high level concepts, such as guarantees for the user experience of a particular component.

For example, here’s a reusable tab panel that doesn’t do a very good job of upholding invariants relating to tab panel behavior.

Here’s the same reusable tab panel, but this time its implementation does a better job of getting the kind of guarantees that one might expect for a tab panel.

The latter example demonstrates that through careful data modeling and carefully designed interfaces, certain bugs can become literally impossible because the compiler will not allow them. This also means you don’t need to write unit tests for those situations that are guaranteed not to occur.

Going back to the Github users example, if we pause and think about how to model the problem before writing our first test then we can potentially prevent certain bugs entirely and make certain unit tests unnecessary.

If we rely on the compiler for invariants that can be guaranteed at compile time, then we don’t need unit tests to obtain those particular guarantees.

What guarantees do we want with the Github users example?

Rather than writing a test first, let’s stop and think about the invariants we want to have.

The Github users data is asynchronous data. As such, there are a few different states that represent the data:

Initial state

Retrieval state

Success state

Failure state

We can model this explicitly using Typescript’s discriminated unions (or disjoint unions in Flow or tagged unions in Elm).

Modeling the data in this way will give us a compile-time guarantee that the asynchronous Github user data will only ever be in one of these four states, and it will guarantee that any other code, such as a view function, must handle all four of these states.

In other words, the types defined above have made certain bugs completely unrepresentable and we are forcing consumers of this type to handle all four states. No more forgetting loading indicators. No more forgetting to handle errors.

Note that all of this is being guaranteed by the compiler before writing a single unit test.

Step 2: Write a [failing] test

Unit testing is still a necessity to get certain invariants. At a minimum, it’s required to uphold invariants that aren’t possible or practical to obtain at compile-time.

However, it’s important to note what unit tests are for. Unit tests are for upholding invariants, not steering the implementation.

In summary, writing tests first can give you some confidence about your code, but those tests may introduce undesirable opinions into the solution.

I’d assert that before writing any tests you might be better off asking “What invariants or guarantees can we get?”. Identify those guarantees. Identify which guarantees can be reasonably upheld at compile-time and model data and functions accordingly. Any invariants that won’t be upheld at compile-time can have unit tests written for them, and then implement a solution to pass the tests.