Previous in the series: https://itnext.io/types-and-specifications-c4d34ade6d5c

“We only see what we know.” — Goethe

In my last article about testing I introduced the concepts of QuickCheck using what is ostensibly an untyped language, clojure. While I make the case that specification based checking is superior to relying on type systems for building correctness you may find yourself in a situation where a typed language is what you have to work with. One of my favorite typed languages is the hybrid functional/procedural language golang. Golang has native support for QuickCheck but the way you use it is quite different than spec/check and it is not a fully functional implementation of QuickCheck such as you would find in erlang or Haskell. None the less this package allows developers to create a variety of fuzz tests for their work that with enough time and creativity can produce powerful generative tests.

Testing Framework: goConvey

Before I get into the code I want to introduce you to one of my favorite golang testing frameworks: goConvey. While golang native testing is wonderful it is quite procedural and it can lead to tons of conditional logic around t.Fail() or t.Fatal() lines. While the default testing is sufficient for small projects goConvey can make denser tests and as projects grow these dense tests can document your code better. Also, goConvey operates on functions not strictly on conditions which allows composition and data driven testing.

While this is covered in my “Playing the Wrong Game” post goConvey also facilitates TDD and code coverage by starting a local webserver to serve as a continuous build integration. While I urge you not to play the coverage game simply having a fast feedback on a browser indicating builds/tests passing automatically in the background creates a great continuous workflow. You can find it here: http://goconvey.co/

Collecting Tools

At the basic level testing/quick provides two tools: Check and CheckEqual. These functions accept either one or two functions and a configuration. Check is simply CheckEqual but the second function is the equivalent of “continuously true”.

While QuickCheck is always nice when it works the reason to use it is finding bugs and in order to find bugs we need feedback on the input that creates the bug. This information is captured within the CheckError and CheckEqualError interface. Casting the output Error allows us to see the count of calls, the input (expressed as a slice of interface) and in the case of CheckEqualError the mismatched outputs (also slices of interface).

Time Some Free Lunch

While the above is as simple as it can get for the sake of argument let’s do something real, but straightforward and pure. A pure function can easily be compared with a simplification of the function with CheckEqual. Back to date parsing, not because it is particularly interesting but because there seems to be countless ways to do it. We will compare a regex based parser against the built in parser in the time package to show how QuickCheck will magically do all our work for us and find a bug in the regular expression:

Checking goConvey I see this isn’t as easy as hoped:

The static tests that illustrate the bug passed but QuickCheck wasn’t able to find the very specific string that could find the error without some help. Increasing the iterations may eventually find the invalid string but given the unimaginable variation possible in a arbitrary length string doing so with more iterations is futile.

Bounded Randomness

What we need to do is give QuickCheck some help creating these inputs, just like we did in clojure spec. There are a few options for this. If your inputs are not simple types you can implement the Generate interface, or you can implement the “Values” function for quick directly. For this example I will chose the later.

I want to create the tests where the RFC3339 spec has some variations. The basic head of every string should be fairly consistent but there are a few timezone formats along with possible milliseconds/etc. I model this with a random valid date format but with “garbage” where I let random generation cover my laziness in reading the RFC.

With the TimeValues function defined we can use it in the “Values” portion of the quick.Config along with a bigger max iterations to be more confident we will find the culprit:

And as hoped QuickCheck found an invalid format for us:

Of course the date portion is not relevant but the missing “Z” for Zulu/GMT time was quickly found and I could use this for fixing the bug. Now it is straightforward to pursue this path further, possibly incorporating bigger runes that are unicode or altering the format but in general this shows how a single function can be checked.

Sequential Inputs

Generative testing of pure functions is helpful and it can certainly find corner cases that unit tests may miss. It would be nice, though, to test stateful systems with QuickCheck so that more complex bugs can be found. These are more like “component” or even “integration” level tests and QuickCheck in golang does give us a means of doing so.

Assume we have an API that we need to test. There are a number of possible endpoints/procedures available and we want to find any issue that occurs after arbitrary sequences of commands. The guarantee we want to make is that no sequence of commands will cause the database to enter a broken state. To illustrate this sort of is the following “API”

It has a handler and some state that will track if we run events 1 through 6 in order and generate an “error”. You can imagine that these integer lookups could be a key to a map of commands but for the sake of argument this is a very simple API that simply takes the integer values direct. We will also need some helper functions to wire things together:

The first is a quick way to get a remainder function that handles negative values correctly so we can use the slices of arbitrary int64’s QuickCheck will hand to us by default. The second is a function that takes an API, runs the events given in order and will return false at any time the API “IsBroken”. With these tools we can use quickCheck to find the sequence of events that breaks the API:

Which passes and gives us one of the sequences that break the API:

Which is not the ideal solution (the real QuickCheck would reduce the input to the smallest possible solution, golang doesn’t give this to us for free) but it is one of the solutions that “breaks” the API under test.

While this implementation of QuickCheck doesn’t go all the way to minimal failure sets it does allow customized fuzz testing of pure functions and as outlined can also run more stateful system tests when dealing with systems such as API’s, external databases, etc. Code examples can be found here:

https://github.com/weberr13/Kata/blob/master/gogen/gen_test.go

and as all things in that repo, are presented under the MIT license.

Next in the series: https://itnext.io/gopter-property-based-testing-in-golang-b36728c7c6d7