My Opinions

First of all, I want to set one thing straight: I believe that the reasoning behind Property-Based Testing is sound. As Fred puts it…

[With Property-Based Testing] you’ll be able to write simple, short, and concise tests that automatically comb through your code the way only the most obsessive tester could.

Writing good properties and letting a good framework check them against your code will undoubtedly test your code much much better than any unit test you (or the most obsessive tester) can ever write. I don’t think anybody can deny that.

But…

What stuff should be tested this way?

Is it worth it to test all your code with properties? I don’t think so. That’s the basis of my thesis project problem: I tried to use QuickCheck to test everything (including the GUI). That was not a wise choice, for various reasons:

Property-Based tests are slower than unit tests. This is by-design since each property is tested a multitude of times instead of using just one example.

Writing good properties is not easy. Fred states that multiple times in his book. Sometimes, a good unit test is not hard to write even if it covers just one possible scenario, but writing a property that captures that same thing in such a general way that you can have 10000 different instances with which to test the system is much more complex.

Writing enough properties is not easy. In the same way that you can hardly ever be sure that you wrote enough unit tests to really cover all the functionality in the module/component/system you’re testing, it’s hard to be sure that you wrote enough properties for it. Fred does a great job of showing this situation in his book when he describes the different types of properties you can write, particularly when you’re not verifying your system behavior against a model. But you should also be wary of adding too many properties since each property takes time to run and you don’t want your suites to run forever…

So, what is worth testing with properties? I believe Fred summarized this idea quite well in this podcast when he said:

It comes down to figuring out what you want to write a test for first […] With Property-Based Tests this is really really hard because you need to find a general rule, but if you have something so stupid simple that the rule is the test itself you can not just do that.

[…]

You want to find something that is not trivial, but that you understand well and has significant complexity in its implementation. Usually data structures are interesting for that […].

From my perspective, there are things that are totally worth checking with properties and those are, in general, the ones used by Fred as examples in his book…

Data Structures

When you’re writing a module to manage a new complex data structure, like a new model for lists, a particular kind of trees, hash table, etc., writing unit tests will almost certainly miss a bunch of corner cases that property-based testing will not. Writing tests as properties will also help you define your module better and come up with a nicer API for it. Besides, as the book shows with list tests from OTP, property-based tests will use far less code, too.

Optimizations / Refactoring

One of the best scenarios you can find to use properties for testing is when you’re refactoring something that you can be sure that works correctly. Let’s say, you’re trying to optimize a certain function for performance or a module for readability, etc… Then you can use the previous version as a model and verify that your new implementation works exactly as the old one did. Writing properties will be at least as easy as writing unit tests (I think it will be easier since you don’t need to come up with the examples yourself) and it will give you much more confidence.

Libraries for Generic Processing

Much like data structures above, you can be writing a library like the CSV parser Fred presents in his book. Something where you don’t really know all of your use cases beforehand. Maybe you have a standard or RFC to guide you (That’s great, since encoding RFCs as properties is easier than coming up with general properties yourself) or some other properly written specification. Another thing that makes the use of properties easier is having complementary functionality (like encoding/decoding) so that you can write symmetric properties.

Complex Algorithms

This is, at least in my experience, the most common place to reap the benefits from writing properties: In most of your systems, at some point, you’ll have to write an algorithmic piece that sits at the core of your system’s logic. You will likely won’t get this for free from a library and it might involve multiple data structures and/or some complex pieces. In his book, Fred presents the checkout code kata as an example of this. When your algorithm has to respond correctly to some parameters that may vary widely with quite a few edge cases, writing properties instead of unit tests definitely pays off. I would personally still write a unit test for the shrunk value produced each time the properties found a bug in my system, but that’s just me.

Complex Stateful Systems

Then you have Stateful Properties, which is a great way to test your system as a whole when it has multiple API endpoints that can be executed in different sequences and conflict with each other. So, if you’re writing a system where, as stated in the book…

“what the code should do” — what the user perceives — is simple, but “how the code does it” — how it is implemented — is complex.

…then you’ll get a lot of benefits from writing Stateful Properties. But if your system is big but not complex (like your usual CRUD HTTP server) or if “what the code should do” is really hard to model, maybe not so much.

What about your Systems?

Now the question is: How often do you write systems that are worth testing with properties?

How often do you create new data structures? While I work with opaque data structures almost every day at work, they’re generally flat (i.e. a map or a record with multiple fields and their accessors). I don’t regularly have to create a new type of trees or hash tables.

How often do you work on large enough refactoring/optimization tasks? Large /complex enough to merit adding properties to compare the new version with the old one thoroughly? Some people (I remember Hernán, for instance), may do that all the time… Me? Not so much.

How often do you write new generic libraries? I actually used to do that a lot at Inaka and I believe many of those libraries would certainly benefit from property-based testing, indeed. On the other hand, in scenarios like the one described by the book, where you find yourself writing a system that needs to parse a CSV file… well, I don’t think I will ever face that requirement and go “OK. I have to write a generic CSV parser now.”. I will either try to find a library for that or write just the code to do exactly what my system needs (i.e. parse a 3-column/0-peculiarities CSV file in the case of the book).

How often do you write complex algorithms or whole stateful systems? This actually happens more often than any of the stuff in the previous paragraphs, it happens to me at least once per system, maybe more. Of course, this only includes relatively small pieces of the systems I build, but I do believe that using property-based testing for them would be a nice addition.

Final Notes

In conclusion, as I see it: Property-Based Testing is a great tool that you should seriously consider using for those places where its benefits over traditional example-based tests outweigh its drawbacks (basically time consumed writing and running the property tests). But, as with any other tools, it shouldn’t be the only tool at your disposal and therefore I would not advocate for Properties-Driven Development.

If/when you decide to start using Property-Based Testing, you should totally read Fred’s book. It will guide you through the process and it will make it smooth and enjoyable. It won’t make you an expert in writing properties, but it will get you as far as a book can take you.