Sometimes you find bugs in the most surprising places.

I've long used test-driven design for code I care about. Knowing that it works, works reliably, and will continue to work helps me write the right thing. As I've gained experience with testing, I've come to appreciate the nuances of how and what to test and why. For example, I often don't bother to test exploratory code—it's more important for me to learn how code might work to help me figure out how to design the real thing than it is to hew to a rigid mindset of "You must always test everything."

Similarly, I see little value in rigorously testing basic accessors, and thank goodness for Moose for turning class declarations into declarative code.

In more serious discussions of testability and testing, people often argue that testing UIs is impossible, or merely difficult, or rarely worth the trouble to do so completely. I agree to some respect; the most effective way I've seen to test UIs is to test the backend data model rigorously, write a very thin controller to marshall and unmarshall requests and responses, then to write a handful of tests which explore the edge cases of UI interactions. That requires a deep understanding of what happens for each interaction, and you can only test those interactions which produce measurable side effects.

Let that latter point sink in for a moment.

My most recent work on multiple projects includes a lot of embarrassingly parallel batch processing work, where the user interface is largely static, produced daily from a series of templates. Generally the templates are correct if they pass the eyeball test with live data.

This works except when it doesn't.

Consider a snippet of Template Toolkit code to display a list of stories:

[% FOR e IN entries %] <div class="story story[% loop.count %]"> <h3><a href="[% e.url %]">[% e.title %]</a></h3> <p>[% IF e.author %] [% e.author | html %]<br /> [% END %] [% f.format( e.posted_time ) %]</p> [% IF e.has_image %] <img src="[% e.image.full_file %]" alt="[% e.title | html %] image" class="image" /> [% END %] <p>[% e.short_content | html %] <a href="[% e.url %]">Read More</a></p> </div> [% END %]

That code had a bug. Can you find it by eyeballing it? Neither could I, for a long time. I'll give you a hint: it never displayed images associated with stories. Give up?

I wrote backend test after backend test, fixing a couple of other bugs along the way, then traced through the code in a debugging batch until I had convinced myself that the data model was completely correct. The bug had to be in the template because it could be nowhere else.

Rigorous template tests didn't seem to help either, and I'd almost convinced myself that I had a nasty spooky action at a distance bug somewhere in the data model before I saw the problem:

[% IF e.has_image %] <img src="[% e.image.full_file %]" alt="[% e.title | html %] image" class="image" /> [% END %]

When I added a test for the has_image() method on story objects to the data model tests, it failed immediately because that method did not exist. It used to exist, but a refactoring somewhere went too far and removed that method and its tests from everything except the template. Because Template Toolkit is not so strict about the distinction between method and property access, the lack of a method is no big deal. It fell back to property access on a hash and, finding no such value in the hash, returned a false value.

There are ways to fix this. I'm sure there's a plugin for Template Toolkit, and probably a debugging/diagnostic mechanism somewhere I haven't yet discovered. Yet the problem remains.

I've heard it said (Rich Hickey? Martin Odersky? I think it was Rich.) that every bug in your program has passed the type checker. That statement contains wisdom. Yet it's also true that every bug in your program has passed the test suite. (We assume you run with type checks and keep your test suite fully passing.)

Neither is a substitute for careful thought and understanding. Both protect you only as far as you can take advantage of them.

My next post will explore some options for making these kinds of errors easier to catch way before the point of weird debugging sessions.