I suppose people usually stop early because most problems do not have a simple solution; thus they figure there’s no reason to spend any time looking further. If we start with the assumption that a simple solution does exist, we’re much more likely to find one.

On the other hand I have been reasonably successful at designing torture tests for software that I’ve written, mostly by

(1) imagining myself as the enemy of the system, rather than as its friend;

(2) thinking of inputs that are legal but bizarre and unlikely ever to be useful;

(3) embedding an incredibly complicated construction into another that’s even less scrutable.

Some parts of my test programs for TeX and METAFONT required many hours of thought before I could convince myself that the program did the right thing. But in the process I discovered bugs that I’m pretty sure wouldn’t have been found by any other method that I’ve ever heard about.

Even better results will presumably be obtained if several different people independently create the torture tests. I can imagine that test creation would be a satisfying career.

I guess I do tend to use systems in unexpected ways, and I get some satisfaction when this reveals a flaw. For example, I remember having fun while visiting my sister and playing with a `shoot-em-up’ game that my nephew showed me.

Since I’m kind of a pacifist, I tried shooting bullets at the wall, instead of trying to kill the hidden attackers. Sure enough, I could spell my name on that wall, by making bullet holes in an appropriate pattern. But later when I came back to that same position, after wandering around further in the game, my name was gone! I think that was a bug in the software.

Long ago when I was a teenager, probably in 1952 or 1953, I saw my first “computer game”, a demonstration of tic-tac-toe at a museum in Chicago where visitors were challenged to beat a machine.

The exhibit was designed by AT&T, and I think it was controlled by relay switches rather than by a real general-purpose computer.



‘Hmm. No Angry Birds here!’

(Don Knuth with an IBM 650 in 1958)

I knew that I could never win if I made normal responses; so I intentionally made the stupidest possible moves. The machine got to a position where it had two ways to beat me by completing three in a row. I decided not to block either possibility. In response, the machine made both of its winning moves … and this was clearly in violation of the rules. So I had won a moral victory.

Thus, it appears that an encouragement to “think outside the box” helps for designing test cases, just as it does in many other situations.