One of the aspects of traditional test management that has always troubled me has been the obsession with counting test cases at some places.

If you don’t understand what’s going on, if you don’t really understand what software testing is all about then start counting, and you’ve immediately got some sort of grip on well, er something.

100 is bigger than 10. 10,000 is pretty impressive, and 100,000 is satisfyingly humongous. You might not really understand what’s happening, but when you face up to senior management and tell them that you’re managing thousands of things, well, they’ve got to be impressed.

But counting test cases? Is that useful? Well it is if it’s useful for managing the problem, but it’s not an end in itself. It’s nonsense to expect all testing to be measurable by the number of test cases. It can even be a damaging distraction.

I was once interviewed for a test management role. I was asked about my most challenging testing problem. I replied that it was working as a Y2K test manager. It seemed like a good answer. It was a huge task. There was nowhere near enough time to do all the testing we’d have liked. The dates couldn’t be pushed back, and we were starting too late. We had to take a ruthless risk based approach, triaging some applications out of sight. They’d have to run over the millennium and we’d wait and see what would happen. The cost of testing, and the limited damage if the applications failed meant we had to forget about them and put our effort where it would count.

What seemed like a good answer was really a big mistake! “How many test cases did you have?”

I was surprised. The question made no sense. The applications were insurance management information systems. There was an on-line front end, but technically that was pretty simple. My responsibility was the huge and fearsomely complex back end processing. The strategy was to get all the code fixed, then hit the most complex and date sensitive areas hard in testing.

We were looking at batch files. We had a perfect test oracle. We ran the batch suites with 1996 data, and the non-Y2K compliant code. We then date shifted the input data forward to 2000 (the next leap year) and ran the Y2K compliant code. The 2000 output files should be identical to the 1996 output files in everything except the years. It was much more complex than that, but in principle it was pretty simple.

There was a huge amount of preparation, ensuring all the reference tables were set up correctly, allowing for any hard coded dates in the programs, setting up the data and test jobs. The investigation of the inevitable discrepancies was painstaking and time-consuming, but it worked. We got through the planned high-priority testing on time, and there were no serious incidents over the millennium. I explained all this. We didn’t have “test cases”.

My interviewers asked me to try and put a figure on how many test cases we would have had, as if it were merely a matter of terminology. Even if you multiplied all the files we were checking by each time frame in which we were running tests, the total number would have been under a hundred. You could have called them “test cases” but it wouldn’t have meant anything. I explained this.

“So, under a hundred”. Pens scribbled furiously, and I seethed as I saw all our challenging, sometimes quite imaginative, and ultimately successful testing being reduced to “under a hundred” by people who hadn’t a clue.

I wasn’t hired. It didn’t bother me. I’d lost interest. I could have just lied and said, “oh, about 10,000 – yeah it was pretty big”, but I wouldn’t have wanted to work there.

I’ve seen the obsession with counting test cases taken to extremes. The need to count test cases created huge pressure on one project to execute testing in a way that facilitated reporting, not testing.

It was another big batch financial system, and there were some strong similarities with Y2K. However, this time we had to have “test cases” we could count, and report progress on. We had to “manage the stakeholders”. It was important that test execution should show that we’d got through 10% of the test cases in only 9% of the testing window, and that everything was just dandy.

Sadly, reports like that meant absolutely nothing. Easy test cases got passed quickly, complex ones took far longer and we were running badly late – according to the progress reports and the metrics. The trouble was that the easy test cases were insignificant. The complex ones were what counted and there were many inter-dependencies between them. The testers were finding out more and more about the application and the data, and if all went satisfactorily there would be a rush of test cases getting cleared at the end, as eventually happened.

The reaction of senior management to the daily progress reports was frantic concern. We weren’t getting through the test cases. That was all that mattered. No amount of explanation made a difference. I was spending most of my time on reports, explanations and meetings; very little with the testers. Management thought that we were politically naïve, and didn’t understand the need to keep the stakeholders happy. Bizarrely, the people who knew almost nothing about the work, or its complexities, thought that we were out of touch with reality.

Reality for them was managing the process, counting the test cases, playing the organisational game. “The quality of the application? Whoa, guys! You testers are missing the point. How many test cases have you got?”