(Thanks to Michael Bolton for structural editing and for helping me develop this concept, which grew out of our recent testopsy. Thanks to Frances White and Marius Francu, on the RST Slack forum, for their comments.)

To test well is to find the bugs that matter, assuming that those bugs exist (and we always, always, begin with that assumption). These bugs begin in the darkness. We bring them into the light. We do this by operating the product in all the right ways. I sometimes feel that the bugs are stuck in a box, and I am shaking that box; banging on it like someone who just lost a coin in a soda vending machine. Notice I said that I feel that. I certainly don’t think it, and I rarely say it, because it makes testing seem like a brute effort instead of a thoughtful design process that is worthy of smart people like us. (But yeah, it can feel like I am the gorilla in that famous luggage ad. Come out you buuuugs!!)

Let’s call that the “kinetic” model of the test process. There is truth to that model, because we can ask “why don’t the bugs just fall right out when we want them?” The answers to that are interesting and deep. The product, the tester, and the test process itself— all these things together— can be thought of as a sieve with different filters that can block the good bugs from being reported. Part of testing well is to keep those filters unclogged. But they are not just filters, they are also steps in a chain of evidence. When you report a bug you are reporting an unbroken chain of reasoning that spans “this is a problem that matters” back to “this exists.”

Assuming we’re talking to the developer, the logic of this process goes something like this:

You’re grateful to know about this bug, now that you’ve got it, because I reported it after determining that it is a genuine bug, following my investigation of it, starting when I noticed it soon after it appeared, having triggered it by a method I now know how to perform at will, which became possible only since this bug was left in the product by you.

That is one convoluted sentence, isn’t it? Bear with me. Since it is both a sieve and a chain, let’s call it something that evokes both ideas: a bug reporting pipeline. Let’s look closely at it and how it can go wrong:

To be correctly reported, a bug must exist.

To report an actual bug, the bug must first actually exist. By “bug” I mean something about the product that threatens its value in the opinion of someone who matters. That means the word “bug” is a broad term that encompasses faults, failures, annoyances, and even enhancement opportunities.

What does it mean to say that a bug exists, even if users haven’t seen it, yet, or never will? Start with the first principle: a bug is always a relationship between a product, people, and context. As people learn more about the product, that relationship becomes ever clearer, and they become happy with the product, or dissatisfied. Now imagine that all the people who matter were to have god-like, perfect knowledge of the product. Then, whatever they would agree is wrong with the product would be the “bugs that exist” at that point in time. That’s what I’m talking about. Of course, you can’t know for sure if any bug exists before you encounter it, but the existence of the bug doesn’t depend on your knowledge or what you see— it depends on the product itself, the people who matter, and the context in which they operate.

The work of testing is to explore the status of those things.

For a bug to ever affect anyone, it must be producible.

The bug must be able to “happen.” Producibility means that there must be some way of manifesting behavior that leads to the trouble that the bug will cause. In the case of a code fault, “producing the bug” means running the code in such a way as to trigger some sort of failure.

Developers and managers can get pretty tetchy about the producibility of bugs. Sometimes they will tell you not to report anything that you can’t reproduce on demand. Push back on that attitude! Maybe try this speech:

Of course, in any non-trivial situation I will try to provide steps to reproduce, but what if I try for a while and don’t find a way to reproduce that bug? And what if it turns out to be a very important bug? And what if, had you known about the one time that I saw it happen, you could have realized what the problem was even without reproducing it? If I spend too much time trying to reproduce it, I might spend too little time in other testing. If I don’t report it at all, I’m not even giving you a chance to fix it. I think if such a bug gets released, you will not come back to me and say “we are suffering terrible embarrassment in the field, {insert your name here}, but don’t worry because you correctly followed our policy of only reporting obvious bugs.” How about, instead, we say that reproducibility of a bug is desirable, but not essential? I propose: it shall be our policy that anyone who makes a good faith report of any trouble shall be heard, and not attacked. Bug reporters shall make a reasonable effort to report clearly, concisely, respectfully, and in enough detail to be of use to developers and management. It shall be our policy to welcome any such reports, whether or not they are reproducible (or even actual bugs), because it is better to have occasional bad reports than to create a hostile atmosphere that discourages even one good report.

(I realize this is too long to be an elevator speech, but remember that an elevator speech is mostly for people who don’t care about you.)

Can a bug exist and yet not be producible?

Yes. You can have a fault in code that is unreachable. This might still matter in the case that future development of that product activates that code. The only way to find bugs in unreachable code is through static analysis (meaning methods that do not involve running code).

Can a “bug” be producible and yet not exist?

Yes. Because you might see some behavior and think it’s a bug, then later realize that the behavior was actually correct.

For a bug to come to your attention during testing, it must be produced.

The bug must “happen.” In testing, as distinct from code review (or any form of static analysis), you must manifest the observable behavior of the bug. This you do through operating the product under the right conditions. Good testing is a process of systematically covering the various surfaces of the product (functions, data, platform configurations, interfaces, etc.) so as to maximize the chance of producing every bug that matters.

Can a bug be easily producible and yet not produced during testing?

Yes. You might be testing in a way that is quite different from the way real users use the product. What they see readily, you might never encounter in the test lab.

Can a “bug” be produced and yet not producible?

Yes. You might trigger a bug using input or a system configuration that will never be available in realistic use. Or you might not be testing the correct version of the product.

For a bug to come to your attention during testing, it must be observable.

You must be able to see it. Observability is a key part of testability. Detailed log files and access to back-end interfaces can greatly improve observability. Obscure interfaces and heaps of data stuck in JSONs reduce it.

Can a bug be produced and yet not be observable during testing?

Any bug that is produced is in principle observable, but it might not be practically observable to you given your tools and methods at the time it occurs. Perhaps the bug corrupts a database, but you have no means of checking the entire state of the database.

Can a “bug” be observable and yet not produced?

Possibly. Your eyes and memories are not perfect. You may fall prey to optical illusions or misfiled memories such that you believe you see something that later is proven never to have been present. Testers need to appreciate the frailty of human perception.

For a bug to come to your attention during testing, it must be observed.

You must actually see it. It is not a given that you will see a bug just because it can be seen. In my own experiences videoing and replaying testing, I have been shocked at how easily I miss bugs that were right in front of my eyes.

In the case of automation, you are obliged to be narrow and specific in your observations, because you have to write code for everything you want to observe. Humans are less reliable at making any given explicit observation, but far more able to notice unexpected things—things no one told you mattered, but that you realize are important as soon as you see them occur.

Can a bug be observable and yet not observed during testing?

Easily! One of the things that can reduce the chances of seeing a bug is the belief that there are probably no bugs to be found. It’s very important, as a tester, to convince yourself that bugs are likely to happen.

Another dangerous phenomenon is known as inattentional blindness. This happens when you are trying to pay attention to one particular factor (perhaps looking to see if the titles of the windows are correct), and then you mind automatically ignores other factors (perhaps some numbers that don’t add up) that may be obviously wrong. This is one reason why following a detailed test procedure can cause you to miss important bugs: script following tends to turn off your thinking.

To maximize what you observe you can test in pairs, perform tests more than once, video your testing so that you can play it back, don’t test when you are sleepy, or at the very least: be aware of how easily you can miss important bugs that you see only once.

Can a “bug” be observed and yet not observable?

Possibly. For the same reasons given above: your eyes and memories are not perfect. Also, the very mental state we need to be sensitive to bugs—the belief that they are likely—may also trick our minds into seeing something that is not there. For instance, as an arachnophobe, I have had the experience of seeing spiders out of the corner of my eye that turned out to be smudges on the wall, or a piece of fluff on the floor. Still, my brain would rather that I be too sensitive to spiders rather than not sensitive enough, and I agreed with my brain about that.

For you to make a compelling report, you have to comprehend the bug.

You must think you know what’s happening, to some reasonable degree. You don’t necessarily have to know all of what’s going on. But you strive to understand enough to be able to describe it, localize it, and know how to produce it at will.

Can a bug be observed yet not comprehended during testing?

Easily! You may see any given output, plain as day, and yet not understand why the output was produced, or how it could have been produced. Too many testers, too often, have only a shallow understanding of how their products work.

Can a “bug” be comprehended and yet not observed?

Yes. You might guess, or infer, that a bug exists just based on your knowledge of the design of the product, or on your memory of bugs in similar products. This is called risk-based test design. That comprehension may lead the way to producing the actual bug, or maybe you will find that the bug you feared does not actually exist.

For you to know that the behavior is actually a bug, you have to authenticate it using a good oracle.

You must have reason to believe that the behavior or attribute you are looking at is in fact wrong. Some things might look wrong at first and turn out to be right after further review. Some things might look wrong to you and yet be right in the minds of the people who matter. This is called the “oracle problem.” How do you know it’s a bug?

As a tester you are an agent for the people who matter. But it’s probably not economically viable to get all of those people to watch you test. So that means you must internalize how they think and what they want. Of course, you may have written requirements or some other detailed reference to work from, but often—for many kinds of technologies and for many kinds of bug—you won’t have definitive oracles. This can be an especially difficult problem to solve if you are automating, because you literally cannot encode all aspects in a which a product can misbehave, and you will drive yourself crazy trying.

The bottom line for bug reporting is you must be able to explain why the bug is a bug, and also why it’s a bug that matters.

Can a bug be comprehended and yet not authenticated?

Yes. The ordinary way this can happen is that the process of comprehending the “bug” taught you that it was correct behavior. What you must guard against is dropping a real bug because you are using weak oracles, or the wrong oracles.

Can a “bug” be authenticated and yet not comprehended?

Yes. If you misunderstand or mis-characterize some output, it is easy to see how you might think it’s a bug. For instance, if a certain report is only created in “subscription plan mode” and you see that it is produce in “basic plan mode” that would be a bug—unless it was in fact in basic plan mode and you had gotten your browser tabs confused.

For your client to find out about the bug, you must report it.

You have to tell the story of the bug. There are many ways of reporting bugs, from formal (a Jira ticket with steps to reproduce) to extremely informal (“hey look at this!”). In your reporting you decide that you have the right level of knowledge, what forum in which to offer the report, what form it should take, what sort of supporting material is needed, etc.

Can a bug be authenticated and yet not reported?

Too often! This can happen if you feel intimidated by an angry development team or managers. Maybe it’s very late in the project and you don’t want to make them upset. It can also happen if the bugs are being found so quickly that you lose track of some of them. We call this the “money booth” syndrome.

Can a “bug” be reported and yet not authenticated?

Yes. This can happen if there is a disagreement about the oracles but you elect to report anyway on the chance that it matters. Or it can happen if people who aren’t committed testers are looking at the raw results that come out of automated checks. What often happens in this latter case is that management shrugs and says “oh these are probably just glitches in the automation.” (No project I’ve seen works by letting machinery “report bugs” without the intervention of human judgment, unless it’s something slam dunk obvious, like a crash or safety violation. Even then it’s not really a bug report, it’s a trouble alarm that hasn’t yet been investigated. The value that human authentication brings— when done properly— is that you believe a report you didn’t write is definitely worth your time to read.)

For your client to appreciate the bug, they must receive the report.

Your client must “get” it. Your reports must land with them. This is a social and cognitive problem, and I’m afraid it has a lot to do with your personal credibility. You must build that credibility and protect it. That process of building credibility can take months. Destroying your credibility doesn’t take long at all.

The reporting usually begins before there’s a formal report, in the sense that you discuss your findings with the developer before you go to all the trouble of writing them up.

When a team member says to me “we don’t need testers, because we can get feedback directly from real users” I ask them “Have you ever seen feedback from real users?” Because you know— if you have seen comments about quality written by frustrated outsiders who are not being paid to get along with you— that it can be quite difficult to read and process those messages. As a tester, I enjoy reading it and find it useful. Most developers have little patience for it.

Can a bug be reported and yet not received?

Yes, especially if there are a lot of reports, or if they come in at the eleventh hour. They can overwhelm developers and management. This is why you must organize the reports well, categorize them, and especially give each report a short and compelling title. You often need to follow up to make sure it doesn’t get lost in someone’s inbox.

Can a “bug” be received and yet not reported?

Possibly. This can happen when developers or management watch users in action, and see directly the frustration they experience. That’s one reason why getting groups of users to experience the product in the presence of the product team can be such a powerful experience. When I do that, I don’t ask those users to write any reports, I have specialists do that. But in that case the formal reports are just icing on the “cake” of direct observation by the people who matter.

For your client to do anything about the bug, they must think it matters.

We want to find bugs that matter. If you report bugs that don’t matter, testing will not be seen as an important process. You will not feel valued. Now, of course, you don’t control whether there are important bugs in the product. That is entirely outside of your scope. The developers create the bugs, or else the customers do, with their changing wants and needs. But there are probably good bugs in your product. In any case, you should have faith that there are.

If you test in a risk-based way, you will orient yourself to find the bugs that, if found, probably will matter.

Can a bug be received and yet not matter?

Yes. Lots of things can be bugs in a technical sense but don’t really impact the user. These can range from cosmetic problems, to minor performance problems, to behaviors that simply were not exactly what was in the original spec.

Can a bug matter and yet not be received?

Yes. In other words, a bug may do damage to your company or your customers without you becoming aware of it.

Alas, bugs frequently get into the field, in our industry. DevOps keeps wanting to push software out and feel that the testing process is slowing them down, perhaps assuming that no problem can be all that bad. Agilists preach that “everyone should test” (which means, at best, that everyone tests poorly and no one really takes responsibility for it). Technologists preach that “all testing should be automated”, which means that no testing is automated (because testing is not a thing that machines can do) and what does get automated is a smattering of narrow output checks that look like testing to people who don’t care about testing. So, of course, many bugs get in to the field. But whether they really matter is a question for your industry, your customers, and your level of risk acceptance.

For people who take testing seriously, we want to find every bug that could matter. And when we fail, which we often do, we learn from that and improve ourselves— the way automation can’t and amateur testers won’t.