How do you run regression testing and maintain a high level of QA effectiveness when you have limited time?

Image credit: Pexels

It is always good to have a planned time margin before a deadline so that you can run through regression tests. But sometimes you need to give quick feedback on the upcoming release. How to manage such a situation, choose the necessary tests, and decide what functionality to test — these are the considerations we will examine in our new blog post.

Introduction

A Criticality-complexity Matrix (or just Criticality Matrix, CM) is a method that allows the classification of a software product’s functionality based on its business value and the complexity of testing.

The main goal here is to get maximum test coverage with minimal resources spent. The Criticality Matrix is also helpful in solving the task of reducing testing costs while keeping overall quality at a high level.

Criticality-complexity Matrix

There are three basic principles (“whales”) that lay at the foundation of the Criticality-complexity Matrix approach.

The process — here we study the functionality of the tested software.

Labor costs — this is one of the parameters establishing complexity. It is the answer to the question regarding how much time is needed to run a test. We use man-hours to calculate this parameter.

Criticality — how important the released functionality is for the business.

So, now we’re done with terms and definitions, let’s dive into how the matrix can look.

Doesn’t make much sense yet, does it? It’s better to study it with a real example.

A Criticality Matrix in real life

Suppose that our task is to test an online aggregation service, say, for an e-commerce website.

We can divide the whole website into modules (for example, personal account, search tab, item description), and prepare test cases for each of them. Then, we need to set criticality and complexity levels for each test case (here we will most likely need the help of business analysts).

Here is the result:

Note: during the project you will change the number of hours (i.e. the time taken to run one test, which is set up as an interval for a group of tests) and the average test time (which is the average time taken to run a whole analysis: it is set up for a group of tests, and used to build a Criticality Matrix using hours).

Here is the outcome table:

Note: You choose Criticality and Complexity from a drop-down list, while Group is set up automatically.

Also, we will have automatically updated Criticality matrices:

Criticality-complexity matrix (number of tests)

Criticality-complexity Matrix (hours)

Final thoughts

As can be seen from these examples, it is not that hard to build a Criticality Matrix. It is therefore better to discuss why it is important to do this in the first place. Here are some of the advantages to this approach:

The Criticality Matrix allows the division of testing into groups, which leads to increased speed in choosing those dedicated to the assessment of critical software functionality (valuable if you’re in a hurry).

It is a great tool, allowing you to understand the importance of testing. You can easily define tests covering primary functionality, and identify those which deal with those not-so-important features.

Using a Criticality Matrix enables understanding of regression testing time constraints.

Regression testing model development becomes easier, as we can clearly see the direction in which to go in.

More articles on software development and QA + useful links: