Testing, over the years, has become an integral aspect of software engineering, and for a good reason: testing is how we know our software systems meet their specifications and perform as expected.

Various kinds of tests are carried out by software engineering teams around the world and these tests do fall into a couple of widely accepted categories.

Unit tests are the most common among software engineering teams as they help teams easily ensure a particular piece of code does what is expected of it and has no anomalies. Integration tests help software teams ensure the entire system functions as expected in unison.

Other kinds of tests such as snapshot testing, UI testing, etc, are usually carried out by certain teams, such as UI teams.

These tests could be carried out by teams before, or after code has been written, depending on what development approach the team decides to adopt. However, the software is always subject to change and it is important that the software works as it should after the change is made. This forms the essence of regression testing.

Regression testing ensures a change to a program has no adverse effects on existing features. It involves the use of already executed test cases to ensure existing features continue to work as expected and new code doesn’t introduce side effects into the system.

Regression testing could be carried out by rerunning all or selected parts of an already existing test suite. This could be a cumbersome process, depending on the number of tests to be rerun and in this article, we would explore a way of easily automating regression tests on REST APIs for staging and production environments of monoliths and microservices.

Considerations to be Made In Properly Running Regression Tests On APIs

Changes in software systems are often made to correct a defect, improve the quality of an aspect without resulting in a behavioral change, adapt to a new environment, and avoid future problems without resulting in a change in behavior. All of these changes apply to APIs and also require regression testing.

When performing regression testing on APIs, it would be nice to take a few factors into consideration. Firstly, we would want to know how our API would perform under an expected workload. It would also be nice to know if our API would be read or write-heavy, the expected number of concurrent clients and if we expect most responses to be cacheable. Once these questions have been answered, we could then proceed to create a test suite.

In this article, I would be using a demo E-Commerce API I created with Node.js and TypeScript which features a few methods for product retrieval, upload, and removal, as shown below. Typical real-world E-Commerce APIs would also include CRUD (Create, Read, Update, Delete) endpoints to handle orders, merchants, ratings, authentication, search, reviews, etc.

These logical business sections would each experience varying read/write ratios, which depends on the company’s business model and what matters the most to its customers.

A common way of carrying out regression tests on these APIs would be to create a test script that tries to replicate a typical session with the API client. Next, the 99th percentile (the slowest 1 in 10,000 requests) and error rates are determined for different concurrency levels. This gives an idea of the time a resource takes to render and the number of machines required to support a certain amount of users.

In this article, we would explore an easy way of carrying out regression tests on APIs without much hassle.

Let’s Get Started

I would be using Postman to test our E-Commerce APIs during development. Postman provides a rich environment for API development and I’ve used it on almost every API project I’ve worked on.

In the postman sample shown above, we have a set of API endpoints that allows us to make CRUD actions on the products of our demo E-Commerce API. We could make API requests to DELETE, UPDATE or GET a particular resource by using the resource’s ID, provided by our MongoDB database in this case.

With the addition of unit tests, this could prove to be enough while working in a development environment. But what about our staging and production environments? As earlier discussed, we need to be able to easily ascertain the state of our APIs after changes have been made and also ensure that they perform acceptably under varying amounts of load.

In order to be able to accomplish this, we would be using a platform called Loadmill to automatically test our APIs. It is quite simple to use and has a free tier which allows us to try out the platform.

Creating tests on Loadmill is pretty straight forward. After creating an account on the platform, we create a new Test Suite and a new Flow. These Flows host multiple requests which are automatically fired in sequential order.

Before creating any requests, we should store all variables needed for most requests, such as the base URL, or a valid JWT Token for requests requiring authentication.

After that, let’s create a Flow to automatically test or product API endpoints as in Postman above.

For the POST request, we define the request body and URL. The document ID found in the response can be saved for future requests as Loadmill allows us to extract values from our JSON response and store them as parameters for consecutive requests.

Next, we create a request to GET all products

Quite easy! Notice we also set our Content-Type header to be application/json for all routes. Lastly, we enter the endpoint to obtain a single product.

If one or more regression tests fail, there is a need to determine if the change to the software is faulty, or if the regression test itself is broken.

After creating all tests in our test suite, we run the test suite to perform our regression tests which in this case ensures our APIs perform as expected.

Load tests could also be performed through the rich user interface provided which enables us to add requests which could include configurations such as the number of iterations, cache penetration, and email notifications for a successful or failed load test.

Clicking on the Run Test Button opens up a modal which asks us to choose the maximum concurrent sessions and the period for which the test should run

Next, the test begins execution and we can see the Performance Over Time and Throughput in realtime!

From the Performance Over Time chart above, we see at a particular point in time, 50% of our API requests were executed under 17ms. This is commonly known as the 50th percentile and sometimes abbreviated as p50.

Looking at higher percentiles tells us how bad our outliers are, hence the reason for the 95th percentile.

As shown at the point highlighted in the chart above, 5 out of 100 requests took 439ms or more to execute. Higher percentiles are important to companies because they directly affect a user’s experience of the service, even though only 1 in 1000 requests might be affected. This is because the slowest requests are often made by customers with the most data on their accounts, hence they’re considered to be more valuable customers.

These high-value customers need to be kept happy and prevented from moving over to the competition. However, optimizing for these higher percentiles can be expensive and not so beneficial to a company.

Clicking on individual requests in the Configuration section shows the error rate, failures and successful attempts for that request endpoint.

Seems pretty easy, isn’t it?

I hope I could offer a glimpse at regression testing and certain considerations when performing regression tests on APIs.