Performance testing

What do you do when you want to check the system to see how it can perform, respond and scale under some simulated traffic? You can prepare performance tests.

There are many different types of metric that you may want to check and a range of tests that you can execute, for example:

Load testing — checking how well a system performs under expected traffic generated by virtual users

Stress testing — checking upper limits of system capacity

Soak testing — checking how a system behaves after long period of usage by virtual users

Configuration testing — how changes made to parts of the system influence its responsiveness

and many more tests that can simulate different scenarios. I’ll concentrate on load tests.

Load tests

Load testing is part of performance testing. It is designed to check how the system behaves under concurrent load of virtual users. As a result, you will have some metrics that describe how the system is performing. For example, if you are testing a REST API, you will know average response times of called endpoints. The tricky part of load testing is to get information about bottlenecks. Unless you add more monitoring and run different kinds of tests, you won’t know if the cause of potential problems is a database, code, hardware or network layer. However, load tests should answer the question of whether the system can handle the desired load and how it can scale

There are different tools that can be used to create load tests. JMeter and SoapUI are the most popular. In this article, I would like to introduce you to Gatling. In Gatling, tests are written in Scala. It uses Akka actors to be able to efficiently execute the prepared simulation and scale it to the desired load. It also uses the concept of session to maintain state between calls.

The plan

I want to create a simple load test. I’m going to use Jsonplaceholder as the server that I want to check.

We will simulate a user who is browsing through some posts. He will open some of them, read comments and add comments of his own.

In order to start, we will need Java 8, as Gatling supports only this version, and Gatling binary that you can find here: https://gatling.io/download/. I’ve created a Java project with dependency to Gatling which contains my scenario. You can find the project here https://github.com/antusus/gatling-posts.

First simulation

A test in Gatling is called a simulation. Extending Simulation class will give us access to tools needed to set up and execute tests. It will allow us to define actions, virtual users and the scenario that will be executed.

The whole code is in com.pragmatists.gatling.InitialSimulation and you can find it here.

The most important part is calling the exec function on scenario.

val scn: ScenarioBuilder = scenario("Initial Scenario")

.exec(http("Get all posts")

.get("/posts"))

Exec is using actions, that in most cases are requests. I’m testing REST API so I’ll be using HTTP requests. For the first step, I’m calling GET on /posts path. Gatling will automatically verify if the request is successful (response code is 2XX or 304). The name "Get all posts" defined in http creation will be shown in the report generated after successful execution.

I’m configuring HTTP protocol

val httpConf = http

.baseURL("https://jsonplaceholder.typicode.com")

.acceptHeader("application/json")

I’m executing requests against one API, so I’ve specified baseUrl . All requests that will not be an absolute http path will use baseUrl and the relative path defined in request. There are many options to configure in HTTP protocol and all are defined in the documentation. I’ve set default options which specifies that I want the API to respond with JSON.

With scenario and protocol defined, I can set up the simulation.

setUp(

scn.inject(atOnceUsers(1))

).protocols(httpConf)

I’m defining users. Right now, I’ll use just one user and later, we will have a look at how to define more interesting load strategies.

The last thing is to execute Gatling. If you applied configuration changes defined in the Github project, you can execute:

./bin/gatling.sh -s com.pragmatists.gatling.InitialSimulation

Gatling will run and should produce a report that will look like this:

====================================================================

2018-07-20 17:15:48 1s elapsed

---- Requests ------------------------------------------------------

> Global (OK=1 KO=0 )

> Get all posts (OK=1 KO=0 ) ---- Initial Scenario ----------------------------------------------

[##############################################################]100%

waiting: 0 / active: 0 / done:1

====================================================================

There will be also a link to the generated report. We will have a look at it later. For now, I can see that there was one successful request OK=1 and no failures KO=0 .

Using session to maintain state

In this part, I’m going to get posts and put them into session, so that we can later extract IDs and view some of them. The session concept is nicely explained in here. It’s a message that is exchanged between scenario steps, that are executed by Akka actors. It is a Map[String, Any] (where Any is similar to Object in Java). A session belongs to a virtual user, so that each user data is separated. You can add data to a session using feeders or extract values from requests.

The code can be found in com.pragmatists.gatling.DefaultSimulation and you can find the file here.

I’ll start with getting all posts:

exec(

http("Get all posts")

.get("/posts")

.check(jsonPath("$[*]")

.ofType[Map[String, Any]]

.findAll

.saveAs("posts"))

)

You can see that after getting posts, I’m converting the response into a map. Calling saveAs will store posts in the session of one virtual user. I can use it later. To read data from the session, you can use expression language or session expression:

exec((session: Session) => {

val postsMap = session("posts").as[Vector[Map[String, Any]]]

println("============ Sample post from list ============"

println(postsMap(0))

session

})

This is a session expression. It takes the session and it must return it. You can modify the session, but remember that it is immutable, so any modification will return a new session object. I’m getting all posts from the session, converting them and I’ll print the first post. After running the simulation, I’ll see:

============ Sample post from list ============

Map(

userId -> 1,

id -> 1,

title -> sunt aut ...,

body -> quia et suscipit...)

Saving response parts in sessions and later printing them is a way of debugging with Gatling. Calling session(“posts”) will retrieve an optional object from the session stored under the key “posts” . Now when I have posts stored, it’s time to open some of them.

Using Expression language with Session

Gatling comes with a simple expression language. I’ll use it to access posts stored in a session in order to get a random post.

exec(

http("Get one post")

.get("/posts/${posts.random().id}")

.check(jsonPath("$.id").saveAs("one_post_id"))

)

The expression ${posts.random()} will return a random element from the indexed collection. Next, I know that the post is a map. To get ID of a post, I’ll need to call $posts.random().id . I’m storing the opened post ID in the session. Next, I’m going to simulate reading comments for the opened post:

exec(

http("Read comments of post [${one_post_id}]")

.get("/posts/${one_post_id}/comments")

)

I’m using the expression even in the http request name. It will be shown in the report as a separated entry. The last thing is to add a comment of my own.

Using feeders to generate data

Feeders can be used to generate data for virtual users and store them in the session. They can be generated on the fly, read from a file and other places. You can read about feeders here. I’ll create a simple CSV file with some comments (you can find it in src/main/resources/feeders/comments.csv ). I’ll define feeder:

val commentsFeeder = separatedValues(“feeders/comments.csv”, ‘#’)

It will be a custom feeder with separation marked as “#” (I’m using # only for parser, so it won’t create more columns; I need only one column named body). Feeder is basically an Iterator[Map[String, T]] . My CVS file looks like this:

body

Lorem ipsum dolor amet...

The first row defines the keys of the map that will be created in feeder, while subsequent rows define values. I’ll add feeder to my scenario:

feed(commentsFeeder)

This step will add the same feeder for every virtual user. Every time the execution reaches this step (for different users), it will get a record from feeder and place values in the session. It is important to provide as many entries as users in the file.

In my example, the name of the session entry will be called body . To create a new comment using feeder:

You can see that I’m using ${body} in the POST request to retrieve data from the session using expression language. I’m also validating with check that the newly created comment has body with a value that I specified.

I would like to simulate opening three posts and adding a comment to them. I’ll use repeat function for that:

exec(repeat(3) {

exec(

http("Get one post")...

}

For each iteration, I’ll open a random post, get its comments, and add one of my own.

What is missing here is simulating the user looking at a post and comments. To do that, I’ll add pause:

pause(10 second, 17 seconds)

Pause will suspend execution of a scenario. It accepts either a fixed value or a range. In the latter case, the pause time will be random, but in an enclosed range.

Defining load pattern

For now, I was using only one user. It’s time to change that and simulate some traffic. It’s done through injection and you can read more about it here. There are plenty of options there, for example:

several users started at once — atOnceUsers

linear injection of number of users for some duration of time — rampUsers

users can be injected at a constant rate across time intervals, where the injection times are randomised — constantUsersPerSec

users can be injected from the starting rate to the target rate during intervals — rampUsersPerSec

I’m going to simulate two users starting, with three new ones showing up at random intervals. I want the test to run for 3 minutes (3 minutes ignoring pauses; with added pauses on my machine, it took almost 5 minutes for the whole simulation to complete):

scn.inject(rampUsersPerSec(2) to(3) during(180 seconds) randomized)

Reading the test report

Right after running the simulation, I can see in the console some basic information about request count.

---- Default Scenario ----------------------------------------------

[##############################################################]100%

waiting: 0 / active: 0 / done:422

==================================================================== ====================================================================

---- Global Information --------------------------------------------

> request count 4220 (OK=4220 KO=0 )

> min response time 44 (OK=44 KO=- )

> max response time 4020 (OK=4020 KO=- )

> mean response time 168 (OK=168 KO=- )

> std deviation 255 (OK=255 KO=- )

> response time 50th percentile 54 (OK=54 KO=- )

> response time 75th percentile 265 (OK=265 KO=- )

> response time 95th percentile 444 (OK=444 KO=- )

> response time 99th percentile 713 (OK=713 KO=- )

> mean requests/sec 14.552 (OK=14.552 KO=- )

---- Response Time Distribution ------------------------------------

> t < 800 ms 4184 ( 99%)

> 800 ms < t < 1200 ms 11 ( 0%)

> t > 1200 ms 25 ( 1%)

> failed 0 ( 0%)

====================================================================

There were 422 users. Each executed 10 requests. I can quickly spot if something went wrong. I can also see the distribution of response times in the response time ranges. Those ranges can be defined by you, and you can read more about reports here.