In a recent engagement I’ve been challenged on how to stress test bots developed with Microsoft BotBuilder SDK and deployed to Azure Bot Service.

The objective was to load test the bots and assess not only that the request was sent but also the responses that the bot provides. We also had the demand to assess if the response was ok according to the scenario in context.

While this seems trivial like any web application stress testing, it’s not. Bots operate in fully asynchronous mode and integrate with another web application, which is the channel. This application translates from the Bot protocol to the channel protocol like Facebook Messenger, Telegram, Microsoft Teams and so on. Load testing a scenario like this requires more than just HTTP requests and responses.

The sequence diagram below shows how is the flow of messages between the channel and the bot and you may figure why a traditional stress testing approach will test only the first leg and not assess the full scenario.

A sequence view on the interactions Bot x Channel

Giving this asynchronous nature of bots and the fact that responses are returned as a POST from the bot to the channel, it’s not possible to stress test a bot simply using a stress testing tool. Traditional tools make HTTP requests and wait for responses and for bots we need to make requests and also accept requests containing the responses which come asynchronously. If we rely only in sending requests, we will be able only to stress test the steps 1 and 2 of our diagram and we won’t be able to measure the response latency nor assess if the response is correct.

Why a two parts article?

In this first part of the tutorial we will see how to setup the stress test and run it locally. In part 2 we will see how to run it against your bot hosted in Azure Bot Service and also run using the command line instead of JMeter UI. Stay tuned!

A toolkit to the rescue!

Giving the challenges described, I’ve developed a toolkit based on Apache JMeter to stress test the bots developed with BotBuilder SDK .

The toolkit can spin up an HTTP server to receive the responses as a POST and makes testing the whole cycle (request to response) easy like any other JMeter test. The toolkit is available here:

And the documentation describes how to install, create and run tests, what we will go step-by-step throughout the rest of this article.

Creating your first Bot Stress Test

The first step to stress test a bot is to have ah… bot! For this step-by-step tutorial, we are going to use a sample you can get cloning the https://github.com/Microsoft/BotBuilder-Samples repo. So just clone it by running:

git clone https://github.com/Microsoft/BotBuilder-Samples

Also, download the latest version of JMeter from https://jmeter.apache.org/. At the time of this writing it’s version 5.0.

You will also need Java Development Kit which you can get from https://jdk.java.net, Apache Maven which you can get from https://maven.apache.org/, the Bot Framework Emulator V4 which you can get from https://aka.ms/Emulator-wiki-getting-started. Node.js will also be required and you can get it from https://nodejs.org.

Build from sources

You need to build the stress test toolkit from the sources, so clone the repository from https://github.com/damadei/BotServiceStressToolkit by running

git clone https://github.com/damadei/BotServiceStressToolkit

Go into the BotServiceStressToolkit folder and run the following command (make sure Maven and Java bin folders are on your path):

mvn clean package

Now, copy the generated jar from the BotServiceStressToolkit\target folder to the lib/ext folder of your JMeter installation and start JMeter.

To verify if the toolkit is available, right-click the Test Plan element > Add > Config Element and check if the elements with the prefix Bot Service are there. If they are, it’s everything good so far. If it’s not, you probably missed some step along the way.

Checking if the plugin is available in JMeter

We need a bot

Go to the BotBuilder-Samples\samples\javascript_nodejs\04.simple-prompt folder inside the folder you cloned the BotBuilder samples.

Inside the folder, run npm install to install the packages and then execute npm start to start running your bot.

Bot running locally

Now connect to it via the emulator by opening the .bot file available in the sample root folder. Open the development endpoint inside the emulator.

You should see a welcome message. Type anything to start and then enter your name, that will be displayed at the end of the flow. This is the flow we will stress test later. The picture bellow shows the whole flow.

Bot conversation flow in the emulator

Creating the same scenario in JMeter

In JMeter, the first step is to right click the Test Plan element and select Add > Threads (Users) > Threads Group. This will add a Thread Group element to our test plan. The thread group is responsible for configuring the number of threads which represent the number of concurrent users and/or calls our stress test will make, it’s also responsible for configuring things like the test duration time, number of runs per thread, behavior in the occurrence of errors and so on. More info here.

Bot Service Configuration

After we have a thread group, right click it and add a Add > Config Element > Bot Service: Configuration.

It will look like the following:

Bot Service Configuration

We can leave most of the fields with their default values, however it’s important to understand what each of them mean:

Bot URL : is the endpoint where your bot is deployed. If your bot is running locally, the value will probably be http://localhost:3978/api/messages

: is the endpoint where your bot is deployed. If your bot is running locally, the value will probably be Callback URL : is the callback URL so your bot can return messages to JMeter. This is the address a server that JMeter spins up and should be http://localhost:callback-port if running locally.

: is the callback URL so your bot can return messages to JMeter. This is the address a server that JMeter spins up and should be if running locally. Channel Id : id of the channel emulated by JMeter. If the value specified is emulator , Bot allows calls without security enabled. If any other value specified, security configuration is mandatory and the Security Configuration element should be included in the test. Keep it as emulator for now.

: id of the channel emulated by JMeter. If the value specified is , Bot allows calls without security enabled. If any other value specified, security configuration is mandatory and the element should be included in the test. Keep it as emulator for now. From (Default Member Id) : id of the user making requests. If item Gen random user id per thread is used, this value is ignored.

: id of the user making requests. If item is used, this value is ignored. Gen random user id per thread : generates a new user per thread so that there’s no conflict between requests with the same user id. It’s recommended to keep this value checked to avoid concurrency issues.

: generates a new user per thread so that there’s no conflict between requests with the same user id. It’s recommended to keep this value checked to avoid concurrency issues. Recipient (Recipient Member Id) : id of the recipient for the message. Will be default-bot most of the times .

: id of the recipient for the message. Will be . Callback Server Listen Host : hostname or ip to bind the callback server to. If empty, callback server binds to all network interfaces what is enough in most of the times. Remember that responses are sent as callback HTTP POST messages and this is what this and the next properties are for.

: hostname or ip to bind the callback server to. If empty, callback server binds to all network interfaces what is enough in most of the times. Remember that responses are sent as callback HTTP POST messages and this is what this and the next properties are for. Callback Server Listen Port : Port of the callback server started by JMeter to receive callback messages. Defaults to 45678. If changed, you have to change the callback server URL to match the port.

: Port of the callback server started by JMeter to receive callback messages. Defaults to 45678. If changed, you have to change the callback server URL to match the port. Response Timeout: Time (in seconds) to wait for a response before considering the test step as failed because no response was received. Once again, remember that responses are asynchronous and are received by a callback HTTP POST made from the bot to the JMeter internal server that we spin up. This timeout is the amount of time we wait for such messages with the same correlation id before considering that no response was received and failing the test.

Conversation Update: Starting your conversation with the bot

After the Bot Service: Configuration element was added and configured, it’s time to add a sampler called Bot Service: Configuration Update.

This sampler is the one responsible for starting a conversation with our bot and can optionally receive one or more response(s) which is/are generally welcome message(s).

To add the sampler, right-click on Thread Group > Add > Sampler > Bot Service: Configuration Update.

Adding the Conversation Update sampler to the test to start a conversation with the bot.

After you have added it, move it after the Bot Service: Configuration element.

As said, this sampler is responsible to start the conversation with the bot and it’s comprised of the following fields:

# of Responses Expected: Number of responses expected after the conversation is started/updated. If your bot welcomes user with one message, for example, you should configure this field as 1, if you send 3 messages, put 3 or if your bot stays mute after the conversation is started, leave it as zero. If you put a number bigger than the number of responses sent by the bot, JMeter will keep waiting for a response that will never arrive, will timeout and your test will fail, so take care on putting the exact number of responses you send. Each response is an “send activity” call inside your bot code.

Number of responses expected after the conversation is started/updated. If your bot welcomes user with one message, for example, you should configure this field as 1, if you send 3 messages, put 3 or if your bot stays mute after the conversation is started, leave it as zero. If you put a number bigger than the number of responses sent by the bot, JMeter will keep waiting for a response that will never arrive, will timeout and your test will fail, so take care on putting the exact number of responses you send. Each response is an “send activity” call inside your bot code. Members added: Members which were added to the conversation, separated by comma. If you want to add the current user which is random and attached to the thread (and which you don’t know the id as it was generated at runtime), the special syntax ${user} can be used.

For our test, leave the Members added with default values and change the # of Responses Expected to 1 because our test welcomes the user with the following:

Welcome message sent by the bot on when a new conversation is started

View and understand what is going on through Listeners

We are at a point that we can validate this piece of our test and at least receive the welcome message. For that, would be nice to see what is returned and also check if it was success or error. This is done in JMeter via the Listener kind of element, so let’s add two of them: Results Tree and Aggregate Report.

Right-click Thread Group > Add > Listener > View Results Tree. This listener will show each request in a tree. Make sure it’s added as the last element. Repeat the process and add an Aggregate Report listener.

After adding both elements and positioning them, your test should look like the following:

Run test, run… Validate the test running it for the 1st time

Make sure your Bot is running or start it again via npm start. Click the Run menu, then Start. Wait for the test to finish execution which can be seen by the top right icon turning from green back to gray.

Click on the View Results Tree element and we can see the one request we did via the conversation update sampler. It should be black indicating it succeeded. If it’s red, an error occurred and you should review the steps so far and look for something you missed along the way.

Click the Response Data tab and check we received the correct response text as part of the response. This shows our whole round-trip succeeded (if you forgot to change the number of responses expected from 0 to 1, the RESPONSE item will not be present and the test will still succeed).

Response received from the bot

Validating the response

Now, it’s time to validate if the response is what it should be. For that we have a component in JMeter that can assert the response and it’s called guess what? Assertion.

Right-click the Bot Service: Conversation Update element > Add > Assertions > Response Assertion.

Change the pattern matching rules to Contains and enter a piece of the response in the Patterns to Test field.

Response assertion

Run the test again and it should work fine, proving we are assessing the response correctly. Force an error in the Patterns to test field and run it again and the test will fail, proving the response assertion is working.

Sending and receiving messages

The next step in our journey is to send messages and receive responses (which are also messages). To add a message activity, right click the Thread Group > Add > Sampler> Bot Service: Message.

Adding a Message sampler

Move the element we just added to after the Bot Service: Conversation Update and before the View Results Tree element and rename it to Say Hi. Your test should be looking like the following:

Test structure now

This is the step of the conversation where we are asked to send anything to the bot, so enter Hi in the text field and keep the rest as is. Interesting to notice that as we have a single callback response, we should keep the # of Responses Expected as 1 so we wait for one response.

Configuring the message to be sent

Run the test again and see we get a valid response from the message, asking “What is your name, human?”

We could add an assertion here as well to validate the response but we won’t so we can make the tutorial shorter. Do it by yourself to practice your JMeter skills.

Inspect the response in the View Results Tree element.

Response received for the message we sent

Now let’s add a new message sampler to send your name which is what the bot is asking for. Name it Send Name and place it after the Say Hi element. Run the test again and you will see the following:

Whole flow tested

If you remember correctly, this is the last step of our conversation. We can now move forward and add load.

Adding Load

So far, we have a stress test with one user that executes the conversation flow only once. This is not what we expect from a stress test.

To add more users and iterations, click the Thread Group element and change the Number of Threads (users) to 5 and the Loop Count to 100. This way we will have 5 concurrent users and each user will run the flow 100 times, so we will end with 500 executions.

Configuring number of threads and iterations

Verifying

When we run our test with load, it’s not feasible to use the View Results Tree to inspect item by item. It’s better to have a more synthetic view and that’s when the Aggregate Report comes into play.

Inspect it after the execution and you will see it provides the average, median, percentiles and a bunch of other kind of information about our bulk execution.

Aggregate report showing the execution of our test

This is the last step of our tutorial. We know that running a stress test locally is not the best option and in part two you will learn how to run it against a bot deployed to Azure Bot Service.