10 Minutes: Codeless Test Automation for IBM Watson Chatbots

TL;DR: With Botium, the Selenium for Chatbots, there is a tool enabling chatbot developers and testers to ensure consistency and correctness of IBM Watson Chatbots without a single line of code.

UPDATE 2020/06/15: As Chatbots grow in importance, automated testing solutions will remain critical for ensuring that Chatbots actually do what their designers intend. We’ve been busy working on a product that allows testers to have visual insights and deeper understanding in their Chatbot’s performance, offering several solutions to boost their interaction!

Botium Coach will be introduced to the market as part of our online event on the 24th of June.

Recently I’ve been working on a customer service chatbot based on IBM Watson Assistant (formerly known as “IBM Watson Conversation Service”) for a large Austrian telecommuncation provider. The chatbot was trained to answer questions on the website and to lead the user to the right website section. It currently handles 60k-80k conversations per months and covers ~25% of the customer service interactions. The IBM Watson Assistant workspace includes:

133 intents, 55 entities

438 dialog nodes

more than 3500 trained user input samples

It happened several times that minor changes in the dialog design or training caused previously working dialogs to fail — so we were in need of regression testing.

Even for a medium-sized chatbot it is not feasible to test manually. Botium, the Selenium for Chatbots, to the rescue.

With Botium it was possible to generate test cases from the IBM Watson Assistant workspace and setup automated testing within some minutes.

Step 1: Install and configure Botium

The only requirement for Botium is a Node.js installation. In a command line window download and install the CLI tool for Botium, and check installation by showing command line help:

> npm install -g botium-cli

> botium-cli help

The connection to the chatbot under test is configured in a JSON-file. Create a file called “botium.json”, containing:

{

"botium": {

"Capabilities": {

"PROJECTNAME": "whatever",

"CONTAINERMODE": "watson",

"WATSON_USER": "...",

"WATSON_PASSWORD": "...",

"WATSON_WORKSPACE_ID": "..."

}

}

}

Enter your IBM Watson user and password, as well as the “Workspace ID” — you can find it in the “View details” section of your workspace.

The user and password to use here are NOT the same you are using for logging into the Watson workspace. In most cases, you will have to generate service credentials. More info can be found in the Watson Assistant API Docs.

You can find out about additional Botium capabilities in the Botium Wiki.

Validate the configuration by running Botium in “emulator”mode — it will connect to your workspace and provides a simple chat interface in the terminal:

> botium-cli emulator --config path/to/botium.json

Step 2: Generate test cases

Next step is to generate the test cases from the IBM Watson Assistent workspace:

> botium-cli watsonimport watson-intents --config path/to/botium.json --convos path/to/directory/holding/test/cases

Botium will connect to your workspace and extract the intent resolution and example texts from there. There are test cases (in Botium speak: “convo files”) created for all your intents and utterances.

In order to execute all generated test cases, run Botium again in “run” mode:

> botium-cli run --config path/to/botium.json --convos path/to/directory/holding/test/cases

It will output progress information and notification on any failures.

Note: Using an additional command line parameter, Botium will output test results in standard file formats like JSON or XML.

Step 3: Verify test output

After 10 minutes and without a single line of code, we set up test automation for an IBM Watson Assistant chatbot.

We instantly discovered and documented a couple of issues in our chatbot workspace, resulting from failures in former training (user input resolved to unwanted intent), and it just took me 10 minutes.

We also added some eye-candy by letting Botium export a well formatted, interactive test report and publishing it to the developer intranet automatically, and we integrated easily into our continuous delivery platform (Atlassian Bamboo).

Conclusion

Without a single line of code and within 10 minutes, here is what we got:

3667 test cases generated from IBM Watson Assistant workspace

100% test coverage for intent resolution

8 serious issues detected and resolved immediately

Continuous testing pipeline component

[UPDATE: Export Real User Conversations as Botium Test Cases]

The Botium-CLI now includes an additional option to export real user conversations as recorded by Watson Assistant as Botium test cases.

> botium-cli watsonimport watson-logs --watsonfilter "response_timestamp>=2018-10-21, response_timestamp<2018-10-29" --watsonformat convo --config path/to/botium.json --convos path/to/directory/holding/test/cases

Running this command will export all recorded user conversations from the Watson Assistant workspace to an Excel file which can be re-used by Botium.

Using “intent” for option “watsonformat” will create a flat list of “date / last_intent / last_input / last_output” records, mainly for analysis purposes.

Watson Assistant only keeps recorded conversations for one month, afterwards they are removed. For getting information about how to build up the filter criteria (watsonfilter option), see here.

More on chatbot testing

If you want to get more into the topic of testing chatbots, there is an article series on Chatbots Magazine available: