Testing a Chatbot — Getting Started

Getting started with chatbot testing — running conversation transcripts automatically, and introducing an open source tool.

UPDATE 2020/06/15: As Chatbots grow in importance, automated testing solutions will remain critical for ensuring that Chatbots actually do what their designers intend. We’ve been busy working on a product that allows testers to have visual insights and deeper understanding in their Chatbot’s performance, offering several solutions to boost their interaction!

Botium Coach will be introduced to the market as part of our online event on the 24th of June.

So you decided to test your Chatbot — great idea!

Apart from philosophical questions about human language itself and about 100% test coverage, building a testing strategy is hard work and takes time. This article is written by experience in real-life projects and shows the first steps on this journey.

Recording test cases with Botium Box Live Chat

Basic Assumption: Conversation ↔ Test Case

What we are doing here is “UI/UX testing”. A chatbot has a very simple, intuitive user interface — spoken or written human language. A test case in this context can be described as a conversation the chatbot should be able to follow.

In the test level hierarchy, UI/UX testing is on the highest level, as you are basically testing the whole system including all dependent systems. Usually, for a testing strategy you start with lower levels.

Conversation Library↔ Test Suite

A test suite for a chatbot consists of a library of conversations transcripts. Head for the most important use cases for your chatbot, and the most important edge cases. Identify the “Happy Paths”, the use cases where user behaves as expected, and continue with other conversation flows then.

So, whenever one of those conversations with the Chatbot fail, you know there is some trouble and you have to investigate.

The Testing Process

Running the test suite manually (for example, by starting Facebook messenger, open your Chatbot and start chatting) is feasible for very small projects only. It is time to introduce an additional concept “test automation” — with test automation, it is possible to run your chatbot conversations without your intervention.

A chatbot interacting with your chatbot — that’s what test automation for chatbots means.

The hard work is done by chatbots, you will receive the test reports when they are ready.

Verification vs Validation

Experienced software testers will spot an issue here: the “conversation ↔ test case” approach does not address if the chatbot actually meets the requirements. Indeed, this approach is a simplification to get you started with test automation. People new to software testing are often confused with the meaning of the terms “verification” and “validation”.

Verification addresses the concern: “Are you building it right?”. Validation addresses the concern: “Are you building the right thing?”.

What we are doing here is verification of your chatbot — we assume that the chatbot does the right thing (meets the functional requirements), but we have to check if the chatbot is actually working as designed.

When getting started with test automation, this approach is usually more easy to follow.

Botium Box

In general, there is domain knowledge as well as technical and development knowledge required to compose a test suite of conversations and feed them to your chatbot automatically.

Selenium, a very popular software for test automation engineers, provides a low-level tool for entering the world of test automation for websites, called Selenium IDE — it provides a point&click experience to record your interaction with a website, and makes it possible to replay the interaction automatically.

The Open Source project Botium has similar goals in mind: to provide tools and knowledge for automated testing of chatbots, with lowest entry level barrier possible. One of the tools is called “Botium Live Chat” and helps in recording your conversations, replay them anytime you want automatically and point out any difference with the transcript.

Screencast 1 — Recording test cases

Screencast 2 — Running test cases

P.S: Please have a look at Github project for Botium (The Selenium for Chatbots).