What is a chatbot ?

Before we look at the three metrics that you will need to track in order to measure the performance of your chatbot, let’s first establish what a chatbot is and how they are being used today. In layman terms, a chatbot is a computer program that can have text or voice based conversations with a human. Recent advancements in natural language processing have made it possible for chatbots to hold longer, deeper and more natural conversations with humans. Thousands of chatbots already exist on platforms such as Messenger, Telegram, Slack, Kik and Skype. It is important to note that chatbots are still fairly new, so there is a lot of confusion with entrepreneurs and developers as to what metrics are the most important to track. This article was written to try and resolve this problem. Below are three key metrics that you should consider tracking when measuring the overall performance of your chatbot.

Number of users

The number of users that your chatbot interacts with is an important metrics as it allows you to get a good idea of the overall reach of your chatbot. In order to really evaluate your chatbot and look at other metrics the initial sample size has to be large enough. In other words, the number of users that your chatbot interacts with has to be high, around a few thousand should be enough. A sample size of 100 users is simply not high enough to truly evaluate the performance of the chatbot and also not high enough to look at other metrics such as session duration and retention rate. A user in this context can be defined as anyone who has had at least one conversation with the chatbot, even if the user no longer converses with the bot.

2. Average session duration and interactions per user

If the number of users is high enough, then other useful metrics can be looked at and evaluated. The average session duration is often a metric that is associated with the duration of a user on a webpage. This metric, can still be used in the context of chatbots. The average session duration is essentially the amount of time a user will spend talking to the bot in a single interaction. A low session duration indicates that users are either not engaged with the bot or may be having difficulty in communicating with the bot. This may resort to users leaving quickly as a result of frustration. A higher average session duration could indicate that the users are engaged with the chatbot.

It is worth noting that a higher sesion duration doesn’t necessarily mean that the users are content and satisfied with the chatbot (it could possible mean the opposite), so it may be useful to look at a small sample of the actual conversations to get a better idea of how people are actually interacting with the chatbot.

3. Retention rate

If you want to build a sustainable and scalable chatbot, you need to cut down on churn. Churn is defined as the rate at which people are discarding or stop engaging with the bot. To do so, calculating your customer retention is the first step in this process.

Simply put, the retention rate is the percentage of users that still use your service in relation to or relative to the number of active users you had at the start. Retention rate can be measured monthly, quarterly or yearly. Retention rates can even be measured on a daily or even a weekly basis. A high retention rate ie 90 % means you are keeping most of your customers. On the other hand, a low retention rate such as 50% indicates that you may need to improve the bot. It suggests that customers are either frustrated with the functionality of the bot or simply do not see it as useful.