Chatbot retention has been a real problem… a problem so big that most people don’t even get past the first two messages!

According to İlker Köksal CEO of BotAnalytics, the initial drop off is huge: “about 40% of users never get passed the first text and another 25% drop off after the second message. Daily retention rate is at a paltry 1–2% and the monthly retention rate for bots isn’t much better, sitting at about ~7%.”

Fortunately, after hacking for the better part of 6 months+, a few bots -such as Poncho- have found the light and are seeing awesome retention and engagement rates.

Recently, I had the pleasure to meet with Kuan Huang Founder of Poncho, Omar Siddiqui CEO of Kiwi (Creators of Sequel) and İlker Köksal CEO of BotAnalytics and we discussed the issue of retention and engagement.

How to Benchmark your Bot the Right Way

There is such a wide variety of chatbot use cases that it does not make sense to compare against the average. You might have a use case that solves a problem and hopefully the user never comes back (eg: DoNotPay Bot) or you might have a daily use case.

Best way to benchmark is by comparing your bot to mobile apps in your category. At the bare minimum, your goals should be to surpass them.

Part 1: Know Thy User

How Poncho increased engagement by 300%

Poncho, like most other bots, suffered from low engagement. The bot did its job every well, it could tell you what the weather forecast is while telling you a funny joke. And when it comes to personality, Poncho is playful and funny, easily one of the best bot personalities out there. So why was Poncho struggling?

Solving a Problem

To put it mildly, Poncho was facing a number of giant challenges. For one, why would someone use the bot? It is easier to get the weather from Siri or to look at your home screen(on Android Devices). Further, apps like Google Now automatically push the weather to you. How could Poncho compete with this?

It’s going to Rain in 10 in San Francisco

Poncho began segmenting their best users, people who were using the bot daily. After digging into the data, they tried to answer the ‘Why’ question. As it why did these users come back, what problem was Poncho solving and more importantly what problems could Poncho solve?

After doing a lot digging they got to the most critical element, they found their Core! Users wanted to be notified of upcoming weather events that were unusual. For example, every time Poncho tells me, “Bring your Umbrella, it is going to rain in 10 min in SF” I feel a wave of gratitude. Siri doesn’t do that, neither does Google Now. Poncho found their core!

According to Kuan Huang founder of Poncho, here is how to find your core:

Identifying the cohort that has the highest engagement and retention. Once you identify the type of “high quality” users, start optimizing your funnel to convert new users (as close as possible) to that cohort. Focus on the Most critical piece of value for your users: For example, ‘Facebook’s 7 friends in 10 days’ was their key driver. Always start at the top of your funnel. Benchmark with apps in the same category first. Your bot has to at least do better than that :)

“Once you identify the type of “high quality” users, start optimizing your funnel to convert new users to that cohort” Kuan Huang

Using Behavior and Leveraging Psychology

Once you have your most successful cohort you can start creating a User Persona based on this data.

This is extremely powerful as it is based on actual user behavior and can give you insights into ‘why’ users do what they do.

One of the easiest ways to pull this data is by using Facebook Audience Insights. Create a custom audience based on your top performing cohort and Facebook will quickly compare your audience to the entire Facebook population.

The data will come rolling in, and then you can then drill down into the specifics. Facebook will give you a ton of data such as: demographics, interests, affinity groups, location, purchase activity, online activity, lifestyle, relationship status, and more.

During this process, you might discover that you have a few different User Personas. You will have to take this into account when you are personalizing their experience and solving their problem. Here is an example of a Shopping Bot’s User Personas:

User Behavior & A/B Testing

Once you have a clear understanding of your User Personas and have tracked their behavior in the bot, it is time for A/B testing. In this phase, you simply make hypothesis and run experiments.

Your Goals are to:

Improve Retention Improve Engagement

“The best place to start A/B testing is in the onboarding. You need to catch you audience right away, if you don’t, it’s much harder to retain them” İlker Köksal CEO of BotAnalytics

Part 2: Improving Chatbot Retention