“It’s very difficult to design something for someone if you have no empathy”

- Stewart Butterfield, co-founder of Slack

Winning customers is hard, but keeping them is harder. The best companies invest in proper customer support because they know that a good CS team is more than just a team of firefighters: it’s a treasury of knowledge about your product, your users and what those users are thinking.

A customer usually contacts your company when things go wrong. The more critical the issue is, the more emotional that message will be. Often these messages are the first indicator we get that a critical bug exists in the system, or that we’ve made a poor UX decision, or that a feature is causing more problems than it solves.

Customers also contact us to convey positive sentiments, whether that’s via the App/Play Store, Trustpilot or directly via email. It’s important that we understand and are able to sort the sentiment of these messages as well, so that we know what we’re doing well. Both kinds of feedback are crucial to understanding our users and building the right product for them.

Azimo now moves hundreds of millions of dollars around the world every year. With that kind of scale, it’s not easy to identify and sort all of our customer feedback so that we can refer to it effectively in customer research. As there is nothing more helpful than people’s feedback, we decided to tackle the challenge by integrating sentiment analysis into our customer support tools.

Data flow

The core of our support system is Zendesk platform, this is where all of our customer communication happens. Zendesk handles:

User and partner emails

Messages coming from the website and apps

Social media, including Facebook and Twitter

Review systems like Trustpilot and the AppStore or Play Store

Each new message is analysed and dispatched to the right team and the agent to handle it. However, besides internal automation (agent assigning, automatic replies, SLA policies), Zendesk supports external service plugins through webhooks.

One of our webhook implementations for Zendesk is a service we built called Ticket Enrichment. When a new message appears in Zendesk, a call is made to the Ticket Enrichment service, which then analyses the content and parses data through various internal functions, like Natural Language Processing among the others. When all data is ready, it sends back the enriched ticket to Zendesk via their API.

Ticket sentiment analysis

One of the features of our Ticket Enrichment service is text analysis, which helps us classify what the sentiment of the incoming message was. Thanks to Natural Language Processing, we can tell whether the message was:

Negative (e.g., the user was angry, disappointed or just had a bad experience),

(e.g., the user was angry, disappointed or just had a bad experience), Positive (user was happy with our service),

(user was happy with our service), Mixed (user could be happy with some of our services, but had bad experiences or issues with something else),

(user could be happy with some of our services, but had bad experiences or issues with something else), Neutral (e.g., the user reported a less critical issue or gave us non-emotional feedback).

The implementation

We’re in the early days of building a centralised and automated customer support experience, so it’s crucial that we’re able to build and learn fast. That’s why we decided to prototype and build an MVP on top of the Firebase platform. Firebase provides features such as environment scaling, logging and simplified deployment, allowing us to focus on building the functionality we need. Think of it like hiring a shared workspace instead of building and maintaining your own office 😉.

Deep dive into details

Our Ticket Enrichment service is built on top of Firebase Cloud Functions, as an HTTP trigger function. Whenever a new ticket is created on Zendesk platform, it makes a webhook call to our function. After data preprocessing, we do text analysis including syntax and sentiment analysis with Google Cloud Natural Language API. Natural Language processing gives us some raw numbers representing sentiment and content emotionality. As the sentiment varies for different use cases and customers we use our formulas which we’ve built based on historical data and experiences.

What are we learning?

Thanks to all these features, we can analyse thousands of tickets every week. We now have a much better idea of which features people like, which are giving them trouble, and which critical bugs need to be fixed.

And this is just the beginning. Natural Language Processing shows a lot of promise and its accuracy is increasing all the time. Today we can identify and understand a user’s problem programmatically before a customer service agent even reads the message.

Right now an engineer still needs to triage the problem and deploy a fix, but we’re hopeful that in future we will be able to fix a user’s issue, or at the very least provide them with accurate information while they’re waiting, automatically and immediately.

For the rest of cases, text classification will substantially reduce response time, and allow us to assign a ticket to a customer service agent with the appropriate expertise to deal with the customer’s issue. No more “let me just pass you on to the best person to deal with this.” 😎

Examples

As our users’ privacy is crucial for us, we won’t share any real conversations. Instead, here are some test cases we wrote which show how sentiment analysis works. In the cases below, we would have been able to identify not only sentiment but also the nature of the problem/compliment.

Negative sentiment

User 1:

“Hey, I have a problem with your app. It is constantly crashing on my device. I cannot login and I’m not able to do transaction. It is useless. And I cannot contact with you. For sure this is very last time when I’m contacting with you.”

User 2:

“I don’t like your service! You are slow, you don’t respond my questions. Your app doesn’t look very good on my phone. I won’t recommend you to anyone.”

Positive sentiment

User 1:

“Hey Azimo!

I love your app! I’m your client for years now and I have never had any issues with transactions. Money is delivered on time, very quickly. Thank you for doing your hard work, you are the best company!”

User 2:

“Your app is awesome. It makes transferring money to my home country easy and fast. I can’t ask for more. I love it.”

Neutral sentiment

User 1:

“Dear Azimo. I’ve been trying to contact you, I have questions about my transfer. Can you tell me what is the best way to talk with you?”

What next

While sentiment analysis is powerful technique that helps to understand your customers, for us it’s just the beginning. The state of the art in Natural Language Processing gives us tools to not only classify emotions hidden behind a message but also find its deeper meaning. By extracting keywords, categories and sentence structures it is possible to build solutions that automatically analyze all communication channels in real-time, suggest trends (what people like or dislike the most in our product), and track it over the time (has the last direction in building UI made people like our app more?).

With growing scale, it is extremely important to understand your customers. So what can be better than listening their feedback? With Natural Language Processing we are able to hear everyone who wants to speak. No matter when and how much they want to tell us.

It would have never be possible for us, humans.