Real-world implementation and testing

There’s a lot of talk about chat-for-commerce, or #convcomm. Some point out the difficulty of user-testing, and the exciting possibilities of AI bots in chat.

At Kip, our primary focus is sales transactions for teams on Slack. Basically we have to make a sale, or we don’t get any revenue. That puts a real pressure on getting commerce-chat to work, beyond a theoretical or future-state scenario.

We’ve talked a bit about how to build AI personas for chat and how transparency affects sales conversions. In this post, we’re going to discuss how we designed Kip chat through research and user testing.

Curation is fundamental to Chat UX

This sounds like a no-brainer, but a conversation is not like a website. You cannot display pages of search results and other info for users to scroll through. Imagine a World with No Scroll, that’s what you’re building for. You don’t want info overload like:

Too Much Information! (Image credit: http://blueballfixed.ytmnd.com/)

If you think about how conversations work, key information comes first and in multiple info-bites where each piece expands on the previous. In journalism, it’s known as an Inverted Pyramid:

News VS Sales funnel (Image credit: www.nextnature.net)

The problem was that Inverted Pyramids were opposite of how Sales Funnel worked. In an inverted pyramid, the most critical moment was on the top. In a sales funnel, the most critical moment was at the bottom where the sales purchase occurred. The key question: How to design a shopping experience that was flexible enough for a point of sale at discovery and also after evaluation?

Optimizing User Intent

The old version of Kip was just a search engine, and one of the things we learned was optimizing user intent. When you put a query in a search engine, you’re declaring an interest or ‘intent’. When you click on a results link, that’s the ‘meaning’ of your intent. For example:

User A searches “Presidents” → declares intent

Clicks on “President Barack Obama” → intent meaning User B searches “Presidents” → declares intent

Clicks on “President Brie Cheese” → intent meaning

Once you have a large mass of queries, user-clicks and profiles you can infer greater likelihood of what User A is looking for when they ask Kip for “blue” which could signify a color, a music genre or a steak preparation. Extracted keywords from search terms are indicators of meaning.

We took the same system and applied it to chat. When a user initiates a conversation with Kip, we take their keywords and syntax, and match it up against similar queries from others. Using machine learning, we can infer what the keyword indicates as a shopping preference. Better matches means the sales funnel becomes shallower and moves the point of sale upwards.

Using machine learning to improve Kip’s understanding

Refining Intent for Sale Conversion

Machine learning optimization works in the long term, but how could Kip immediately help users discover products better?

We studied chat UX and decided not to show every product result. Instead, we would show only 3 results, and let the user know how we curated our best matches.

We call this the ‘3-choice’ model. 3-choice was developed based on problems of over-choice, and psychology of match 3. Sets of 3 also have great associations in people’s minds, from trinity to set theory to winning positions and games.

Types of Match 3 games

We started laying out different mockups and observing how people interacted with each layout. We took turns as Kip. As testers conversed with Kip “bot”, we extracted key interaction types and found patterns.

Testing Kip conversations on Slack

Then we mapped key user interactions to different contexts and actions within Kip’s chat code to make a functional prototype of Kip for user testing on a large scale:

First functional chat prototype, Kip 0.1 (Sesame)

Limiting choices to 3 meant that users had to converse with Kip, instead of passively interacting in a search or browsing. Customers are more likely to buy from a store with engaged salespersons, we wanted users to actively chat and build a relationship with Kip.

Using Conversation to Guide Sales

How could we make banter useful? The goal of Kip is to help users find what they need and drive a quick sale, not just chit-chat.

After recognizing intent, Kip’s role is to guide the user through their choices by asking for preferences and suggestions. In this, we developed our own language of sales guiding similar to how a salesperson would interact: “Would you like something similar to this?” became “More like option 1”. “How about the same item but in blue?” became “Option 2 but in blue”.

We carried this idea through when designing our system. Like using ‘more’ to show more results, and ‘save to cart’ which is implicit.

Navigating through data and switching context between chat threads with Kip

It’s important that chat banter is relevant to your product, and help develop trust with users. When Kip returns modified results we send back a snippet of their chat: “Here are some choices in Blue, which do you like best?” to acknowledge the user. In the future, we will let users ask Kip to modify the learned preferences on their profile.