Choose your use case

First thing is to choose the scenario you’d want to control.

Obviously controlling a smartwatch OS is very different than a smart TV or a smart home device.

What is the right User experience for the Smart TV scenario

Analyze

A smart TV in essence is a menu and a media player.

The Media Player requires a minimum of 4 actions:

1. Play/Pause

2. Next/Skip Forwards

3. Previous/Skip Backwards

4. Volume Control

Menu control also consists of a minimum of 4 actions:

1. Select

2. Menu/Back

3. Scroll Up/Right

4. Scroll Down/Left

Apart for that, since the Mudra device has an air-mouse we chose to make the selection more versatile by adding a cursor which gives another dimension to the scrolling (more on the TV OS itself in a future article).

Observe

We built a Smart TV OS so we could try various ways to control it and also so we could change the OS itself in order to have the right Interaction.

Within the smart TV OS there’s a genre menu (that contains the movie items). These items can be scrolled right and left.

Then we ask users : “What would you do to scroll to the right on this screen?”.

While they move their hands to operate the TV OS, someone from the team would mimic the designated response by clicking a wireless mouse from a distance.

The first reaction is usually extreme excitement (of course we don’t initially tell them we’re “cheating” in this stage, so we could keep learning from them).

This is not a novelty! Making the interaction link directly to the intention is key for understanding what works, as opposed to what you think would work…

Create a wish-list

After observing enough users, you can really get a sense of what works and what doesn’t . So you write down your top three gestures for each interaction,

then you cross reference all the results from all the interactions your use case requires. Finally, you try narrowing it down to as few gestures as you can.

There are two reasons why you should aspire to as few gestures as possible:

1. Your users will need to learn and remember all the gestures

Usability studies have shown that most people are comfortable holding around 4–6 slots in their memory at any given time. more than that just wouldn’t work — they might not remember them.

2. Developing multi-gestures AI models requires a lot of data gathering and engineering work

so if you can choose only the optimal gestures, you could save the time it would take to develop them and go to market faster.

In the TV OS example, after my wish-list held 10 Gestures, we cross-referenced it and created a 6 Gestures User experience that we believed in:

1. Tap- Play/Pause | Select

2. Swipe inward- Next/ skip Forwards | Scroll Down/Left

3. Swipe outward- Previous/ skip Backwards | Scroll Up/Right