Eyetracking has long been a common way to enhance usability studies with insights that are more detailed than those gleaned from users’ thinking-aloud comments. Since 2005, Nielsen Norman Group has run many eyetracking research studies, some documented in the book Eyetracking Web Usability.

Eyetracking is particularly useful for understanding details in users’ reading behaviors and how they deal with advertisements. But new UX teams shouldn’t employ eyetracking for their initial usability studies. Only at the highest UX-maturity levels should a team start using eyetracking because eyetracking has several downsides, including:

high cost of the specialized equipment

challenges to design and moderate a methodologically valid eyetracking study

difficulty of tracking the eyes of people with thick-rim eyeglasses or heavy eye makeup

Luckily there is now an alternative: instead of tracking users’ eyes, we can track their ears.

Eartracking in Its Infancy

We first became aware of the benefits of eartracking during our research with cats and dogs. Many animals have ears that visibly turn in the direction of their attention. Turning the ears is clearly an evolutionary adaptation that allows predators to keenly follow the prey and it also enables potential preys to notice a stalking predator before it comes into visible range.

(Some dogs have floppy ears that don’t turn, but even these ears are adaptive: floppy-eared dogs are usually so cute that humans will feed them, and they won’t need to hunt in the first place. Also, floppy-eared dogs have high job security: the United States Transportation Security Administration has decided that the vast majority of bomb-sniffing dogs in airports should be floppy-eared breeds because of better passenger acceptance caused by extra cuteness.)

Eartracking Measures

Although humans don’t have floppy ears, our ears don’t visibly turn in the direction of potential dangers or potential food. However, evolution has preserved vestiges of ear-turning muscles, as is clearly demonstrated by anybody who wiggles their ears. Humans have some ability for small ear movements under conscious control. What is less commonly known is that we also exhibit subconscious micromovements of the ears. When analyzing ear movements as an indicator of human reaction, we look at two new biometrics: 1) the distance the ear moves, and 2) how many times the ear moves (known as microwiggles) per second.

These micromovements are not observable by the naked eye, because the ear moves less than 0.1 mm (0.004 inches), and most people don’t know that the human ear can microwiggle up to six times in one second. In fact, because microwiggles of the ear are so unobtrusive, they have not been the subject of serious research until now.

Ear–Mind Hypothesis Is Real

The main finding from our research is that the ear–mind hypothesis is as valid as the eye–mind hypothesis that underlies the use of eyetracking in user research. The eye–mind hypothesis states that people look at what they are interested in, which is why we can use measures of gaze direction to estimate what the user is attending to. Similarly, the (now confirmed) ear–mind hypothesis states that micromovements of the ear are directed toward things that are startling or surprising to the user.

Note the difference between the two sense–mind hypotheses: the eyes might look at anything that is merely interesting, while the ears react only at unexpected stimuli that are potentially of high interest or importance. This difference is obviously caused by the evolutionary background for micromovements of the ear. Fossils more than 160,000 years old have indicated that our ancestors had ear macromovements, about a thousand times more pronounced than our ear movements today. These movements supported survival in eat-or-be-eaten scenarios, where noticing surprising or startling things were of utmost importance.

Eartracking-Technology Advancements

Though small, ear micromovements can be picked up by an 8K video camera that’s placed closely enough to the ear that’s being studied. (8K cameras are not yet common, but NHK in Japan has been experimenting with this next-generation video technology since December 2018 and was kind enough to lend us one of their cameras.)

A second technology advance now allows us to turn micromovement video streams into true eartracking and tell where the user’s attention is directed. A machine-learning algorithm has been trained with 10,000 hours of video recordings from our most recent eartracking study, during which we tracked users’ ears as they attempted standard tasks on a wide variety of websites. Unfortunately, running the resulting AI software in real time (which is obviously required for practical use of eartracking in a usability study, since, in order to ask followup questions after a task, the facilitator needs to know what the user attended to, as well as how far and how many times her ears microwiggled) currently requires a supercomputer rated at 50 petaflops per second.

Currently, only 4 computers in the world are fast enough to do eartracking: 2 in the U.S. and 2 in China. Luckily, this distribution allowed us to continue our tradition of testing with both American and Chinese users. After all, we’ve previously found that Chinese users and Western users differ in their approach to the visual complexity of website layouts. So it was a plausible hypothesis that results from eartracking studies might differ across cultures as well. However, we didn’t find any differences, so the rest of this article will refer to the combined data from the two studies.

Eartracking vs. Eyetracking

Eartracking is not a highly sensitive technique: it can only capture environment stimuli that receive a high level of interest. Thus, a phenomenon like banner blindness, which we have documented through our eyetracking studies, would not be something that we’d expect to capture with eartracking because we do not expect users to be surprised by advertising banners (unless they come with loud noises that play automatically). Thus, banners would not register in eartracking even if users did pay attention to these ads (which we know from eyetracking that they don’t).

Another interesting contrast between eyetracking and eartracking relates to gender differences. First, let me point out that we almost never observe any substantial differences between male and female users in usability studies. In terms of the user interaction, users of both genders are equally annoyed by, for example, zigzag layouts that impede scanning.

However, there are certainly some differences in the content that different genders find interesting. For instance, men in our eyetracking research looked much more at certain body parts in photos. (For the sake of remaining a family website, we won’t go into further details, but the heatmaps are in our book.)

Eartracking found another interesting difference between male and female users: Male ears twitched quite substantially when the user interface included pictures of wooly mammoths or stereo equipment. (In fact, these are the only instances in our entire research study where the micromovements reached their maximum extent of 0.1 mm, and up to five wiggles in one second. Nonmammoth and nonstereo UI elements rarely registered more than 0.05 mm, though photos of elephants scored 0.08 mm — possibly because of the resemblance between mammoths and elephants during the initial 100 ms of exposure.) In contrast female ears didn’t twitch any more on pages showing wooly mammoths or stereo equipment than on webpages with other pictures.

Why do men’s ears react more strongly than women’s ears to webpages with pictures of wooly mammoths? We can only speculate, but it’s likely that during the era of the cave people, it was mainly the men of the tribe who were assigned to hunt the wooly mammoth, and since such a kill would be a major win, the hunters got highly attuned to recognizing this animal. As for the stereo equipment, your guess is as good as mine.

The finding of gender differences in mammoth webpages was due to a lucky coincidence: we employed the new eartracking technology with a few of the users during our recent study of UX design for children, and happened to include the wooly-mammoth page on National Geographic Kids’ website. (Subsequently, we repeated this test with adult users and confirmed the finding.)

Eartracking Strengths over Eyetracking

It’s still early days in applying eartracking to UX research, but it seems a promising new methodology. Compared to eyetracking, eartracking has the advantage of being able to measure surprises, which will be particularly valuable for game user research. There’s also an obvious accessibility advantage when testing with people who are blind and can’t participate in eyetracking studies.

Eartracking Weaknesses

On the other hand, eartracking has some weaknesses that parallel the disadvantages of eyetracking. As we mentioned before, certain user characteristics are difficult for current eyetracking equipment. Similarly, eartracking equipment can’t track people with:

hair that covers the ears

diamond stud earrings that reflect light into the camera, or any ear cuffs

very small ears

hearing aids that go over the ear in any way

earmuffs

earbuds

Also, while an eyetracker is fairly expensive, the supercomputer required for an eartracking study runs close to $100 million. (Luckily we were granted free supercomputer time due to the revolutionary nature of our research.) A bigger supercomputer currently under construction is named after a senior eartracking researcher, showcasing its promise for wider use of this exciting technology.

Summary

If your user research budget has an extra $100 million that you don’t know how to spend, and you have a highly advanced UX team and mature product team, consider allocating the money to an eartracking study.

Other April Fools Articles