At this point, our block tower demo felt much, much better. The haptic feedback wasn’t nearly as laggy and disconnected. Users felt impacts the same moment they saw them.

Success! Or was it? 🤔

Perceptual simultaneity

The human brain is a curious thing. Humans are astonishingly good at building a mental model of the world based on multiple sources of sensory information.

But sometimes our senses can be deceiving. You may have noticed a problem in my previous infographic. Visual, haptic, and audio latency aren’t the same. Haptic latency is sub-100 milliseconds. Audio is about 40 milliseconds slower.

During our improved demo, someone asked if we could disable the audio. What happened next blew me away. The haptics felt radically more responsive.

I wish I could share this experience with you. With audio enabled, the haptics felt good. With audio off, the haptics felt great. Not just better—obviously better. We excitedly ran around the office grabbing people to see if the difference affected everyone. It did.

What’s happening is that the brain is fusing multiple sensory signals into unified perceptual events. When you see, feel, and hear your hand tapping a table, your brain interprets the three sensory stimuli as a single impact event.

When one of those signals, such as sound, is slightly delayed, then our perception of the other stimuli shifts with it because the sound “captures” the other stimuli. This causes us to perceive the haptic sensation to occur later than it actually occurred.

Here's where it gets weird and cool. This perceptual shift is automatic and involuntary. Knowing that the audio delay exists doesn’t allow your brain to account for the delay and correct it. No matter how hard anyone focused on fingertip haptics, the delayed audio still caused haptics to feel less responsive.

Psychophysics

Now we have all sorts of interesting questions to ask. How much latency is too much? How synchronous do stimuli need to be? Is it more important to lower latency or increase synchrony?

To start answering these questions, we ran a psychophysical experiment. Recall for a moment the title of Max Di Luca’s paper, “Perceptual Limits of Visual-Haptic Simultaneity in Virtual Reality Interactions.”

Our first study was simple. We wanted to assess how much latency is allowed between a visual and a haptic stimulus for users to consider them simultaneous. This test was performed in virtual reality, with tracked and fully articulated hands. Participants performed a task to tap a virtual cube.