Former Microsoft Senior Researcher Dr. Hrvoje Benko gave a talk entitled ‘The Future of AR Interactions’ at the International Symposium on Mixed and Augmented Reality (ISMAR) conference in October. This week the talk was uploaded on the ISMAR YouTube channel.

Dr Benko had worked at Microsoft since 2005, but moved to Facebook Reality Labs (formerly Oculus Research) in late 2017. He now leads the human computer interfaces (HCI) division there.

A Great Display Is Not Enough

A core point that Benko stressed multiple times during the talk is that a great AR display in itself is not good enough — a new input paradigm that takes advantage of spatial computing is needed.

Benko used the example of smartphones with large displays that existed before the iPhone but lacked a multitouch input interface. He pointed out how Hololens and other current AR devices unsuccessfully try to use existing input techniques.

Finger Tracking Is Not Enough

Benko explained that while finger tracking technology is rapidly progressing, humans don’t often interact with empty air — we interact with objects. The only time we tend to use our hands in empty air is when gesticulating during speech.

The lack of haptic feedback with only finger tracking, he claims, is jarring, and is unlikely to be the basis of future interfaces.

Surfaces May Be The Key

Benko pointed out that mixed reality interfaces could leverage the already existing surfaces in the environment to provide real haptic feedback.

Menus could appear on the nearest table or wall, and your fingers could manipulate the virtual UI elements on these surfaces.

This obviously requires a very advanced sensor system with a precise understanding of all the major objects in the room, as well as almost perfect finger tracking.

‘Haptic Retargeting’

The issue with using real world surfaces of course is that often they won’t be sufficient for the task at hand. As Benko puts it: “passive haptics don’t scale.”

One of the most interesting research projects Benko recounted was ‘Haptic Retargeting’. This trick allows one physical haptic object to act as many virtual objects by warping the body movement you see in the headset. This is situationally dependent, but the concept if applied correctly to the user’s environment could be revolutionary.

Haptic Controllers: ‘VR Revolver’

One of the last projects Benko worked on at Microsoft before his move to Facebook was a pair of haptic controllers which can ‘render’ the feeling of an object under your fingertips.

Dubbed ‘Haptic Revolver’, the controller used a roller of different materials which can be actuated on demand based on what you’re touching in game. Benko stressed that it didn’t particularly matter exactly what input the fingertip received. As long as something happened, the brain accepted it as valid input.