Behold The Next Generation VR Technology: Part 7 — Controllers and Haptics

Sceptics say current VR headsets are clunky, heavy, uncomfortable diving masks that won’t make their way to people’s homes. Well, I agree, but with one significant exception — the technology is moving lightning fast.

In less than 2 years of VR-consumer-version era headsets have become completely wireless, field of view doubled, and the display resolution tripled.

In fact, VR technologies are evolving much faster than consumer versions of headsets can be released.

In the series of weekly articles called “Behold The Next Generation VR Technology” I will guide you through the world of the latest and most promising tech that will finally make VR the next computing platform.

Mostly on the early stage of development, all this tech will be implemented in consumer version headsets during the next 10 years. Some sooner, some later.

I divided this series into parts — one part for every vital aspect of VR technology. This one is about:

Controllers and Haptics

There are some objects in VR that it would be such a pleasure to touch and squeeze gently. I would stroke them with my fingers to feel their texture, rounded shape and warmth. God… I like chocolate muffins so much.

Current industrial designers are doing an unbelievable job by creating super-advanced controllers like Oculus Touch(I beg you, Vive owners, forgive me). Sometimes I literally forget that I am holding a half of a gamepad and my brain starts thinking of it as a real gun or a bow.

But what about more complex objects? How fun would it be if we could grip a virtual cup of coffee with two fingers and feel its smooth and polished texture?

Knuckle’s four finger grip zones / Source: Valve

Knuckles‘ sensors tracks motion of each finger independently— Full video / Source: Cloudhead Games

A player grabs a ball with with different fingers — Full video / Source: Zulubo Productions

I strongly believe the best technology is the one that is invisible to the user. So, Valve is trying to free your fingers from buttons oppression. They released a very limited number of the prototype controllers called Knuckles.

The significant difference is that it has four grip zones: three on the handle and one on the trigger. These are the areas for sensor arrays which detect index, middle, ring, and pinky fingers curl motion. And they do it independently.

It means the controller can tell whether any of those fingers are pointing straight out, fully curled around the grip or somewhere in between. And the adjustable strap doesn’t let it slip from your arms while you unbend the fingers.

Knuckles finally gives the freedom of interactions with a finger accuracy, and at the same time you still have the tactile feel of, let’s say, a gun handle with a trigger.

Unfortunately, there are no plans or release date from Valve.

Leap Motion’s algorithms reconstruct arm bones from black and white 2D images / Source: Leap Motion

A player grasps a virtual cube with bare hands — Full video / Source: Leap Motion

The other technology with that kind of interactions is a camera sensor, like Leap Motion.

It’s a small device that attaches to the front of the HMD and tracks your arms with a 150 degrees wide and 120 degrees deep field of view.

Leap Motion’s technology is just infrared LEDs that beam light(invisible to the human eye) on your hands and two 2D cameras to track them.

Contrary to popular belief the device doesn’t have a depth sensor. All the magic is in the computer vision algorithms(that are a bit of a secret the company holds) which somehow reconstruct the skeleton of your palm from that images, even if fingers are occluded.

However, the results are impressive: the device tracks all movement inside its force field at 200 frames per second with a precision down to 0.01mm.

JFYI: Oculus has bought as much as 3 companies with similar technology: Pebble Interfaces, Nimble VR, and 13th Lab. All of them focus on both tracking hands and developing a model of objects and spaces around a subject.

And then there is touch. Engineering fantasy in that field has no limits.

User holds the rock and squeezes the rubber duck with Dexmo exoskeleton glove — Full video / Source: Dexta Robotics

First one is an exoskeleton glove technology. Like Dexmo from Dexta Robotics.

It imitates the shape of objects by using tiny motors that limit finger movements. So, when you grab a virtual bottle of water, they let your fingers bend to the diameter of it, no more. Your brain will compare it with what it sees and will finally accept it as believable information.

This tech allows you to feel the elasticity. You can squeeze some virtual objects, like a virtual rubber duck, and it will also feel real.

HaptX’s glove has the same force feedback mechanical technology, but they also added what they call “Microfluidic smart textile”. It’s a layer inside of a glove, that always in touch with the skin. They describe this tech like that:

HaptX’s glove textile actuators simulate the sense of touch / Source: HaptX

“Our flexible, silicone-based smart textile contains an array of high-displacement pneumatic actuators and embedded microfluidic air channels. The actuators provide haptic feedback by pushing against the user’s skin, displacing it the same way a real object would when touched. High-performance, miniature valves accurately control the pressure of each actuator to create a virtually infinite variety of sensations — texture, size, shape, movement, and more. An optional second layer of microchannels can add temperature feedback by delivering variations of hot and cold water.”

NormalTouch controller traces the virtual surface — Full video / Source: Microsoft research

TextureTouch controller simulates the virtual shape — Full video / Source: Microsoft research

Hapto VR gaming controller imitates shape of a virtual object — Full video / Source: Hapto

And here comes some more actuator tech prototypes.

Microsoft research showed two types of such controllers called NormalTouch and TextureTouch.

NormalTouch renders object surfaces and provides force feedback using a tiltable and extrudable platform. It rotates and changes angle under the player’s forefinger imitating the the virtual surface.

The other called TextureTouch has a platform with 4x4 matrix of actuated pins in the centre that are being put forward to render shape virtual forefinger is touching.

A similar technology but in a more completed form is proposed by Hapto.

Their controller has 20 little pushers under the player’s palm that become active when player’s virtual hand touches the surface of an object. Thus it simulates the feeling of touching a real object in VR.

But what if instead of controllers, simulating touch we could use real objects that can be corresponded to multiple virtual items distributed throughout a virtual environment? Or simply “props”.

Well, it is possible with a technique called “Redirected reach”.

Player grabs the real prop with redirected reach — Full video / Source: mohamed suhail

Like any “redirected” technique it fools your brain by insensibly adjusting the movement of your virtual hand towards the real prop position, so you can grab the real prop with your hand even if it’s position is different from the virtual object.

The same prop can be both a door handle and a totem on a table in a room you just entered.

Ultrahaptics’ speaker panel generates a soundwave that can be feeled — Full video / Source: Tim Schofield

The last haptic technology is a bizarre one. It is made by the company called Ultrahaptics and uses mini ultrasound speakers to emit tactile feedback for gesture interfaces.

Basically, it’s a 14x14 transducers board that can generate shapes like buttons, sliders, and surfaces in mid-air so you can touch them or even drag.