Video: Screen sees in 3D

An everyday LCD screen has been modified to “see” the world in front of it in 3D. That means a viewer can control on-screen objects by waving their arms in the air without touching the screen, let alone a mouse or keyboard.

“This is a level of interaction that nobody’s ever been able to do before,” says Ramesh Raskar at the Massachusetts Institute of Technology Media Lab, who created the prototype shown in the video above with colleagues Matthew Hirsch and Henry Holtzman, as well as Douglas Lanman at Brown University in Providence, Rhode Island.

The screen – dubbed BiDi, short for bi-directional – allows users to manipulate or interact with objects on the screen in three dimensions. It will also function as a 3D scanner, he adds. “If you spin an object in front of screen, the software will stitch together a 3D image.”


Sharper focus

Raskar and Holtzman’s team were inspired by the way manufacturers of LCD panels, including Sharp and Planar Systems, are experimenting with adding optical sensors between a panel’s pixels so that it can act as a touch-screen interface.

But such displays have poor vision, like a camera with no lens, says Lanman: they can clearly image objects that are in direct contact with the screen, but anything further away is blurred. The researchers set out to modify the concept to let the screen see the world in front of it more sharply.

Placing a tiny lens slightly in front of each sensor would do that, but the layer of lenses would adversely affect the images produced by the display. Instead, Raskar and Holtzman’s team used a standard 20-inch screen to show how a basic feature of all LCD screens can perform the job of a lens array.

Pinhole pixels

The brightness of each of an LCD’s pixels is controlled by a layer of liquid crystals, which can swivel to physically control how much light passes from the display’s backlight. In BiDi the team use that function to control light passing in the other direction onto an array of sensors behind the display.

When the screen is “looking” around it, most of its pixels are shut off by the liquid crystals. But a regular grid of hundreds of pixels spread across the screen use their liquid crystals to create a tiny hole that acts as a pinhole camera lens, focusing an image of the scene in front onto a thin translucent film a few centimetres behind the LCD. Those images are detected by a camera inside BiDi, allowing the device to know what is happening before it.

The LCD screen’s pixels must also do their usual job of presenting images to the user, though. They oscillate between their two tasks many times per second, too fast for the viewer to notice that while they are watching the screen, the screen is also watching them. “We take the normal LCD layer and put it to double duty,” says Lanman.

Stereo imaging

Exploiting the different viewpoints of pinholes in different places on the screen makes it possible to reconstruct stereoscopic images, by taking a small amount of information from each of the pinhole images. Several stereoscopic image pairs are produced, each sharply focussed on objects a particular distance from the screen, from which the system can calculate how far away the object is.

“We produce multiple images, each focused on a different plane in front of the screen all the way to, say, 50 centimetres away from the screen,” says Lanman. “For instance, your hand will be blurred except in the one image that’s at the right depth.”

A further processing step singles out the sharply focussed parts of each image in the stack, creating a sharp depth map of objects within the screen’s field of view that can be used to track 3D gestures in the same way a standard touch screen captures touch gestures.

The whole process takes place in real time, and computer software interprets those gestures as an input to control the objects on the screen.

The BiDi screen will be presented at Siggraph Asia later this week.