Augmenting your world Lightform

You don’t need special glasses to see an augmented view of the world. A new combined camera and computer can superimpose images over real-world objects without the need for a head-mounted display.

Lightform connects to a video projector to beam images and animations on to surrounding objects, essentially turning any surface into a screen – a technique called projection mapping. To do this, it scans the environment using depth sensors to map the shape of objects, then tailors its lighting effects to fit. “The idea is to seamlessly merge the virtual world with the physical world, and to do it without wearing anything on your face,” says Lightform CEO Brett Jones, whose firm came out of “stealth mode” this week.

Initial demos show a coffee shop’s price list materialising on a blank slate (pictured, top), squiggly lines dancing across a store window display, and a cactus undulating with decorative pulses of light.


Augmented reality has so far often been delivered through a wearable device: Microsoft’s HoloLens uses a headset, and the much-hyped Florida-based start-up Magic Leap is expected to launch its AR headset later this year. In February, Facebook’s Mark Zuckerberg hinted that the company is interested in developing AR eyewear alongside its Oculus Rift virtual reality headset. “The goal is to make VR and AR what we all want it to be: glasses small enough to take anywhere,” he wrote in a Facebook post.

But eliminating headsets would mean many people can share in an AR experience at once and without special preparation, and removes problems around the comfort, weight and power cables of wearable displays.

Projection mapping technology is usually used for large-scale, one-off events. Jones previously worked at Disney Imagineering, developing projections for theme parks, and led projects at Microsoft Research to create expansive projected gaming experiences.

AR for anyone

Lightform, based in San Francisco, aims to turn the technology into something anyone can use. The device is designed to work with existing projectors and comes with software that Jones says is as easy to use as Photoshop.

The size of the projection depends on the projector. “You could do your coffee mug using a tiny pico projector or you could do the side of a building using a really big projector,” says Lightform’s design director Phil Reyneri. The camera periodically rescans the scene and recalibrates projections if things have moved, making it suitable for long-term installations, and you can control or modify the graphics through an app. The whole package will cost more than a depth sensor like Microsoft’s Kinect but less than a mid-range laptop when it starts shipping later this year, says Jones.

The mapping is not quite real-time – it takes about a minute to do a scan – and you can’t interact with the projected images, unlike with some systems that use haptic devices or motion tracking to give users the illusion of touching what they see. A projected AR prototype by Texas-based Argo Design, for example, uses computer vision to allow people to play air hockey using real objects as bats to hit a virtual puck, with the pitch markings projected on to a tabletop – though it only works on a flat surface.

Argo chief technologist Jared Ficklin imagines projected interfaces being used to control smart home devices alongside voice-recognition technology like Amazon’s Alexa. It could project a recipe on to a kitchen surface, for example.

Using light to augment reality is exciting, says Natan Linder at the Massachusetts Institute of Technology. Adding projection mapping means you can “paint with light” and give real-world objects virtual textures that fool the eyes. But projected AR has its own drawbacks, he points out. Shadows can be a problem if anything gets between the projector and surface, and it does not work well in bright spaces.

If projected interfaces become integrated into the spaces we live and work in, they could usher in a new kind of ubiquitous computing, says Linder. But first they need to find really useful applications.