In the two years since Microsoft launched Kinect, there’s been huge interest among enthusiasts in adapting the hands-free controller to new settings beyond gaming.

Techies and geeks of open- and closed-source shades have hacked the hardware, used the tools or adapted Kinect to steer computerised, wheeled armchairs.

Among those doing the fiddling is Microsoft Research in Cambridge, the Microsoft division whose work on real-time human pose recognition and depth-sensing algorithms (here and here) helped make Kinect a reality in the first place.

These engineers have been using Kinect in different experiments in an attempt to construct Star Trek-style holodecks and Minority Report-style 3D touch-displays.

They have combined feeds from multiple Kinect cameras positioned around a room, stitching together a 3D map of the room and then combining the map with moving pico projectors. The result has been to combine Kinect’s depth sensing with the pico data so you can see and touch the computer-generated pixels without the need for special glasses, headsets or gloves to operate. Among the achievements so far is a pixel-generated car driving around the floor.

The Kinect sensor has also been combined with mirror and web-camera tracking to produce a holodesk and Kinect sensors used as hand-held units to scan objects in 3D.

Augmented reality car using Kinect, from Microsoft Research

Shahram Izadi, who jointly leads Microsoft Research Cambridge’s interactive 3D technologies group, reckons augmented reality – or what he also calls “looking at pixels in the air without a screen” - will be the “next big thing” thanks to Kinect and other industry efforts. On Izadi’s team is Andrew Fitzgibbon who helped author that original paper on human skeletal 3D recognition for the Kinect.

“We are starting to move towards this notion of augment reality. It’s about augmentation with the instrumentation of the environment, instead of augmentation of the user,” he said on a recent visit to Microsoft in Cambridge.

“We have been working a lot with the Kinect and know the potential,” Izadi told The Reg.

Izadi points to a number of innovations in a relatively short period of time, including Kinect. We've also had multi-touch motion gaming and augmented reality in smartphones from Apple and Google.

“Companies like Microsoft will be looking to exploit it,” Izadi predicted.

Izadi reckons on reason augmented reality is about to step from science fiction to fact is the relatively ubiquity of the Kinect. The Kinect sold so many units during its first 60 says (eight million) that it set a Guinness World record. The Kinect's companion Xbox 360 game unit – it has since been made available for the PC, too – is also now the biggest selling games console in the US.

Another factor on the Kinect’s side is its price point: £129 from the Microsoft online store in the UK. This makes getting your hands on the Kinect doable - which is not the case with most augmented-realty kit. “Kinect is just a starting point,” Izadi says.

“The reason I’m so excited is two years ago there was nothing like this on the market and it cost tends of thousands to buy [the equipment]. Now we are talking about natural language user interface and augmenter reality as if they are almost upon us, and two years ago we were far away form that vision.”

Izadi’s team has been capitalising on Kinect’s depth-sensing technology to meet the vision. The depth-sensing camera can tell the system how far an object is from the sensor. The Kinect is a rich source of 3D data and it is mostly this information that is being interpreted. An image is smaller than one you’ll find in most digital cameras, 640 x 480 pixels, but the Kinect manages to track 300,000 points at a rate of 30 frames per second.

But there are niggles, hence the work’s still at the prototype stage.

One problem is that the various Kinects interfere with each other when set up and running at the same time, a fact that creates visual noise on the display. This is something Microsoft Research realised could be solved for now with the addition of a little vibration to each unit; the vibes are supplied by a small motor fixed to the top of each Kinect in the holodeck idea (here and here).

Also, sensor readings can be incorrect, with 3m being read as 2.9 meters for example, or they can fluctuate. In other scenarios depth data can’t be scanned, for example in the case of the Dell logo on a Dell PC box.

Izadi also reckons the next challenge is not input - although issues still remain over the input received via the Kinect – but rather the lag in output technologies, for the moment filled by pico projectors.

When will holodecks and holodesks become a reality?

Despite his enthusiasm, Izadi points to the fact the work at Microsoft Research tends doesn’t pay off immediately – it’s generally a good decade before something gets turned into a product. Kinect was quicker, of course. It was first discussed in 2007 and officially unveiled by Microsoft at E3 in June 2009 under the codename Project Natal.

In the meantime, the pace is being pushed on tablets and smartphones thanks to companies like Layar and Qualcomm, which lets you point your phone at a real-world object and receive information, with the phone acting as a head-up-display.

“Many companies are now developing augmented reality or immersive technologies,” Izadi says, "and that will accelerate a lot of what we are seeing. But to add the disclaimer: we are a search group and the work we are doing is five to 10 years out on what could feasibly be a product... [however] there is this real acceleration effect in we are seeing." ®