When we need to send a command or a file from our smartphone to our laptop, we do it through menus. But imagine if we didn’t have to send something to another device by dragging it to a file, or selecting it from a menu. What if we could use our iPhones as a magic lens instead, so that our smartphones could seamlessly interact with what’s happening on the screens of our laptops and tablets?

This is the vision for THAW, the latest project to emerge from the MIT Media Lab. THAW can see and interact with what’s happening on other screens, allowing your iPhone to download a song from your laptop just by looking at it through the camera, or to literally “pick up” a video game you were playing on your television console and seamlessly continue playing it on your smartphone. And that’s just for starters.





It’s the result of an internal collaboration between Philipp Schoessler of MIT’s Tangible Media Group, which also birthed a revolutionary, shapeshifting display you can reach through and touch, and Sang-won Leigh of the Fluid Interface Group. The former group spends its time trying to make the digital world more physical; the latter works to find new ways to make the physical world more digital. Sometimes partners, sometimes rivals, the two are like Yin and Yang.

The question we wanted to answer was: How can we combine computer interfaces into a single seamless experience?

“We live in an increasingly digital world, but that world is fractured between many screens and interfaces,” Schoessler says. “The question we wanted to try to answer with THAW was how can we combine these computer interfaces and screens into a single seamless experience.”

THAW works by placing a color grid on the monitor, and using the iPhone’s camera lens as a way to detect what part of the monitor it’s hovering over, similar to the sensor on a computer mouse. But THAW is much smarter than your average mouse: it can see what’s underneath it, and use the phone’s screen to interact with it. To not distract users, the team devised a way to hide the color grid everywhere except directly underneath the iPhone camera.

Although the THAW’s position-detecting mechanism for the iPhone is relatively simple, the interactions it enables are surprisingly complex. A lifelong fan of Super Mario Bros., Leigh programmed up a simple video game that shows how THAW can be used in a variety of ways that put the likes of the Microsoft Kinect and Nintendo Wii to shame. These are video game consoles that can see you, sure, but what they can’t do is see the other screens in your life. And that opens a lot of gameplay possibilities.