John Underkoffler is the CEO of Oblong Industries, and he is perhaps best known as the science advisor for the landmark sci-fi film Minority Report.

Long before the body suit of Ready Player One, Underkoffler helped envision Minority Report’s scene where actor Tom Cruise uses “data gloves” and gesture controls to manipulate a transparent computer. He talked about the creation of that experience and what comes next at our recent GamesBeat Summit event.

To bring that vision to reality, he founded Oblong Industries in 2006, and launched Mezzanine for enterprise collaborators in 2012. Oblong’s technological and design trajectories build on 15 years of foundational work at the MIT Media Lab, where Underkoffler was responsible for innovations in real-time computer graphics systems, optical and electronic holography, large-scale visualization techniques, and the I/O Bulb and Luminous Room systems.

Image Credit: Michael O'Donnell/VentureBeat

I visited Oblong’s warehouses in downtown Los Angeles last year. There, I saw huge video walls, curved to immerse the user inside a visual experience. You could really see data and the connections between objects in a visual way. He showed me you could grab an image and toss it from a computer screen to a big video wall, seamlessly.

Sadly, outside of games, we don’t see this kind of user interface in general computing.

“General purpose user interfaces in every application except for gaming have made no strides, no quantum leaps, since 1984,” said Underkoffler. “That’s the year that Apple turned the command line interface into the GUI, the graphical user interface. Everything that has come after has been a step to the side.”

The smartphone brought us mobile computing, but it gave us tiny windows and feels like “sideways progress,” Underkoffler said.

Image Credit: 20th Century Fox

Underkoffler hopes that games will work better as user interfaces because they enable you to maneuver through a visual space. Data has to be organized in spatial ways, and that’s why “everyone who works in UI should play games,” he said. Descent was a game that helped us maneuver in 3D, and Portal helped people figure out how to reorient themselves in 3D.

Underkoffler liked the depiction of frames and space in Gorogoa, which plays with the idea of space to create puzzles. He also liked Monument Valley, 4D Toys, and a game called Empty, where you get furniture line up in a room and it disappears. Each of these games teaches you to interact with a digital world in a new way. Underkoffler tied the innovations in these user interfaces to innovations in architecture, such as those made by Lebbeus Woods.

“What we need to drive for is a new spatial language, a new UI,” Underkoffler said. “We need a language that’s made of images, graphics and space that works better than language does at describing most of the things that we care about.”

As I watched it again, particularly with the images, Underkoffler’s talk kind of blew my mind.

Check out the video of Underkoffler’s presentation.