Growing up reading sci-fi and fantasy books, I’ve always found it easy to imagine how spectacular our world could look in the future. For the most part, the entertainment industry has done a great job visualizing a future filled with advanced augmented reality technologies like in the world of Tron, Minority Report’s floating motion-controlled computer screens, Avatar’s holotable or, perhaps one of my favorites, the absolutely amazing hologram of Tupac. Okay, the Tupac hologram is real life.

We know what deep augmented reality should look like but why isn't it in our hands yet? Augmented and virtual reality are certainly on the right track, but we've got a long way to go until we get our Holodecks.

Working at Paracosm, I get to participate in brainstorming sessions on making the technologies found in popular sci-fi cinema a real thing. We believe the key to truly advancing technologies, like augmented and virtual reality, lies in the ability to connect the digital world with the one we live in. We do this with 3D mapping.

How can 3D mapping improve Augmented Reality?

1. Physics

Virtual objects can interact with the physical realm when they understand its terrain. When game engines have a detailed 3D map of an environment, they can use simulated physics to allow virtual objects to behave as if they were real and obey our Earthly laws. For example, if we dropped virtual kittens on your coffee table, they could interact with your coffee table and the objects on it as if those tiny, adorable, rubbery kittens were real.

2. Realistic occlusion & lighting

If a virtual system understands your space, the lighting of virtual objects can be accurately displayed. Deep compositing and surface occlusion make the interaction appear present in the real world. For instance, if you’re playing an AR game in which you chase NyanCat around your home, it wouldn’t be all that convincing if NyanCat floated through your couch. For it to appear real, it would be to be aware of the couch and go around it, while being partially occluded by the couch’s shape. Our brains are very good at distinguishing real from unreal. Little things, like shadows and reflections, need to affect virtual objects in the same ways they would in the real world. Otherwise, the magic is spoiled.

3. Real-world AI

With 3D mapping involved, game logic can know more about the world than the players. It knows where we are, where our enemies are, where important virtual items are hidden, even where your couch is. The game can share this knowledge, or use it against us. Either way, the gaming experience is more powerful and immersive when a game has spatial intelligence.

Let's say you're play a game where monsters invaded your home through portals on your wall. The virtual creature wouldn’t just know about the wall it was crawling out of, it would also know where its friends are, and could navigate its way to another room. The creature would know more about your space than you do because it would be aware of both the physical and virtual elements at play.

4. Unique Tailored Experiences

In a real-time strategy game, your couch is a mountainous region where turrets can be placed. The flow of the battle field is dependent upon the shape of your space. Strategies and tactics for battling in your living room are different than your friend's living room. Currently, AR technology is reliant on markers, and maps are limited to table top sizes. With 3D mapping at work, your entire house can become the playing field.

Are you ready for deep augmented reality? I definitely am.