As the fields of virtual and augmented reality continue to advance, researchers, developers, and content creators alike are tasked with finding ways to improve computer vision and graphics in their quest toward more immersive models of the world—whether real or imagined. While reconstruction techniques have come a long way, particularly since the advent of motion sensors, mirrors and glass panes have been notoriously difficult to reproduce in a convincing manner. Later this month at SIGGRAPH 2018, Facebook Reality Labs (FRL) will present a fully-automated pipeline to reconstruct mirrors and other reflective flat surfaces, which could improve the believability and realism of 3D scenes. Today, we’re excited to introduce you to some members of the team behind these advances and share the story of this impressive research project.

“If you look around the mirror section of a home decor shop, you’ll immediately see a wide range of shapes and sizes—almost no two mirrors are the same!” says Research Scientist Thomas Whelan. “It became obvious to us very early on that we needed a general solution that didn’t make too many assumptions about what was being reconstructed. We really wanted a solution that just worked in real-world environments because that’s where it’s most useful.”

The Problem In-Depth

Existing 3D scanning systems produce unwanted artifacts with mirrors and glass because the 3D geometry shown on the reflective surface gets incorrectly reconstructed behind it. Whelan kicked off the project by suggesting that a simple, easy-to-detect target could be used to identify reflective surfaces. By identifying mirrored surfaces, the team could then re-render the scene with correct geometry and reflections.

“The sheer variety of mirror types and shapes was quite stunning in the beginning,” notes Research Scientist Julian Straub. “Designing a system that would be able to handle most or all mirror shapes and sizes was the main goal. Then we realized that the system would also work with glass surfaces with a minor additional classification step. That was pretty cool.”

“It’s surprisingly difficult to describe how a human recognizes a mirror as distinct from simply a window or doorway into a different space,” adds Research Scientist Steven Lovegrove. “To create a 3D reconstruction that can work in a wide variety of real spaces, the challenge was to collect a representative sample of mirrors and create a robust algorithm that would work on each of them.”

Challenge Accepted

Whelan worked with the team to define which features were most useful for determining border cues, while Straub focused on the segmentation of arbitrarily shaped mirror boundaries. Once the team put all the pieces together, Lovegrove built a system to calibrate the rig and estimate the location of mirror planes in a given environment. The capture rig itself includes an infrared depth capture camera, an RGB color camera, wide-angle cameras to estimate motion and a backlit pattern showing the mirror image of a standard AprilTag commonly used in robotics applications.

This lets us localize the mirror’s boundaries with a good degree of accuracy—even when the mirror doesn’t have a frame. We can even add virtual objects to the scene and they’ll correctly interact with the real-world geometry and reflectors.

The end result: an end-to-end system that goes from raw data to final reconstruction.

Visiting Researcher and Professor at TU Darmstadt Michael Goesele worked with the hardware engineers and technicians to build the actual prototypes and detect the targets in the image streams.

“I’ve been working on many aspects of scene reconstruction for almost two decades and was able to leverage my experience in building various acquisition setups as well as the full reconstruction pipeline,” Goesele notes. “I had already published a couple of papers on reconstructing and rendering reflective surfaces, so I had a pretty good mental model of the problem, which was a huge help for me.”

It was also a help to the team as a whole, since Goesele coordinated work along the full pipeline and pushed the paper and SIGGRAPH submission forward. “Finally,” he adds, “I bought lots of mirrors online, which was really fun.”

The Road Ahead

As with any great research problem, perhaps the most exciting part of FRL’s work in mirror reconstruction is the potential for continued work in the field.

“There are a lot of cool ideas you can try now based on the idea of a clearly identifiable reflected target,” says Whelan. “Perhaps more relevant to today’s trend in deep learning, this approach gives people a fast and fully automatic way of labeling input data for training that contains these types of challenging surfaces. Previously, images would have to be labeled slowly by hand. With our approach, you get the labeling for free!”

“We hope this work will enable others to more faithfully capture 3D representations of space for their own applications such as robot navigation and scene understanding,” adds Lovegrove. “Toward the goal of creating more realistic and predictive virtual scenes, there are possible extensions to our approach that may support the depiction of even more challenging objects such as non-planar glass surfaces or translucent materials.”

From Dream to Teamwork

By bringing together world-class researchers, engineers, and developers, FRL is well positioned to tackle challenging problems and push the technologies of virtual and augmented reality forward. But it’s the people behind the research that ultimately bring the dream to life.

“I spent my sabbatical at FRL since I was really interested in the problem space the team is working in,” explains Goesele. “I thoroughly enjoyed my time here—particularly the fact that I could work with a fantastic team and great resources. What impressed me most was the speed at which we could move while working on this project and how active and seamlessly people work together.”

“Many members of our team have worked across 3D reconstruction, graphics, robotics, and localization from research labs around the world,” says Lovegrove. “It was really exciting to bring our knowledge together in one place to obtain results that even we didn’t expect to look quite as good as they do.”

“This work really brought together a ton of expertise from everyone’s backgrounds on the team,” agrees Whelan. “Mirrors are typically skirted around in 3D reconstruction, and most earlier work just ignores them by pretending they don’t exist. But in the real world, they exist everywhere and ruin the majority of reconstruction approaches. So in a way, we broke the mold and tackled one of the oldest problems in 3D reconstruction head-on.”

To learn more about the work being done at FRL and to view our current open positions, please click here.

— The Oculus Team