Generally, images of shiny objects tend to befuddle computers. For example, the glare often makes it difficult for computers to identify the object accurately. “What’s really interesting is that they didn’t see reflections as a corruption of the image,” says artificial intelligence researcher Deborah Raji of the AI Now Institute at New York University, who was not involved in the research. “They asked: ‘What can we see in the reflection?’”

To reconstruct the environment, the researchers used a handheld color video camera with a depth sensor that roughly detects the shape and distance of the shiny objects. They filmed these objects for about a minute, capturing their reflections from a variety of perspectives. Then, they used a machine learning algorithm to reconstruct the surroundings, which took on the order of two hours per object. Their reconstructions are remarkably accurate considering the relatively small amount of data that they used to train the algorithm, says computer scientist Abe Davis of Cornell University, who was not involved with the work.

The researchers could achieve this accuracy with so little training data, in part, because they incorporate some physics concepts in their reconstruction algorithm—the difference between how light bounces off shiny surfaces versus matte surfaces, for example. This differs from typical online image recognition tools in use today, which simply look for patterns in images without any extra scientific information. However, researchers have also found that too much physics in an algorithm can cause the machine to make more mistakes, as its processing strategies become too rigid. “They do a good job of balancing physical insights with modern machine learning tools,” says Davis.

The environment reconstruction, however, was merely one task in a larger project. The researchers’ ultimate goal was to generate new 3D perspectives of the chip bag: to have their computer accurately predict the bag’s appearance from all 360 degrees. Creating realistic views of a shiny object is a big challenge among AR and VR researchers. The glare patterns of a chip bag, for example, morph dramatically when you view it from different angles in a brightly-lit room. Because it’s difficult to make a computer reproduce these changing patterns, virtual shiny objects often look distorted and flattened—not very realistic. But the University of Washington researchers found that, by first reconstructing a shiny object’s environment, they could make more realistic views of the objects.

A frame of a video clip used to reproduce the window and house across the street, compared to the actual scene. Photograph: Jeong Joon Park/University of Washington

“I’m very interested in reconstructing the 3D world,” says lead author Park, a graduate student at the University of Washington. “By that, I mean copying the room you’re in and putting it in a virtual world, so that later you can interact with it in a realistic way.” He mentions future uses in VR gaming, for example. More realistic virtual perspectives could also benefit furniture companies like IKEA, which already offers an AR app called IKEA Place that allows you to virtually insert their products into the rooms of your house.