Shan Carter, a researcher at Google Brain, recently visited his daughter’s second-grade class with an unusual payload: an array of psychedelic pictures filled with indistinct shapes and warped pinwheels of color. He passed them around the class and was delighted when the students quickly deemed one of the blobs a dog ear. A group of 7-year-olds had just deciphered the inner visions of a neural network.

Carter is among the researchers trying to pierce the “black box” of deep learning. Neural networks have proven tremendously successful at tasks like identifying objects in images, but how they do so remains largely a mystery. Their inner workings are shielded from human eyes, buried in layers of computations, making it hard to diagnose errors or biases. On Wednesday, Carter’s team released a paper that offers a peek inside, showing how a neural network builds and arranges visual concepts.

This particular line of research dates back to 2015, when Carter’s coauthor, Chris Olah, helped design Deep Dream, a program that tried to interpret neural networks by reverse-engineering them. Olah’s team taught a neural network to recognize an array of objects with ImageNet, a massive database of images. Then they told the network to, say, generate a dog or a tree based on what it had “learned.” The results were hallucinogenic images that reflected, in a limited sense, how the model “saw” the inputs fed into it. (It later turned out that the system could also produce rather pricey works of art.)

Since then, Olah, who now runs a team at research institute OpenAI devoted to interpreting AI, has worked to make those types of visualizations more useful. Neural networks are composed of layers of what researchers aptly call neurons, which fire in response to particular aspects of an image. For each level of the network, Carter and Olah grouped together pieces of images that caused roughly the same combination of neurons to fire. Then, as with Deep Dream, the researchers reconstructed an image that would have caused the neurons to fire in the way that they did: at lower levels, that might generate a vague arrangement of pixels; at higher levels, a warped image of a dog snout or a shark fin. They arranged similar groups near each other, calling the resulting map an “activation atlas.”

LEARN MORE The WIRED Guide to Artificial Intelligence

That lets researchers observe a few things about the network. By toggling between different layers, they can see how the network builds toward a final decision, from basic visual concepts like shape and texture to discrete objects. Olah has noticed, for example, that dog breeds (ImageNet includes more than 100) are largely distinguished by how floppy their ears are. The atlas also shows how the network relates different objects and ideas---say, by putting dog ears not too distant from cat ears--and how those distinctions become clearer as the layers progress.

Jeff Clune, a professor at the University of Wyoming who wasn’t involved in the study, says that the atlas is a useful step forward but of somewhat limited utility for now. Researchers trying to understand how neural networks function have been fighting a losing battle, he points out, as networks grow more complex and rely on vaster sums of computing power. “That increase so far has far outstripped our ability to invent technologies that make them interpretable to us,” he says.

The research also unearthed some surprises. As an illustration, Olah pulls up an ominous photo of a fin slicing through turgid waters: Does it belong to a gray whale or a great white shark? As a human inexperienced in angling, I wouldn’t hazard a guess, but a neural network that’s seen plenty of shark and whale fins shouldn’t have a problem. Then he shows me the atlas images associated with the two animals at a particular level of the neural network---a rough map of the visual concepts it has learned to associate with them. One of the shark images is particularly strange. If you were to squint a bit, you might see rows of white teeth and gums---or, perhaps, the seams of a baseball.