Inside Deep Dreams: How Google Made Its Computers Go Crazy

Why the neural net project creating wild visions has meaning for art, science, philosophy — and our view of reality

Mordvintsev, Olah, Tyka

I gripped the desk and sagged toward her as she held out the envelope, but I refused to accept it. The Woman’s face was changing: swelling, pulsing… horrible green jowls and fangs jutting out, the face of a Moray Eel! …Terrible things were happening all around us. Right next to me a huge reptile was gnawing on a woman’s neck, the carpet was a blood-soaked sponge — impossible to walk on it, no footing at all. “Order some golf shoes,” I whispered. “Otherwise, we’ll never get out of this place alive. You notice these lizards don’t have any trouble moving around in this muck — that’s because they have claws on their feet.” “Lizards?” he said. “If you think we’re in trouble now, wait till you see what’s happening in the elevators.” — Hunter S. Thompson, Fear and Loathing in Las Vegas

In the very early hours of May 18, 2015, Alexander Mordvintsev was wrenched from sleep. A nightmare, he later described it to me, in the first interview he has granted on the experience. Or, at least, a dream, a deeply disturbing dream, where an intruder had crossed the threshold of the Zurich apartment that he, his pregnant wife, and his 3-year-old son had been occupying for the past few months. They had moved to Switzerland from St. Petersburg that last November, when the Russian computer scientist got a job at Google’s engineering center there.

Now it was darkest night and Mordvintsev, jarred awake by his savage slumber, leapt from the bed to check the door. It was closed; all was quiet. But his mind was afire. Okay, it’s 2 a.m., but I can’t sleep, he told himself. So time to write a few lines of code.

It would be a decision that would eventually unleash a torrent of fantastic images, torn from an alien perspective, that intrigued and twisted the minds of those who viewed them. A decision that would reveal the power of artificial neural nets, our potential future overlords in an increasingly one-sided relationship with machine intelligence. And a decision that would change Mordvintsev’s own life.

He turned to a project he’d been working on since the beginning of the year. Mordvintsev had been fascinated with neural nets (NN), a computer analogy to the barely understood thicket of connections in our own brains. Sophisticated artificial neural nets now power “deep learning,” the hottest and most promising development in artificial intelligence. He was tinkering with his own vision-recognition neural net, developed with open-source tools. A number of these had appeared over the last few years, part of a boom in the field after these systems had proved effective in computer vision and other functions that had previously been elusive.

None of this work, it happens, had anything to do with Mordvintsev’s official duties. Google is a leader in NNs, with assets that include pioneering researcher Geoffrey Hinton; Jeff Dean, a legendary Google computer scientist who leads a team that built an NN informally dubbed the Google Brain in Mountain View; and Google’s DeepMind acquisition in London pushing the boundary of machine intelligence. Mordvintsev had no formal connection to them; he worked in Safe Search, which prevents spam and porn from infecting search results. But Google still tolerates its engineers using part of their time to work on passion projects. For Mordvintsev, this now meant neural nets and vision systems. It was an extension of his previous interest in using computers to model biological systems. Years earlier, he had tinkered in simulations of coral reefs.

As an NN newbie, Mordvintsev was teaching himself about the field, absorbing key papers and playing with systems already trained to recognize certain objects. His curiosity was piqued by one the abiding mysteries of neural nets and deep learning: why did they work so well and what the hell went on inside them? Others had been asking the same question, using what are known as convolutional neural nets (ConvNets) to probe vision recognition systems at various points in the process. ConvNets are a specialized form generally used for vision recognition; they take the biological metaphor farther by not only using a neuron-style learning system, but by employing the neurons in a similar fashion to the way light receptors are arranged in the visual cortex. One team in particular, from the Visual Geometry Group at the University of Oxford, had taken an interesting approach to analyzing how successful vision systems can recognize (classify) objects: at a certain point in the training process, they got the networks to generate images of what they were perceiving. By looking at those images, the researchers had a better idea of what the neural network was up to at that instant.

Mordvintsev wanted to continue down that path, with a wicked turn: He was writing code to make a neural net create meaningful images that weren’t there at all, at least not as humans could tell — visions born of machines, oozing out of the metaphorically neural connections in the system. On this restless night in May, while his wife and child slept, he did the coding equivalent of fiddling the dials to change the objective of the neural net. Let’s find something that increases the magnitude of the activation vector, he told himself. Just like, whatever it sees in this batch of images, let’s have more of it.

In other words, he would flip the function of the neural net from recognizing what was there to generating stuff that might not be there. In the middle of the network’s usual practice of trying to verify a nascent sense that a particular pattern may be a target object, he told the network to skip directly to “Go,” and then start making the object. Previously, the mission of convolutional neural nets was to proceed in a defensive-driving fashion, straining to filter out wrong turns and make accurate predictions. Mordvintsev’s process was more Fast and Furious. It was like gunning the system forward, then suddenly slamming on the brakes and reversing. You could almost taste the pixels being spit out like greasy gravel when the wheels spun on digital asphalt, as the system seized on hints of objects and recklessly took license to flesh them out into vivid representations of a target image.

The trick was getting the system to do its thing — reversing itself and then reaching back into itself to find templates for new images — at just the right time and in just the right measures. “It’s easy to write the code and tricky to find the right parameters,” says Mordvintsev. The actual chunk of computer code that turns a neural net into something that churns out images from its hitherto secret life turns out to be only about thirty lines of code. But in this pass, Mordvintsev got the balance just right.

The results came instantly. The open-source tool he was using to build his neural net had been “trained” on a well-known dataset called ImageNet to recognize objects of 1,000 categories, including 118 dog breeds. He fed a photo into it: a beagle and kitten, each perched on tree stumps with a meadow in the background. (He found it on a digital wallpaper website.) Normally, one would use a vision-recognition neural net to identify what it saw. But Mordvintsev was hoping for something novel and unexpected. His code tapped the neurons mid-process, building the half-baked clues of dogness into more fully realized dogs. With repeated passes of modified images, he got a final output that was not at all normal.

The image is of a dog, in the broadest sense. This is startling to begin with, because the source of this image was not the beagle but the adorable little kitty cat. (Maybe not so surprising, given that the network was trained largely on dog breeds.) On the beast’s forehead was a second set of eyes. Bulging from his canine haunches was a separate snout with another pair of unsettlingly alert eyes. In fact, pieces of dog face popped up in all sorts of unexpected places. Overall, it appeared that some horrid infection was brewing underneath the animal’s fur, with teeming sets of snouts and eyes straining to burst through at the next instant. If you looked closely, a pair of eyes had even broken though in the pinkish lower jaw. For good measure, the background of the picture, some sort of green wall, displayed a complex tapestry of patterns, as if Aztecs had finger-painted the surface. In several places on the wall, it looked like spiders had randomly broken through, like arachnid bullet holes.