When we think about machine vision, we usually think about it in a human context. We build systems that can recognize faces in our photographs or count the number of cars in a traffic jam. Rare is the computer that's watching on its own terms. That got artist Ben Grosser wondering: Why not let a computer watch something for its own sake? What would that even look like? To find out, he programmed an artificially intelligent viewer and sent it to the movies.

Grosser let his program loose on scenes from six films, including The Matrix, American Beauty, Inception and 2001: Space Odyssey. The software "uses computer vision algorithms to 'watch' for areas of prominence, patterns, colors, and other aspects for each frame of the movie," he says, identifying them as items of interest and tracking them through a scene.

Some lightweight intelligence algorithms give the computer a measure of agency, letting it pick between, say, a face, a building, a sign, or something interesting in the background. "I choose the clips themselves," Grosser says, "but after that, the computer takes over and decides what to look at it, how long to look at it, and where it goes next."

The program doesn't just watch, however. It also records what it's watching, documenting its algorithmic gaze in real time in a flurry of lines. The resulting clips aren't just portraits of machine vision, but also a unique record of the films themselves.

>The computer's eyes, perhaps even better than our own, register the broad shifts in cinema.

The computer's "eyes" register the broad shifts in cinema over the last several decades, perhaps even better than we can register them ourselves. "The more recent sci-fi clips show lots of rapid visual changes," Grosser notes. In movies like The Matrix and Inception, actors and objects move quickly, or else are moved quickly through rapid cuts and a variety of camera angles. We see this frenzied pace in the program's drawings of these films. After just three minutes of watching Inception, Grosser's virtual viewer has studied nearly the entirety of the frame.

The older films, like Taxi Driver, Annie Hall, and 2001 "represent a different approach to filmmaking," Grosser says, "one with far fewer camera changes and much less active movement on screen." In these, the program traces and retraces, studying subjects much as we would during a longer, steadier shot. The output for these become much sparser sketches of a less frenetic cinematic era.

The clips do, however, help us pinpoint a few ways in which computer vision is different from our own. The frantic drawing during the Inception clip comes during a series of explosions, which the computer registers as a series of tiny, rapid changes. We humans tend to consolidate them into a single event. "When watching that clip myself, I watch mostly the origins of the explosions and as much as anything else focus on those aspects of the frame that aren't moving," Grosser says. In other words, computers can't resist explosions–they're high-bandwidth optical spectacle, as stimulating as it gets. Humans, having seen so many of the things, tend to focus on what's going on in the context.

For Grosser, those types of discrepancies lead to all sorts of questions, not just about what computers watch but what we choose to watch, too. When you compare our gaze to the machine's, "what does that difference reveal about our culturally developed ways of looking?" he asks. "Will a system without our sense of narrative watch the same thing? I'm left wondering why I and the computer see things so differently."