This is the great mystery of human vision: Vivid pictures of the world appear before our mind’s eye, yet the brain’s visual system receives very little information from the world itself. Much of what we “see” we conjure in our heads.

“A lot of the things you think you see you’re actually making up,” said Lai-Sang Young, a mathematician at New York University. “You don’t actually see them.”

Yet the brain must be doing a pretty good job of inventing the visual world, since we don’t routinely bump into doors. Unfortunately, studying anatomy alone doesn’t reveal how the brain makes these images up any more than staring at a car engine would allow you to decipher the laws of thermodynamics.

New research suggests mathematics is the key. For the past few years, Young has been engaged in an unlikely collaboration with her NYU colleagues Robert Shapley, a neuroscientist, and Logan Chariker, a mathematician. They’re creating a single mathematical model that unites years of biological experiments and explains how the brain produces elaborate visual reproductions of the world based on scant visual information.

“The job of the theorist, as I see it, is we take these facts and put them together in a coherent picture,” Young said. “Experimentalists can’t tell you what makes something work.”

Young and her collaborators have been building their model by incorporating one basic element of vision at a time. They’ve explained how neurons in the visual cortex interact to detect the edges of objects and changes in contrast, and now they’re working on explaining how the brain perceives the direction in which objects are moving.

Their work is the first of its kind. Previous efforts to model human vision made wishful assumptions about the architecture of the visual cortex. Young, Shapley and Chariker’s work accepts the demanding, unintuitive biology of the visual cortex as is — and tries to explain how the phenomenon of vision is still possible.

“I think their model is an improvement in that it’s really founded on the real brain anatomy. They want a model that’s biologically correct or plausible,” said Alessandra Angelucci, a neuroscientist at the University of Utah.

Layers and Layers

There are some things we know for sure about vision.

The eye acts as a lens. It receives light from the outside world and projects a scale replica of our visual field onto the retina, which sits in the back of the eye. The retina is connected to the visual cortex, the part of the brain in the back of the head.

However, there’s very little connectivity between the retina and the visual cortex. For a visual area roughly one-quarter the size of a full moon, there are only about 10 nerve cells connecting the retina to the visual cortex. These cells make up the LGN, or lateral geniculate nucleus, the only pathway through which visual information travels from the outside world into the brain.

Not only are LGN cells scarce — they can’t do much either. LGN cells send a pulse to the visual cortex when they detect a change from dark to light, or vice versa, in their tiny section of the visual field. And that’s all. The lighted world bombards the retina with data, but all the brain has to go on is the meager signaling of a tiny collection of LGN cells. To see the world based on so little information is like trying to reconstruct Moby-Dick from notes on a napkin.

“You may think of the brain as taking a photograph of what you see in your visual field,” Young said. “But the brain doesn’t take a picture, the retina does, and the information passed from the retina to the visual cortex is sparse.”

But then the visual cortex goes to work. While the cortex and the retina are connected by relatively few neurons, the cortex itself is dense with nerve cells. For every 10 LGN neurons that snake back from the retina, there are 4,000 neurons in just the initial “input layer” of the visual cortex — and many more in the rest of it. This discrepancy suggests that the brain heavily processes the little visual data it does receive.

“The visual cortex has a mind of its own,” Shapley said.

For researchers like Young, Shapley and Chariker, the challenge is deciphering what goes on in that mind.

Visual Loops

The neural anatomy of vision is provocative. Like a slight person lifting a massive weight, it calls out for an explanation: How does it do so much with so little?

Young, Shapley and Chariker are not the first to try and answer that question with a mathematical model. But all previous efforts assumed that more information travels between the retina and the cortex — an assumption that would make the visual cortex’s response to stimuli easier to explain.

“People hadn’t taken seriously what the biology was saying in a computational model,” Shapley said.

Mathematicians have a long, successful history of modeling changing phenomena, from the movement of billiard balls to the evolution of space-time. These are examples of “dynamical systems” — systems that evolve over time according to fixed rules. Interactions between neurons firing in the brain are also an example of a dynamical system — albeit one that’s especially subtle and hard to pin down in a definable list of rules.

LGN cells send the cortex a train of electrical impulses one-tenth of a volt in magnitude and one millisecond in duration, setting off a cascade of neuron interactions. The rules that govern these interactions are “infinitely more complicated” than the rules that govern interactions in more familiar physical systems, Young said.