“What do we see when we read (other than the words on the page)?” asks Peter Mendelsund in a welcome and fascinating new book. Or more precisely, “What do we picture in our minds?”

Do we see Anna Karenina with her shining gray eyes under thick lashes, her faint smile and red lips; or Uriah Heep with his red eyes, red hair, dinted nostrils, and lank forefinger? Or Captain Ahab, who “looked like a man cut away from the stake, when the fire has overrunningly wasted all the limbs without consuming them”? Certainly this sounds vivid enough. But do we see him?

Mendelsund is convinced that readers already know, or think they know, the answer to this question. “When we remember the experience of reading a book,” he tells us—and throughout What We See When We Read he assumes that this experience is more or less the same for everyone—“We imagine a continuous unfolding of images.” And again, “When we read it is important that we believe we are seeing everything.”

Apparently we have a vested interest in supposing that we are capable of projecting a kind of continuous movie of the events in a novel, or indeed the events of our own past experience, to the point that we find it “terrifying and disorienting that we can’t recapitulate the world in perfect facsimile.” We must possess the world, visually, in our minds.

Art Director at Knopf and a highly respected book-cover designer, Mendelsund himself has an investment in all things visual and sometimes seems to think of visualizing as a necessary part of reading, a sort of proof of our readerly abilities: “I wonder,” he says, speaking of the reader’s passage from illustrated children’s books to adult novels, “if we … need, over time, to learn how to picture narratives unassisted.” Ironically, the least successful aspect of his book is its own obtrusive use of visual “support.” When Mendelsund talks about the timing involved in literary description, the fact that a novelist might withhold one feature of a character’s appearance for many pages—something a film can’t easily do and that readers will instantly recognize—he gives us a photo of a digital wristwatch. It is more a distraction, exhibitionism even, than an “illustration.”

The problem is that upon close examination the reading experience is far more complex and far less visual than is commonly supposed, or than Mendelsund supposes is commonly supposed. One of the pleasures of his book is his honesty and perplexity at the discovery that every account he offers of the process of visualization very quickly falls apart under pressure. We do not really “see” characters such as Anna Karenina or Captain Ahab, he concludes, or indeed the places described in novels, and insofar as we do perhaps see or glimpse them, what we are seeing is something we have imagined, not what the author saw. Even when there are illustrations, as in many nineteenth-century novels, they only impose their view of the characters very briefly. A couple of pages later they have become as fluid and vague as so much of visual memory. At one point Mendelsund posits the idea that perhaps we read in order not to be oppressed by the visual, in order not to see.

So what do we see when we read? First the page, of course, and the words printed on it. No “image” we have of the characters or settings will ever be as concrete, as indisputably and continuously present, as the solid book, or e-reader, itself. Meantime, characters and places are given to us in discontinuous fragments—this kind of nose, that kind of hair, a scar, a limp, a grimace—and in a process that Mendelsund recognizes as having a lot to do with memory we come to have the impression that we know this sort of person, this sort of place. “A man of about forty-five,” says Orwell of Big Brother, “with a heavy black moustache and ruggedly handsome features,” and we are satisfied we could pick out the man in a police lineup. Just find the guy who looks like Stalin.

So the faculty of recognition is important. The novelist says “wry smile” and we are satisfied we know what a wry smile is because we have attached those words to someone’s smile in the past. But do we visualize, or picture this smile? Mendelsund never really puts any pressure on these words—visualize, picture—which curiously do not have parallels for the other senses. There is no word for our deliberately recreating sounds or smells or touch or taste in our heads, as if it was generally accepted that memories of these other sensory experiences are more passive, while visually we can actively reconstruct an experience.

But can we? If I think of people I know, even those closest to me, the shadowy impression I have of their faces, bodies, gaits, has nothing of the intensity, immediacy, and solidity of their real presence. They may “flash upon that inward eye,” as daffodils did for Wordsworth, at the most unexpected moments, in a kind of echo of their presence, but this is not something I can control and it doesn’t last, it can’t be sustained. Often our visual memory is a sort of liminal waiting for the known person to appear: we stand at the airport arrival’s gate thinking of the son or daughter who is returning home. They are vaguely there, in our minds, waiting to be recognized, to become real. But we don’t see them yet.

More banally we may stand at the luggage collection carousel watching endless bags tumble onto the belt. We hold in our minds a shadowy idea of our own bag. Then suddenly it is there and the effort of “visualizing” ceases. Perhaps we realize that the bag is not quite as we remembered it. There are three zips not two. Or at least this is my experience. And when I read, I do not so much see the characters and the places as feel satisfied that if they were to appear to me I would recognize them. Hence our discomfort when we see the film of the book and the actors look nothing like the people we supposed we knew.

In general, as Mendelsund points out, the act of visualizing, struggling to see something that isn’t there, depends largely on semantics, on words. It is verbal as much as visual. If I’m sitting in a park by a river, close my eyes, and try to visualize the scene, I say to myself, river, trees, benches, and I seek to place them in relation to each other, though no idea I build up in my mind will compare with the intense presence of the scene when I open my eyes again. Quite simply, we do not possess the world, visually, in our minds. And if there is no word corresponding to visualize for the other senses—it may be because the other sensory experiences are verbally more difficult to reconstruct: of a smell we could say it was sweet, it was sour, or we could say the name we have given it, musk, lavender, but we cannot piece it together bit by bit, as we might a tree, thinking trunk, bark, branches, leaves, etc. An old half-forgotten smell may flash upon us with great intensity, but it is difficult to evoke at will, difficult even to trick ourselves into believing we can evoke it.

“The practice of reading,” Mendelsund says, “feels like and is like consciousness itself: imperfect; partial; hazy; co-creative.” This seems astonishing to me. My consciousness of the environment about me has nothing hazy or partial about it at all. As I type now, screen, fingers, keyboard, and the room around are all very present and wonderfully real, at least so long as I keep my eyes open. Perhaps Mendelsund means that our experience while reading, or on remembering what we have read, feels like our normal apprehension of all that is not immediately present to us, the places and people we try to imagine when we are far away from them. The reading process reactivates patterns of past experience to create new stories, pseudo-memories, in our minds.

But if we are not actually visualizing the people and places we read about in novels, what is the function of literary description? Quoting Nabokov on Dickens and his “intensely sensuous imagery,” Mendelsund gives these lines from Bleak House: “When the sun shone through the clouds, making silvery pools in the dark sea…” and Nabokov’s enthusiastic response: “these silvery pools in the dark sea offer something that Dickens noted for the very first time with the innocent and sensuous eye of the true artist, saw and immediately put into words.”

Mendelsund is unconvinced. The specificity of an image sparks our recognition and convinces us the author is attentive to a world we know and share. But it doesn’t really make us see. What neither man mentions is that Dickens always gives us lead characters—David Copperfield, Pip in Great Expectations—whose moods oscillate between gloomy depression and bright cheerfulness— and that these states are frequently evoked with references to weather and landscape. The description is part of an emotional pattern, what Mendelsund calls a “play of elements.” Its meaning is other than its visual content.

More generally, descriptions are exercises in voice and part of the overall verbal enchantment—literally, “entering into chant”—that Mendelsund never really discusses in his book and which remains, for me at least, the central experience when we read good fiction. Here is a passage he quotes from Huckleberry Finn. The phrases in square brackets show a few words that Mendelsund omits:

The first thing to see, looking away over the water, was a kind of dull line—that was the woods on t’other side; you couldn’t make nothing else out; then a pale place in the sky; then more paleness spreading around; then the river softened up away off, and warn’t black any more, but gray; [you could see little dark spots drifting along ever so far away—trading scows, and such things; and long black streaks—rafts]; sometimes you could hear a sweep screaking; or jumbled up voices, it was so still, and sounds come so far; and by and by you could see a streak on the water which you know by the look of the streak that there’s a snag there in a swift current which breaks on it and makes that streak look that way; and you see the mist curl up off of the water, and the east reddens up, and the river, and you make out a log-cabin in the edge of the woods, away on the bank on t’other side of the river

Mendelsund remarks that the accumulation of detail in the passage doesn’t help one “see it all”: “I saw the dull, line, and then the spreading paleness, and then I heard a screaking, and then voices, and then I saw the current…” The passage has a rhetorical power, he says, not a combinatory visual power. He is right about the rhetoric, but it is strange that he presents this as somehow a disappointment. To me it seems a triumph. What we have is a description of drifting down a river where things do come at you one after another, not all at once (the section he omits clinches that). As we read it we enter into Huck’s distinctive voice, his earnest wakefulness, his constant concern that the river will throw up some unhappy surprise. A powerful suspense is generated (the passage continues for a couple of pages) in a manner that reminds us that fiction began in the oral tradition of the spellbinding voice, the voice that allows us to entertain the illusion that we have seen things we have not seen and heard things we have not heard, and in general participated in the experience of someone we never met except through this dazzle of words on the page, which are the only things we ever truly see when we read.

Peter Mendelsund’s What We See When We Read is published by Vintage.