In “The Behavior of Love,” Virginia Reeves wants to explore what it feels like when language becomes a door swinging closed on meaning. Photograph by Philippe Matsas / Opale / Alamy

In “The Behavior of Love,” a new novel by Virginia Reeves—whose 2016 début novel, “Work Like Any Other,” was long-listed for the Man Booker prize—marital emotion boils within a cool experimental framework. A brilliant doctor, Edmund Malinowski, has taken a job as the superintendent of a mental institution in Boulder, Montana. It’s a disorganized, underfunded place, but Ed is slowly transforming it. As a behaviorist, he believes in observable actions; his wife, Laura, who is an artist, wishes her husband would pay more attention to their marriage. Ed has an obvious crush on a sixteen-year-old patient with epilepsy, Penelope, who begins to fall for Ed in return. Many books that feature scientists as main characters show them overthinking and overanalyzing, using logic to flee their desires—“Lost and Wanted,” by Nell Freudenberger, is a recent example. Reeves inverts this trope: Ed, for all his training, doesn’t seem to think enough.

Ed has a specific kind of postwar masculinity—an entitled charisma. He is a figure of natural authority from the start, while Laura and Penelope’s arcs must bend toward action and agency. Laura’s sections—which are told in the first person, while Ed’s unspool in a close third—offer up her feelings with a frankness that is affecting precisely because we know her husband isn’t listening. In positioning Penelope and Laura as foils, the book draws a subtle distinction between ministering to a person and ministering to a relationship. To treat your spouse as your patient would be condescending, withholding; but to treat your marriage as a patient might save it.

For much of the book, Ed wrestles with his responsibilities to, and genuine enthrallment with, both Laura and Penelope. (Their names, evoking Petrarch and Homer, make them love objects from duelling literary traditions, and “Ed” sort of sounds like “Id.”) Meanwhile, Laura inveigles her way into teaching an art class at the hospital, with the hope of entering her husband’s life by a different door. The front half of the novel is littered, somewhat mysteriously, with references to untranslatable or nonsensical sounds. (Even the epigraph, from Robert Frost, is about a question that the speaker thinks he hears posed, not in words but in birdsong: “what to make of a diminished thing.”) It is at first unclear why this motif of wordlessness keeps returning, or how it relates to the love triangle. Still, the theme is carefully developed, pulling the reader toward the incomprehensible, the unintelligible, the question of what’s left in language once sense departs. Ed considers the noise of the hospital: “The hall is full of hapless patients, their voices mostly guttural.” Later, Penelope recites a line of poetry while leading a reading group composed of her fellow-wards. Referring to a Philip Larkin lyric about “a voice / Watering a stony place,” she asks, “If the bird’s notes—and this woman’s words—can’t be understood, then what good are they?” Another patient answers that “They are nice to water the stony places. Nice to make the stony places soft.”

The significance of such bread crumbs becomes clear when, in its second half, “The Behavior of Love” reboots dramatically, inducing a kaleidoscopic disorientation that would be familiar to many of Ed’s patients. I won’t spoil the twist, but suffice it to say that Reeves wants to discover her characters’ stony places. She wants to explore what it feels like when the mind breaks, when language becomes a door swinging closed on meaning, and yet she is never so supple or interesting a writer as when she is tracing a character’s fugitive shreds of consciousness: “His brain is a building gutted by fire, walls smoke-licked, ceiling blackened, floor crumbling . . . Every now and then, though, he finds an untouched room, everything just as he left it, and the memories rise up in vivid detail.” But there is not just suffering here—there is also a deep saturation and beauty of experience. “He is warm and alive, summer inside of winter,” one character thinks. The same character reflects on the poetry that Penelope reads aloud: “glowing words that moved on their own, more about sound than meaning, water over rocks, wind in branches—something not said but felt.”