The first thing we know about Madeleine Hanna is her library. "To start with, look at all the books," Jeffrey Eugenides suggests of his heroine, and proceeds with a tracking shot of her shelves: "A lot of Dickens, a smidgen of Trollope, along with good helpings of Austen, George Eliot and the redoubtable Brontë sisters… the Colette novels she read on the sly… the first edition of Couples, belonging to her mother, which Madeleine had surreptitiously dipped into back in sixth grade…"

Madeleine Hanna is an English major at Ivy League Brown University in 1982. Her thesis is concerned with "the marriage plot" as it existed in the 19th-century novel and the way, with marriage having lost its gravitas in her era of quickie divorces and prenups, the novel itself has been diminished. Much as Madeleine may believe this thesis as a critic, however, as a 20-year-old woman there is much about her life that seems Victorian. She is, cliche of cliches, caught in a love triangle herself, torn between two fellow undergraduates: the charismatic and depressive Leonard Bankhead on the one hand and the studious and spiritual Mitchell Grammaticus on the other. Her heart shouts Leonard (most of the time); her head and her Waspish parents murmur Mitchell.

As well as locating the style of Madeleine's dilemma, Eugenides's opening tracking shot of those library shelves is also a nudge to the reader: this is the territory we are in. And here is the challenge he sets himself: to breathe new life into the redundant marriage plot; to create a properly absorbing love triangle, not only as pastiche or irony, but as something as full of life as those books on Madeleine's shelf. In the 400-odd pages that follow he mostly succeeds in this aspiration, both knowingly and brilliantly.

This is Eugenides's third novel. It is 18 years since the precocious and perfectly formed The Virgin Suicides marked him out as a writer who would always be required reading. In between times, the fabulous family saga Middlesex, which, along the way, told of the unlikely coming of age of a hermaphrodite in Michigan, became a huge bestseller and Pulitzer prize-winner, without ever seeming entirely coherent.

The tight plotting and internalised psychology of this new novel, allied to the full sweep of ideas and social observation and quiet comedy that characterised Eugenides's earlier works, are signs of a new maturity. In the generosity and nuance of his characters and paragraphs, you are reminded of the Jonathan Franzen of The Corrections. Like that novel, this one acknowledges the brio and experimentation of American writers such as Thomas Pynchon or Don DeLillo or the late David Foster Wallace; it takes on board some of the philosophical caveats to "conventional" social realism, but does it anyway.

The book begins not only with Madeleine's love troubles, but also with the fact that those "love troubles had begun at a time when the French theory she was reading deconstructed the very notion of love". Her heart breaks even as she sits in seminars discussing Derrida on the bogus nature of romance and sentiment in life as well as literature. When she finally tells the manic-depressive polymath Leonard that she loves him, after a perfect day, he gets up from the bed in his ratty student room and starts quoting Barthes at her: "The figure [je t'aime] refers not to the declaration of love, to the avowal, but to the repeated utterance of the love cry. Once the first avowal has been made 'I love you' has no meaning whatever..." Quite understandably, she immediately chucks the book at him and chucks him. And then immediately regrets it (but not as much as Leonard, who ends up hospitalised with psychotic depression).

Eugenides inhabits the minds of each of the points of this love triangle in turn. The Greek-American Mitchell Grammaticus, who, like the author himself in his youth, volunteers as a gap-year helper with Mother Teresa in Calcutta as he searches conflictedly for spiritual enlightenment, seems very close to home for Eugenides. Madeleine is entirely believable as the ambitious, beautiful and mostly moral young woman slightly out of step with the freedoms of her time.

Leonard Bankhead, though, is both the wild card and the proper heart of his novel. Eugenides handles the difficulties of describing the manic phases of depression from within – through Leonard's eyes – with rare skill, the gradations of self-delusion measured almost incrementally in his prose as Leonard seeks to take control of his teeming thoughts and reduce his lithium dosage. The reader is asked to become just as hyper-aware of the character's mental state as those, particularly Madeleine, who try to protect him from himself. It is a highly affecting portrait that brings to mind some of those Salinger stories that walk the line between reality and mania. There were times in the book when this reader was so alarmed at the first hints of impending lapses in Leonard's behaviour that he felt like calling out his alarm to the other characters.

As he delineates these fracturing lives, Eugenides also pursues cogent inquiries into religion and philosophy and sexuality as his young trio try to make sense of things. (Leonard, who takes up postgraduate work in biology at a genius lab in Cape Cod, also brings with him some scientific insight, notably into the mating rituals of microscopic organisms.) Though the absence of email and mobile phones allows the author to explore the proper frustration and novelistic suspense of airmail letters and poste restante boxes for perhaps the last time, there is much that feels contemporary about the moment of the book: Madeleine and the others are graduating into deep recession; there seems no new idea under the sun.

For much of the novel, the suspicion is that while the disturbing pull of lust and love thrums beneath all of Mitchell's distracting quest for spiritual truth, and Leonard's unhinged normality, and Madeleine's pursuit of happiness – that the marriage plot is still the most important plot of our lives. The novel's resolution allows that possibility, but also reminds us that we are not quite Victorian after all.