Nothing makes a novel seem more vulnerable, more naked, than an armour-plating of literary references. If you're constantly referring to landmarks, it doesn't make you look as if you're striding confidently forward – it makes you look lost. In a 20-page section of Michael Cunningham's new novel, By Nightfall, in which the hero Peter Harris, an art dealer, visits a faithful client, there are explicit references to: The Magic Mountain, John Cheever, Death in Venice, Donald Barthelme, Raymond Carver, Hawthorne and Death of a Salesman. As if that was not enough (there is no shortage of allusions here to visual art, but they at least are relevant), there's a further cluster of references to The Great Gatsby, first to Dr TJ Eckleburg (who appears on an advertising hoarding in the book), then Gatsby himself, Daisy Buchanan and even Myrtle Wilson. All well and good – but Gatsby didn't get to be Gatsby by dangling dozens of previous books behind it, like tin cans tied to a tricycle. The scene never escapes from the net of its own allusions, and all the writer's talent goes for nothing.

By Nightfall tells a story of midlife crisis averted, when Peter is sexually tempted by his brother-in-law (young and messed up, with a certain amount of bisexual history). Cunningham is a gay writer who wants, as so many of us do, to achieve a mainstream readership – he succeeded with The Hours – but here he stacks the odds against himself. Gay male readers tend to feel that their lives haven't been squeezed dry by novelists, not just yet, and may resist even a convincing portrait of a heterosexual man. Others may feel cheated when Peter finds he's not quite as straight as he thought.

The book's pages are filled with thoughts about art, or (more ominously) Thoughts about Art. Since its action occupies little more than a day, the effect is highly artificial, an avalanche of compacted insights, so that Peter can see in his wife's tired beauty in the morning light "a deep, heartbreaking humanness that's the source and the opposite of art". Even when these are golden formulas – like that one – they are leaden as moments, making the narrative degenerate into a string for wise and lovely beads.

There are writers who can describe visual art while honouring its resistance to language – Updike is the obvious example, though his name is missing from the roll call of allusion (perhaps it's the absentees who are the significant figures). Cunningham's alignment remains strongly literary. When he has a conversation between Peter and an old friend, recently diagnosed with cancer, taking place by Damien Hirst's shark in formaldehyde on display at the Metropolitan Museum, it's clearly the work's title that earned it its place: The Physical Impossibility of Death in the Mind of Someone Living.

By quoting on his first page the opening phrase of Ulysses ("Stately, plump Buck Mulligan…") Cunningham aligns By Nightfall with Joyce, just as The Hours was aligned with Mrs Dalloway. It seems to be the prestige of the modernists he admires, rather than their stringency. If he had chosen softer models he would cut a better figure, the contrast being smaller. The sweet, engrossing, middlebrow braid of The Hours would have dismayed Virginia Woolf, its ostensible linchpin, as a serious venture in fiction, and Joyce would have begrudged By Nightfall the rationed reading time (failing eyes) he gave so willingly to Gentlemen Prefer Blondes.

There are some writers you shouldn't challenge if you can help it – as Flannery O'Connor remarked about Faulkner's superior power, "nobody wants his mule and wagon stalled on the same track the Dixie Limited is roaring down". You'd better think twice, while singing your own hushed song (" ("They do, of course, each of them carry within themselves a jewel of self, not just the wounds and the hopes but an innerness…"), before you hand the microphone on to more established performers: "What did Shakespeare say? Our little lives are rounded with a sleep."

Towards the end of his book Michael Cunningham borrows from yet another author: "Silly humans. Banging on a tub to make a bear dance when we would move the stars to pity." He repeats the sentence word for word 20 pages later, and again five pages further on. This time he adds his own gloss: "It's something, though – it isn't nothing – to have a tub to dance to. Not if you're a bear." Perhaps he feels that Flaubert missed a trick by using the sentence just the once in Madame Bovary, where its subject is "language", or human speech, and the "tub" usually translated as a cracked kettle.

At the very least, shouldn't a writer try to shield the kettle of language from further cracks by knowing the meanings of the words he uses? In one scene of By Nightfall, Peter has returned to his apartment and overhears his brother-in-law Ethan phoning a drug dealer. Peter doesn't announce his presence and continues to eavesdrop. Any erotic tension that might develop is thrown away when the word "prone", wrongly used ("He and Peter are now about four feet apart, separated by a wall made of high-tech cardboard, both prone"), indicates that they're both face down, though at least one of them is masturbating. Physically possible, but demanding a giggle.

Peter is lucky or unlucky enough to have had an epiphany as a boy, admiring his handsome older brother, about to be 16, and a beautiful female friend as they wade in the shallows of a lake. The idea of the epiphany is Joyce's great contribution to literary aesthetics, though Cunningham uses the more Woolfean term "vision".

"It's a pure, thrilling, and slightly terrifying apprehension of what he will later call beauty, though the word is insufficient. It's a tingling sense of divine presence, of the unspeakable perfection of everything that exists now and will exist in the future…" Except it's so fatally pretty and second-hand, modelled on the similar moment in Joyce's Portrait, of a girl standing in the shallows looking out to sea. Flaubert times three is better than Flaubert just once, right? And a two-person epiphany has to outrank the single kind. Two comely young people standing in the lake shallows, "looking out at the milky haze of the horizon" – that's not an epiphany, that's a postcard.