The Culture, the post-scarcity, hedonistic, Machiavellian, libertarian, ass-kicking science fiction society created by Iain M Banks, is 25 years old. Over eight novels and one short story collection, Banks has created one of the most enduring and endearing visions of the future, and the publication of the ninth Culture novel, The Hydrogen Sonata, confirms his pre-eminence in the field. That said, the success of the Culture novels is rather surprising. Conventionally, a novel is driven by threat, and the Culture – infinitely capable, technologically miraculous, polymorphously perverse – seems capable of fending off each and every threat that comes its way. Consider Phlebas, the first in the series, established the moral paradox that fuels the whole series. How far will an ultra-liberal society go to defend its liberties? While most of the Culture happily switches genders and ingests safe drugs, the Minds – artificial intelligences that run the show – secretly ensure this utopia through the pleasingly euphemistic "Special Circumstances" division. In part, they genuinely believe in their mission. In part, it just staves off the boredom of being the galaxy's apex predator.

Looking back over all the books, it becomes clear that the dual threat to the Culture is not just anomie, but religion. Consider Phlebas was set during the Culture-Idiran war, where the Idirans refused to renounce their religion and set themselves on a self-destruct course against the Culture. In Excession, the Minds confronted a perfect black sphere that was older than the universe and refused to conform to their ideas of reality; a god to the godless. Look to Windward developed ideas that had been in the books from the start: some societies eventually "Sublime" and leave this metaphorically sublunary sphere; this particular novel involved a society which had partially Sublimed, caught between the ineffable and the actual. Matter was set on an artificial planet, shaped like a series of concentric spheres, at the centre of which was a "god". Surface Detail extended the notion of immortality through back-ups of the self: some societies, to the Culture's pursed annoyance, were condemning people to virtual hells.

In the new book, the Gzilt civilisation is preparing to Sublime. They were almost a founding member of the Culture, but declined to join at the eleventh hour; in part, their decision was founded on the fact that of all species in the universe, theirs was the only one whose religious texts had been proven to be completely and utterly true. As other races come to bid farewell to the Gzilt, one delegate, bearing a message, is obliterated. That's enough for the Minds to take an interest in what the unfortunate individual was going to reveal.

Tracking down why a society about to leave this reality would commit a murder leads the Minds to a Gzilt woman called Vyr Cossont, who is spending the remaining days of her society playing the musical piece which gives the book its title (one character caustically says that the piece should ideally be played in a vacuum so that no one can hear it; another calls it "as a challenge without peer. As music, without merit"). She is exceptional only in that she once met the oldest living member of the Culture, a man who has been hiding from their benevolent overlords for so long that he is almost a myth. He might remember what the Gzilt are so keen to cover up, and given that he has written his memories on every part of his anatomy, they need to find not just him but every part of him.

The sheer ebullience of imagination counteracts the Culture's inherent lassitude, and here Banks suggests, more than in the other books, that there is something wanting about the Culture. Why don't they Sublime, for example? The fact that the Minds in this book – always wittily named and more interesting than the humanoids – fancy themselves as a modern-day equivalent of the "Interesting Times Gang" that dealt with the Excession reinforces the sense that utopia can be very boring. The Culture's obsession with the Sublime is done particularly well; the Mind which, uniquely, Sublimed and then came back is a melancholic still point in the hectic action. I hope the next book deals with the "smatter" Banks introduced in Surface Detail: ''self-replicating entities [that] ran out of control somewhere and started trying to turn the totality of the galaxy's matter into nothing but copies of themselves". It is the dark mirror of the Culture's aggressive moral superiority. The Culture is trapped between two unilateral conformities; transcendence and materialism.

Banks can riff like no other science-fiction writer. The description of the party the Gzilt are having as a prelude to Subliming is gloriously baroque and silly; the anxieties of the officials display a keen satire and a keener anger. The protagonists of the Culture novels have almost always been those on the fringe of the Culture itself, and there is a moral empathy with the refuseniks and the renegades that subtly undermines the Culture's disavowed aggrandising ambitions. If there is an arc for the whole series, it is that trouble in paradise comes from paradise, not from a snake conveniently dropped in to cause trouble.

For all that the books are epic in scope and derangingly replete in detail, The Hydrogen Sonata reiterates a key theme: that personal fulfilment, in whatever form that takes, is rare and to be treasured. Vyr Cossont's commitment to playing an almost unplayable piece of music is contrasted with the Mind who can do it immediately, flawlessly and without effort. Banks is always on the side of those trying rather than those succeeding, and that, ironically, is his great success.

• Stuart Kelly's The Book of Lost Books is published by Polygon.