The Quantum Thief, Rajaniemi's first novel, caused quite a stir in SF circles. In it the legendary thief Jean le Flambeur is sprung from a deep space prison by enigmatic Oort-cloud warrior Mieli and her Iain-M-Banksy sentient spaceship Perhonen to mastermind a seemingly impossible heist. But it wasn't the story so much as the flavour of the book that was so impressive: an SF narrative so hard you'd need a sclerometer to measure it, but written with a sinuous, fin-de-siècle intensity.

Now we have its sequel, The Fractal Prince, a direct continuation of The Quantum Thief in both plot and tone. Once again we start with the hero in a literalised version of a classic thought-experiment (in the first book it was the prisoner's dilemma; in this one Schrödinger's cat). Again Le Flambeur must use his wits against hostile artificial intelligences, competing posthuman clans and virus-infected virtual realities. And again the reader must work through prose as stuffed with neologisms as a chocolate chip cookie is with chocolate chips: Zoku jewels, pellagrinis, soborhost bodies, utility fogs, qutlinks, spimescapes and guberniyas. None of it is glossed; you work out the meanings from the contexts. Or you don't.

When setting out their worlds some SF writers fall back on the "As you know, Bob …" manoeuvre: having one character explain to another something they both already know about their world. It's a clumsy business (you don't find it in contemporary-set novels, after all: "As you know, Bob, a car is a four-wheeled engine-powered vehicle used to transport people from place to place …"). But it happens because SF writers need somehow to get the reader up to speed with a world which is, by definition, unfamiliar. To his credit, and the reader's occasional bafflement, Rajaniemi repudiates this strategy altogether. Despite writing about a far-future considerably stranger than most, he simply throws his reader in at the deep (space) end. It's hard going.

That said, Rajaniemi's prose can be unpacked, with a little effort. Here's the narrator examining a robotic assassin: "I wonder at the intricacies of its synthbio cells in a diamonoid frame, the fusion power source at the base of the spine and the nifty q-dot transmitters" – comprehensible enough, even if we have no idea what q-dot transmitters actually look like, nifty or otherwise. But at its worst the prose becomes positively indigestible. "The q-bubble struggles to keep up with the barrage it is taking across the electromagnetic spectrum and switches to neutrino tomography around the Bekenstein epicentre." This brings, I confess, no images at all to my mind.

Not that any of this is mere technobabble. Rajaniemi has more degrees in higher mathematics and quantum physics than you could shake a stick at, and the consensus from people who know about these things is that all the elements of his world are scientifically spot-on. But they still sit oddly in a book written less like conventional hard SF and much more like verbal art nouveau (and isn't that French phrase a splendid epitome of what SF ought to be?) Here, for example, is Le Flambeur in a virtual reality: "I am standing in the middle of a white forest. There are straight trees with pale, birch-like bark and impossibly symmetrical foliage shaped like crowns or hands in prayer." As writing that's striking, evocative and, frankly, more like it.

The bottom line is that Rajaniemi is a baroque writer. By that I mean not just that his work is characterised by elaborate details and conceptual curlicues – although it is. I mean he writes by folding abstruse science and mannered artistry in architecturally intriguing ways. Now "baroque" can do many things really well, but it can't really do the sublime, that sense of wonder evoked by the vastness of space or the dizzying perspectives of deep time: the stuff that makes the hairs prickle at the back of the neck. When Rajaniemi describes "an artificial sphere the size of old Earth, made from sunlifted carbon, thinking thoughts bigger than the sum of humanity", his description simply lacks the heft, the mind-wow, that a writer such as Stephen Baxter so powerfully creates. It's "a distant amber bubble in the sky, like a snowglobe": more like a model of something than a colossal actuality. And that's true of the whole novel, I think.

The Fractal Prince addresses some of the largest issues – immortality and death, love and betrayal, the power of story – but it feels more like an intricate, bejewelled device than a deathstar. Often beautiful, it's also, often, rather stiff: a complex but inert piece of art, a mannequin rather than a real boy. But that's as much a strength as a weakness. Thoughtful, hard, densely realised and highly patterned, there's nothing quite like it in contemporary SF.

• Adam Roberts's Jack Glass is published by Gollancz.