Writing well about sexual intercourse is not easy, and it has got more, not less, difficult as virtually all legal restraint on the explicit description of sexual acts in literature ceased in most western democracies from the 1960s onwards. Censorship in the past was often oppressive and silly, but it also presented a stimulating challenge to writers, to which they responded by finding subtle ways to convey their meaning indirectly, or more courageously by gradually testing and extending the limits of socially acceptable explicitness. There are a limited number of possible sexual acts. Now that the novelist is expected to describe them, how does s(he) achieve originality? How to avoid revealing, or appearing to reveal, his/her own sexual predilections and fantasies?

Readers have always been drawn to novels – even those as reticent as Jane Austen's – to satisfy a curiosity about how sexual love is experienced by other people; those who wanted more explicit information, and perhaps a shortcut to sexual pleasure, turned to pornography. But in our permissive culture there is no difference in referential explicitness between literary and pornographic fiction. The essential difference is structural. Any novel with erotic content is likely to be arousing to the reader and can be used as pornography, but pornography as a genre is wholly dedicated to that end, characterised by a high ratio of sexual description to total narrative content, and its arrangement in a scale of increasing sensationalism. In that respect, the Bad Sex award, which justifies its decisions by citing very brief extracts from the novels in contention, can do an injustice to serious writers. There is no easier way to disparage a novel, as reviewers often demonstrate, than to quote some intensely written passage without taking into account its context, especially if it is about sex.

Vladimir Nabokov played a crucial role in the cultural shift I have described. Lolita was a taboo-breaking book, which caused great controversy when it was first published in 1955 – in France, since no American or British publisher would touch it for fear of prosecution – and for some years after. It was soon banned in France, and its importation into Britain forbidden. The acclaim the novel received from writers and critics, starting with Graham Greene's selection of it as one of his books of the year, eventually overcame protests that it was pornographic, and it was published in the US in 1958 and in Britain in 1959. This was a landmark in the liberation of writers from censorship, consolidated in the following year when Penguin was prosecuted for publishing Lady Chatterley's Lover and acquitted. Nabokov himself emphatically denied that Lolita was pornographic on the grounds of both its structure and style, but he deplored much of the sexually explicit literary fiction that was published in the changed climate of opinion his novel helped to create. Lolita still provokes disapproval and controversy not because of the explicitness of its sexual passages, which are restrained by today's standards, but because they describe a relationship between a mature man and a girl well under the age of consent, this being one of the few kinds of sexual behaviour that are almost universally anathematised and punished by law.

A new book, Nabokov's Eros, written by Maurice Couturier, the leading authority on the novelist in France (and, I should declare, the translator of several of my own novels) handles this issue, and similar ones raised by Nabokov's oeuvre, frankly and with discrimination. There is little doubt that sexual attraction to a certain kind of prepubescent girl was an important element in Nabokov's sexuality, whether or not he ever acted on it, and the novelist was well aware of the risks he was running in giving imaginative expression to this kind of sexual desire. He hesitated to publish Lolita, and at one point nearly destroyed it. Having taken the plunge and released it into the world, he tried to defend himself against the charge of encouraging paedophilia by inserting an exculpatory foreword to the novel by the fictitious psychiatrist John Ray, and by adding to later editions an afterword in his own name with the same purpose. Couturier exposes flaws and an element of self-deception in these arguments and in the obiter dicta on the subject that Nabokov uttered subsequently, and wishes he had adopted a different stance:

No matter how hard he tried to find an appropriate line of defence for his novel, he refused to admit that this novel, eminently poetic to be sure, was still highly erotic. This word, with its obvious etymology, simply means "of or pertaining to sexual love; amatory, esp tending to arouse sexual desire or excitement" (Shorter Oxford Dictionary). Nabokov would have been well advised to use it for it perfectly fits the general tone of his writing, both in Lolita and Ada, which is never vulgar, never makes use of crude words, and constantly remains at an eminent poetic or poerotic level. Though the stories he tells are immoral, the aesthetic eminence of the scenes he describes transcends their sexual and ethical dimension.

Vladimir Nabokov in 1964. Photograph: Horst Tappe/Getty Images

The poetic quality of Nabokov's prose in his rendering of transgressive erotic experience is certainly crucial in assessing his achievement. How to defamiliarise the limited repertoire of sexual acts was never a problem for him, because of his extraordinary stylistic virtuosity, his metaphorical inventiveness and mastery of an entire thesaurus of other rhetorical devices, which make other novelists (this one anyway) gape in admiration and feel inadequate, whatever his subject. Couturier is surely right to trace in Nabokov's style the influence of French libertine fiction, in which elegant variation and metaphorical figures of speech were applied lavishly to sexual parts and acts. The aesthetic pleasure of decoding the tropes and noting their felicity is an essential part of the pleasure of the erotic text – for example, the description in Ada of the teenage Van's second erection in the chapter describing the first consummation of his love for his half-sister, the young heroine attentively watching, "cheek on fist, the impressive, though oddly morose, stirrings, steady clockwise launch, and ponderous upswing of virile revival". There is humour as well as sensuality in this prose, which distinguishes it from typical pornographic discourse – an argument I don't recall Nabokov himself using in self-defence. It is present in the scene in which the young Lolita squirms provocatively on Humbert Humbert's lap while, Eve-like, eating an apple, until "my moaning mouth, gentlemen of the jury, almost reached her bare neck, while I crushed out against her left buttock the last throb of the longest ecstasy man or monster had ever known". Neither the humour nor the poetry would, of course, prevent a paedophile from using this passage as pornography, though it is unlikely that such a reader would persevere with the novel to its sad, downbeat conclusion.

Nabokov did not always write about sex in heightened language. The intercourse of characters whose sensibilities are coarse and selfish is described in an appropriate style – for example, Martha and Franz copulating in the early novel King, Queen, Knave: "baring her thighs, and not bothering to lie down, and revelling in his ineptitude, she directed his upward thrusts until they drove home, whereupon, her face working, she threw her head back and dug her 10 nails in his nates". This is explicit, but not erotic: both the diction and the syntax distance readers from the action, rather than drawing them into an imaginative involvement in it. The language of sexual description in Pale Fire is also much less poetic than that in Lolita or Ada, but for a different reason, noted by Couturier: it is the only novel in which the central character is gay. There is no avoiding the fact that Nabokov was prejudiced against homosexuality, and it was fortunate for his academic reputation that queer theory did not emerge until well after his death (fortunate perhaps for queer theory too, if one imagines his likely reaction to it). The central character of Pale Fire, however, is double: the romantic figure of King Charles, the exiled king of Zembla, is much more sympathetic than the deranged Kinbote who invented him as a compensatory fantasy, and Charles's homosexuality is treated with humour and pathos, so the overall effect is not simply homophobic.

Couturier accurately describes his critical method as that of "a reader and a quoter, a bit like Nabokov in his lectures on literature". His book escorts us through the novels and stories with a mixture of quotation and summary, not in chronological order but grouped together in various sets: for example, fictions in which sex is linked to sadism and cruelty, fictions that demonise women, fictions in which the main character is signally lacking in sexual desire, and so on. This has the effect of postponing the extended consideration of Nabokov's major works till the latter part of the book, so that it does not end anticlimactically with minor ones such as Look at the Harlequins! and The Original of Laura, but with Pale Fire, Lolita and Ada. (Readers who were disappointed by the last of that trio may be persuaded by Couturier's enthusiasm to give it another chance.)

The result is both enjoyable and revelatory – an exhilarating, speeded-up journey through Nabokov's oeuvre. That the route follows his treatment of sexuality needs no defence: it was a subject of absorbing interest to him and of central importance in his work. It is not surprising that this work is often disconcerting and disturbing because, as Couturier points out, sexuality is such a dangerously powerful force in human life that its treatment in literature is always likely to have ambivalent effects on culture and society. He concludes:

The novelist wants to give a free rein to his desires but claims at the same time that he should not be blamed for the sins committed by his protagonists. There is a great deal of bad faith involved here, not only on the part of the author but also of the institutions, the critics and the readers. The modern novel is a powerful machine that inextricably binds the fate of all those who are involved in its composition, its circulation, its suppression and its consumption. No writer better than Nabokov can help validate such a theory of modern fiction.

I take this to mean that the transcendent effect of a beautiful style has its limits, and we must all accept responsibility for the freedom of expression we have obtained.

• Nabokov's Eros and the Poetics of Desire is published by Macmillan.