They failed with Lolita (twice) and now they've failed with The Luzhin Defence. Why do the novels of the great prose sorcerer Vladimir Nabokov always defeat the film-makers?

Great literature has always been traduced and eviscerated on screen. Yet the peaks of imaginative writing seem to represent an irresistible challenge for film-makers. Now Dutch director Marleen Gorris, who previously adapted Mrs Dalloway with Vanessa Redgrave, has taken on a book by the 20th century's greatest sorcerer in prose, Vladimir Nabokov. Many film-makers have tried to mine the wily Russian, and this new attempt revives the question: can Nabokov ever be filmed successfully?

Gorris has chosen Nabokov's novel The Luzhin Defence. The author wrote that out of all his novels written in Russian, it "radiates the most warmth". Alexander Luzhin is a chess player competing for the world title in a small Italian lakeside town in 1929. Staying at the same resort is an aristocratic young woman, Emilia, who becomes fascinated by the shambling, stuttering genius.

Rejecting a more refined suitor, she falls in love with Luzhin and they decide to marry. But Luzhin is made anxious by the sudden reappearance of his old "chess father", Valentinov, the man who nurtured him as a prodigy but then abandoned him. As he draws closer to the final showdown with his arch-rival, Turati, Luzhin suspects that Valentinov has an interest in seeing him lose.

All the major incidents from the novel are present in the film. As Luzhin, John Turturro gives a remarkable performance of distracted innocence and childlike trust; Emily Watson, too, is excellent as Emilia. In many ways, Gorris's film is the Nabokov adaptation that gets closest to the spirit of its source. But inevitably it pickles the butterfly of Nabokov's conception and leaves it inert, pinned under a glass screen.

Why should this be so? A clue is to be found in Nabokov's 1943 short story, The Assistant Producer, in which the narrator draws a lugubrious parallel between cinema and life, both of which mock the unwary with fatal coincidence. "Indeterminism is banned from the studio," he writes. That is precisely it: the cinema simply cannot maintain creative ambiguity. How do you preserve the master's playful indeterminism when a movie must show one thing or the other?

In The Luzhin Defence, for instance, all the chess games are described through metaphors of labyrinths and music. (Though Nabokov himself was an accomplished composer of chess problems, it clearly would have been rebarbative to describe actual games in technical detail.) So here is the hero searching for a move, halfway through his duel for the world championship: "A kind of musical tempest overwhelmed the board and Luzhin searched stubbornly in it for the tiny, clear note that he needed in order in his turn to swell it out into a thunderous harmony."

You can't film that. Indeterminism is banned. You have to show the chess pieces moving and their dance must be credible to the chess-literate. The film-makers therefore turned to Jon Speelman, British grandmaster and chess correspondent for the Observer, who provided the sequences of the moves that Luzhin plays in his games, in particular a terrific rook sacrifice that he conceives in the climactic game. It is a splendid half-success.

Only a half-success, note, because the film cannot do anything with the complementary way in which chess operates in Nabokov's design: as a fatal pattern creeping into Luzhin's perception of the world and life. Lampposts, bushes and bathroom floors transform into malign knights, pawns and impossible chessboards: they drive him, eventually, mad. Such things cannot be shown on celluloid with any grace.

As Nabokov wrote it, The Luzhin Defence is a tour around the unhinging mind of a genius. The reader inhabits Luzhin's consciousness as his own. Luzhin in the film, however, is necessarily externalised, reduced to a collection of mannerisms, no matter how beautifully orchestrated by Turturro. So Gorris's film is forced to switch narrative focus: it is really about not Luzhin but Natalia. She becomes an anachronistically free-spirited modern woman, faced with a choice between two men and following her heart despite the disapproval of her parents.

Unfortunately the role of Natalia cannot support this extra weight. Emily Watson clearly struggles to live up to being the film's emotional centre. What was a beautifully structured narrative of mental drama becomes a rather over-familiar costume romance, pillowed by a swooningly sentimental epilogue that has nothing to do with Nabokov's novel.

Yet no film-maker approaching the Russian's work has done much better. Stanley Kubrick called his own 1962 version of Lolita, starring James Mason as Humbert Humbert, his "one manifest failure". Nabokov himself was hired to write the screenplay, and duly delivered a 400-page manuscript to Kubrick, almost none of which the director used - understandably, as to film a script of such length would have resulted in a six-and-a-half hour movie. (Nabokov retained, however, the sole screenwriting credit, and was nominated for an Oscar.)

Nabokov was generous about the film in public, calling it "charming". He particularly admired the scene, invented by Kubrick, in which James Mason drinks whisky in the bath after the death of Lolita's mother, giggling to himself while receiving the consolation of mourners. But he remained nostalgic about his own version of the screenplay. "The film is only a blurred, skimpy glimpse of the marvellous picture I imagined," he said later. "I do not wish to imply that Kubrick's film is mediocre; in its own right, it is first-rate, but it is not what I wrote."

Kubrick himself eventually acknowledged the major reason for what he saw as the film's failure. He regretted, he said, his "timidity in translating a novel whose glory was its unique narrative voice". Indeed, the playfully wheedling, super-refined sensuality of Humbert Humbert's prose is something that cannot be filmed. You certainly do not solve the problem by adding a voiceover, as Adrian Lyne did in his soporific 1997 Lolita: Jeremy Irons strikes one vocal attitude, of wistful, monotonous flatness, and holds it throughout the entire excruciating movie.

Nabokov himself evoked his adapters' problem in one marvellous Russian word: "poshlost". The word means, as he explained, "corny trash, vulgar cliches, philistinism in all its phases, imitations of imitations, bogus profundities". Speaking of Kubrick's Lolita in an interview, he asserted sadly: "A tinge of poshlost is often given by the cinema to the novel it distorts and coarsens in its crooked glass. Kubrick, I think, avoided this fault in his version, but I shall never understand why he did not follow my directions and dreams."

Poshlost is also the best diagnosis for other cinematic failures such as Tony Richardson's 1969 film of Laughter in the Dark, in which the action of Nabokov's novel, set in 30s Berlin, is translated by playwright Edward Bond to 60s Swinging London, to egregious effect. Of this film, Nabokov would only say tactfully that Nicol Williamson (who played Albinus) was "an admirable actor", and that the sex scenes were "commonplace". Jerzy Skolimowski's lurid film of King, Queen, Knave (1972), starring David Niven, fared no better. In his excellent book on Kubrick's Lolita (in the BFI Film Classics series), Richard Corliss excoriates it as "coarse", "inhuman" and "clumsy".

The most graphic illustration of the film-maker's impotence in the face of Nabokov's spine-tingling "indeterminism", however, comes in Fassbinder's 1978 film of Despair (screenplay by Tom Stoppard). My explanation of this, I'm afraid, has to give away the novel's ending, so if you haven't read it, look away until the end of the paragraph. Still here? Right. Dirk Bogarde plays Hermann Hermann, a chocolatier in prewar Berlin who is bored with his wife. On a business trip abroad, he spies his identical double, and plots to switch lives with him by killing the other man and stealing his identification. The novel's demonic brilliance lies in the fact that only right at the end do we realise that Hermann's "double" in fact looks nothing like him: the narrator is simply insane, and he is promptly arrested by the police. In the film, however, the "double" has to be shown. He is played by another actor who looks nothing like Bogarde, and we therefore know that Hermann is mad from the inception of his plot. All the story's dramatic tension is lost.

What film-maker, indeed, could possibly have avoided all these pitfalls, brought Nabokov's butterflies intact to the screen in all their iridescent, fluttering glory? Well, perhaps there was one. It is to be filed among the great cinematic collaborations that never came to fruition: Nabokov meets Alfred Hitchcock.

It is known that the two men admired each other's work. Nabokov commented approvingly that Hitchcock's "humour noir" was rather like his own. The two men corresponded, and Hitch asked Nabokov to write the screenplay for what eventually became his 1972 film Frenzy. The novelist, unfortunately, had to decline owing to his work commitments. They never did work together, and Nabokov died in 1977. The rest, as they say, is poshlost.

 The Luzhin Defence is released on September 8.