Aside from being an essayist, novelist and screenwriter, Gore Vidal was a grade-one gossip, his published writings and private conversation stuffed with rumour and conspiratorial hints. But now, less than 18 months after Vidal passed into history, he is himself subject to the most reputation-destructive scuttlebutt there is: the suggestion that he may have been a paedophile.

In connection with a new book about Vidal, his half-sister and a nephew have hinted that he may have had a secret passion for underage boys. And Vidal more or less admitted it himself, writing in his memoir Palimpsest that he was "attracted to adolescent males".

Given this context, an incident in 1977 will also now bring literary cops running. Martin Amis, interviewing Vidal for the Observer, was required to give him copy approval, and was asked to change Amis's description of "homosexual" to "pansexual". Vidal was prissily fastidious about the meaning of words and his suggested substitution can be taken as a confession that nothing and no one was erotically off-limits.

A sexual dirt file allegedly kept by one of Vidal's many enemies, the late conservative thinker William Buckley, is said to have been thrown away after Buckley's death, and so definitive proof will be tricky. It also seems odd that no one claiming to be a victim has come forward, especially as Vidal left an estate of $37m (£23m) – a possible source of ire from his relatives, as he bequeathed the lot to Harvard University, where he had never studied, rather than them.

It's possible, of course, that any young men involved would have had no idea that they were pleasuring the author of Myra Breckinridge and United States: Essays 1952-1992. Vidal spent much of his life abroad, living in Italy and taking regular vacations in Thailand, which must raise the possibility that, like many men with taboo sexual desires, he satisfied them in regions with looser laws.

But, even if definitive evidence of Vidal's pederasty were to emerge, it's unclear what our reaction should be. His defenders in the court of public opinion would cite the Richard Wagner precedent. We know that the composer was a virulent antisemite, whose work became the mood music of Nazism, but it is generally accepted that performers and audiences have a choice over whether this ideological toxicity invalidates his work: great Jewish musicians, including Daniel Barenboim, have concluded that it doesn't.

We must, though, be conscious of the risk of allowing artistic merit to excuse behaviour that would be condemned in the unknown or untalented. In the most testing contemporary example, the films of Roman Polanski, a convicted rapist of an underage girl, continue to be shown to praise from critics, including me. And yet we know that, if Polanski were a children's TV presenter from the 1970s, networks here would not be cheerfully accepting new editions of Roman's Kiddie Funhouse, shot and edited in the foreign exile where he fled to escape justice.

One possible mitigating difference between Vidal and the likes of Jimmy Savile is that it seems clear Savile used (and perhaps even pursued) his chosen profession in order to have access to potential victims, whereas it must be unlikely that Vidal, if guilty, lured his targets by inviting them to help with a New York Review of Books piece on Ambrose Bierce or Edith Wharton. So the work is not directly tainted in the way that it is with paedophile broadcasters or the artist Graham Ovenden, jailed last month for sexual offences against young girls who posed for his pictures.

And, if retrospectively proved encounters with underage males in foreign jurisdictions were to make a writer unpalatable, then there would be many gaps in the literary canon. For instance, Joe Orton's plays could never be performed again, as the diaries published after his death are explicit about why he found Tangiers in the 1960s such an agreeable location for vacations. And then there is the matter of Lewis Carroll, who, if photographing and writing today, would surely not have escaped the attentions of Operation Yewtree.

If Vidal's published sentences on his sexuality are as precise about words as he liked to be, then he almost certainly did have sexual interests or encounters we now consider unacceptable. But if his books were to be metaphorically or literally pulped because of this, then Polanski DVDs must be pulled from stores and productions of Alice in Wonderland banned. Faced with a "pansexual", culture must not become pan-hysterical.