On the first day of this year’s Jaipur literary festival, the American novelist AJ Finn, real name Dan Mallory, was interviewed on stage. He talked about enjoying the success of The Woman in the Window, the thriller he wrote in one year, in one draft, which made him a multimillionaire. He talked about his diagnosis with bipolar II disorder, and the parallel between women’s struggle to be taken seriously and that experienced by people with mental health problems. He also mentioned some of the drawbacks of success. “I am dealing with a particularly unpleasant journalist in the US,” he told news18.com after the event. “This particular journalist, and there have been a few others, hears that I or someone else has a mental health issue, and is like: ‘Oh! I am going to find out what is wrong with you, and rummage through your past and see what you did.’ It’s a little concerning, especially because you don’t remember what you did when you weren’t acting like yourself.”

Six days later, on 30 January, Mallory was able to recall lying to his friends and colleagues for many years about having cancer. “I felt intensely ashamed of my psychological struggles,” he explained in a statement sent to the New Yorker. “I was utterly terrified of what people would think of me if they knew.” Five days later the New Yorker published a profile of Mallory, which alleged or implied that he had also lied, among other things, about having a doctorate from Oxford; his mother dying; his brother taking his own life; and about discovering JK Rowling as a crime writer.

Fiction writers are professional fantasists, so perhaps you’d expect to find a few who can’t restrict their fantasies to the page. In fact, there are so many that it is tempting to wonder whether literary imposture ought to have a syndrome of its own. If so, Mallory, although his lies don’t amount to a crime, belongs in a distinct subtype alongside Jeffrey Archer, who went to prison for perjury and has lied about many minor things. These writers are ambitious and often charming commercial novelists whose gift for storytelling sometimes looks like a skill they have harnessed, sometimes like a compulsion that they can’t control. Perhaps you’d include the crime writers Stephen Leather and RJ Ellory who used online aliases to trash their rivals’ books and praise their own. Generally, their lies do little harm to their careers, because the truth of an author’s life means little to a reader in search of entertainment. Mallory argues that a writer’s life is irrelevant in any case. “I am not especially interested in author’s bios,” he told the Observer last year. “I am buying their novel, not their memoir. I view it as a sign of respect to not want to know too much.”

Facebook Twitter Pinterest Courtney Love, Astor, JT Leroy (later revealed to be Laura Albert), and Speedy in 2003. Photograph: Matthew Peyton/Getty Images

In practice, complete ignorance of the author is impossible. A name alone carries information, even if it is coded in assumptions. Take Rahila Khan, whose stories about British Asian teenagers were broadcast on the BBC, leading to a book called Down the Road, Worlds Away being published by Virago in 1987. Khan was in fact a man teaching RE in a girls’ boarding school. Her real name was the Rev Toby Forward.

Forward had wanted modern writing with a Muslim background to use in his lessons, but found little, so he wrote his own. He wondered if Radio 4 might be interested, but was terrified of rejection so he sent a story under a nom de plume. “It felt like a barrier,” he now says. “I didn’t want them to write back to ‘Toby Forward’ saying, ‘Your stuff is shit’… I chose a name that was appropriate to the material. That was all.”

When the BBC wanted another story, they wrote to Khan, and Khan wrote back. The BBC didn’t like it when she wrote about white boys, so she returned to Asian girls. Forward borrowed his great-grandfather’s name “Tom Dale” for the next white boy stories, which he wrote on a different typewriter, with the postal address of his assistant. The Dale stories were also broadcast, and Forward started to enjoy himself. “I wanted to have a whole week of Radio 4 morning stories, all by me under different names,” he says.

I had women leaving messages on the answerphone, saying: ‘We’re going to take your balls off' Toby Forward

Still, Khan was the star. Listeners loved her work. Khan was encouraged to publish. The only imprints taking this type of short story were the Women’s Press and Virago. Forward hoped that Virago would back out when he refused to meet them. When they didn’t, Khan signed a contract. She was supposed to promote the book, give a radio talk, write an article for the Guardian. There is no good time to admit you’ve been deceiving people, but later is worse. Forward confessed to his agent, who told Virago, who removed the books from sale and pulped them, furious. Forward had a very bad time. “I had women leaving messages on the answerphone,” he says, “saying: ‘We’re going to take your balls off. We’re going to kill you.’” At one stage he was using another pseudonym, a Yorkshireman named Francis Wagstaff, just to answer the phone. If the caller seemed polite, Wagstaff would fetch the reverend.

Critics said that Forward had stolen an opportunity from a real Asian woman. He insists this never occurred to him, nor would it, because he doesn’t see fiction that way. He was writing about being an outsider, a feeling he knew well from his own poor background. The alter egos, as he wrote at the time, “released me from the obligation of being what I seem to be so that I can write as I really am”. When it comes to an author’s life, he agrees with Mallory: “It shouldn’t matter to the reader who this person is.”

Yet sometimes it matters a great deal. A novel about the Holocaust such as The Painted Bird (1965) feels different if you know that it was written by a survivor – Jerzy Kosinski. A novel that makes Germany confront its Nazi past such as The Tin Drum (1959) takes on a new resonance if it comes from a man of principle who was there, such as Günter Grass. It matters that Kosinski secretly used the experiences of others, and that Grass was hiding his own past membership of the Waffen-SS. The same goes for JT LeRoy, who was not a male sex worker writing about male sex work, but the invention of a woman called Laura Albert, and for Émile Ajar, the Algerian émigré who won the Prix Goncourt for The Life Before Us, about a Muslim boy, but was in fact already well known as French novelist Romain Gary, also a Goncourt winner.

Most commonly, novelists pretend that they’re not novelists at all. Indeed those were the novel’s origins, in fake memoirs such as Gulliver’s Travels and Robinson Crusoe. While researching this article, I counted 21 successful memoirists whose books were exposed as substantially false or exaggerated in the past 30 years. (This does not include forgers or plagiarists, or writers of dishonest journalism, such as Jonah Lehrer or Johann Hari.) There’s James Frey’s addiction memoir A Million Little Pieces, which was published in 2003 and exposed in 2006. (Another very famous addiction memoir, 1971’s Go Ask Alice, had been exposed in 1979.) Michelle Smith released a memoir of satanic abuse called Michelle Remembers in 1980. It began a moral panic, and was joined by Lauren Stratford’s Satan’s Underground in 1988, but both books were revealed to be full of inconsistencies. In 2014, Newsweek revealed that The Road of Lost Innocence (2005), Somaly Mam’s memoir of being sex trafficked, contained false information.

Holocaust survival is a sadly common fantasy. Untrue or exaggerated Holocaust memoirs have been written by Binjamin Wilkomirski (in the 1990s), by Misha Defonseca, who included the claim that she was raised by wolves, by Herman Rosenblat (exposed before publication in 2008), and by Enric Marco (2005). When Oprah Winfrey famously reprimanded Frey in 2006, she had already interviewed Smith and Stratford on her show. She has since interviewed Defonseca, Rosenblat (twice) and one of Mam’s fake witnesses.

Facebook Twitter Pinterest James Frey. Photograph: David Levene/The Guardian

Like Mallory, some literary fakers had mental health problems. Some look back with deep remorse. Tim Barrus, who pretended to be a Navajo writer called Nasdijj, will only talk to me by email. “I am reluctant to go anywhere near the whole Nasdijj thing because it brought so many people so much pain,” he says. “I am deeply sorry, and I regret doing it … It was a nightmare I am trying to put behind me.”

If we want to know why people keep doing this, we should ask why we keep believing them. This collection of cases offers an answer. Almost every one claimed to experience more obvious hardship and disadvantage than they really did. They inspired us, because triumph over adversity is the world’s favourite story. We forget that it is also the least likely route for a triumph to take. There would be no successful fantasists if there were no audience for stories that are too good to check.

Memoirists who lie are often in breach of contract with their publishers. Novelists, however, sign a contract to promise that their book is lying. If they also lie about their life, perhaps dropping hints of autobiography, there may well be no legal consequence at all.

Archer appeared at this year’s Jaipur literary festival too, on day three. His event drew huge crowds. He was also interviewed afterwards by news18.com. “If you look at the audience outside,” Archer said, “over 3,000 people who came to hear me speak … If you ask me: ‘Well, explain it, Jeffrey. We want to know why have they come.’ I do not understand it myself.” Professional liars are good at being forgiven. HarperCollins will publish Mallory’s second novel next January, as planned.