One of the recurrent responses to the flurry of #MeToo allegations, in Hollywood and beyond, was for those accused to cast doubt on the credibility of the women involved, either by implying that they were seeking publicity, or that they were too unstable to be taken seriously.

Nowhere was this more apparent than in the case of Rose McGowan, a woman whose sometimes erratic behaviour, candid discussion of her troubled childhood and history of emotional fragility was pounced on by lawyers bent on undermining her version of events.

This is nothing new; for centuries the testimony of women has been held up to scrutiny and frequently dismissed on the grounds that our biology makes us prone to neurosis, hysteria, irrational subjectivity, and that our judgment can’t be trusted. It’s also a favourite cliche of fiction and drama: the heroine who is repeatedly told by men that she is imagining things, until she starts to question her own sanity. McGowan has repeatedly used the word “gaslighting” of her treatment by men in the industry, a term taken from the 1944 film Gaslight, in which a husband attempts to convince his wife she is going mad in order to cover up his own criminal activity.

I found myself troubled by this insistence on the psychological vulnerability of women as entertainment

It’s curious, then, that in our more enlightened times, when women are no longer routinely incarcerated as hysterics, that we should remain so obsessed with the idea of the female narrator who can’t be relied upon to know her own mind, or even what she saw from the window of her train or apartment. The obvious example is Paula Hawkins’s multimillion-selling The Girl on the Train, in which the narrator’s judgment was impaired by her drink problem. There’s SJ Watson’s bestseller Before I Go to Sleep, which also became a blockbuster film and features a female narrator convinced that something sinister is going on in her marriage, but who struggles to prove it because she suffers from memory loss.

More recently, AJ Finn’s The Woman in the Window offers a retelling of Rear Window, with a female narrator behind the telephoto lens. Like James Stewart’s character Jeff in the original, Finn’s heroine Anna thinks she witnesses a murder in a neighbouring apartment. But where Jeff is mocked by his detective friend for having an overactive imagination, Anna is told outright that her agoraphobia and diet of psychiatric meds, alcohol and classic suspense films mean she can’t possibly have seen what she claims, to the point where she no longer trusts herself. The same trope is particularly potent in ghost/horror stories; it’s there in Rosemary’s Baby, Shirley Jackson’s The Haunting of Hill House and Jennifer Kent’s acclaimed film The Babadook.

When I started writing my novel While You Sleep, inspired by these last three, I found myself troubled by this insistence on the psychological vulnerability of women as entertainment. As Finn’s reframing of Hitchcock shows, the tone of a story shifts significantly when the unreliable observer is female. It’s not that a man will never be disbelieved, or have his judgment impaired by drink, drugs or mental health problems, but that he is not made to feel vulnerable in the same way. He does not have to contend with that long history of being infantilised by male authorities, told he is imagining things, or neurotic, or mad, or having his testimony disregarded even when he fears for his life. None of the bestsellers mentioned above would have the same atmosphere of claustrophobic menace if the main characters were male.

But I worried that writing a woman who is off-kilter for various reasons – isolation, distress, hormonal changes – might inadvertently reinforce a stereotype of neurotic women, slaves to their biology. It was an anxiety shared by Lisa Appignanesi in her book Mad, Bad and Sad, a history of women and the psychiatric profession. “Whatever my own wish to separate biology altogether from destiny,” she writes, “my exploration did make me think again that certain events in a woman’s life, whether childbirth or menopause, could well in some cases bring with them a susceptibility to disorder.” My own experience of severe postnatal depression had taught me how frightening it can be when the mind plays tricks, and I wanted to create a character who would acknowledge this specifically female experience while being self-aware enough to recognise how she appears to others.

Gaslighting is alive and well, as so much of the #MeToo reportage has shown. Women live in fear of male violence, and of being disbelieved if they report it, so it’s hardly surprising that some of the most successful fiction of recent years plays on that. But acknowledging that we – and our fictional heroines – can be flawed, and at the same time truthful, might be a way of challenging the dominant idea that women can’t be reliable narrators of our own experience – a story that has served real predators for far too long.

• While You Sleep by Stephanie Merritt is published on 8 March