We don't believe in ghosts, so writing ghost literature for a modern readership presents particular challenges. How does one write for an audience that is cynical, yet still wishes to be terrified? What exactly is a ghost, anyway?

We live in an age of reason, a more secular culture than that of those great ghost writers, the Victorians; we rely on the proofs and disproofs of science, psychology and medicine, on the digital recording of much of our lives. We live in brightly illuminated rooms on streets devoid of the terror of something moving just outside the lamp light. Wraiths don't tend to show up on CCTV cameras, holograms are explicable phantoms and we all know what Freud made of ghosts.

It was only after I was approached to write a novella with a supernatural aspect that I realised all my novels are haunted: by the past, by desire or by guilt. And so it took only a small shift to see that I could take this one step further. The ghosts should not be visible – at least not in any straightforward way. Who can forget Peter Quint standing outside the window in The Turn of the Screw? He is always at one remove: behind glass, or in the distance on a tower, just as his companion Miss Jessel is glimpsed on the other side of a lake. While writing Touched, it felt important to me that unexplained presences were not the walking dead, but were just perceived as sounds, scents or misidentifications; at most, they are reflections, or reported sightings, or something captured in the split second of a film still. As Roald Dahl boldly claimed: "The best ghosts stories don't have ghosts in them." And, as Susan Hill says: "Less is always more."

The contemporary writer must trade on the power of anticipation, on the unnerving aspects of less obvious settings than candlelit wrecks in fog. I sought brightness for my unease: brilliant green grass and relentless sunshine, so the glimmer in the trees, the hint of eyes in a window, were all the more unexpected. Perfection can be eerie. The power of a ghost story lies in what is feared beneath the surface of the narrative, terrors glimpsed or imagined in the cracks, rather than what leaps out of the shadows.

Form is an issue. Novels are far more popular than short stories, but there are very few full-length ghost novels because of the difficulties of sustaining suspension of disbelief. Even in ghost writing's heyday, it was the short story – by Dickens, HP Lovecraft, Charlotte Riddell – that was the dominant form, while the longer classic of the genre, The Turn of the Screw, is only 43,000 words. Readers need to be in a state of tension for the unfathomable to prey on fearful minds, yet this can be maintained by the writer for a limited time without risking nervous exhaustion.

There is a fine balance between the psychological and the spectral. Ghost writing must involve a blurring between reality and madness or projection. So Sarah Waters's doctor in The Little Stranger slowly reveals himself to be an unreliable narrator; the protagonist of Charlotte Perkins Gilman's The Yellow Wallpaper is either insane or accurate. The theory that the Governess in The Turn of the Screw may be a neurotic fantasist began when Edmund Wilson wrote his Freudian psychopathology interpretation in 1934, though I believe that James did not intend this. The dead Rebecca of Daphne du Maurier's novel skews the narrator's mind as powerfully as if she had appeared thumping round Manderley. The modern ghost writer inherits a tradition of unreliable narrators, vastly ramped up by later psychoanalytic thinking. I found it interesting to subvert this by writing about apparent madness, in a girl who insists on dressing as a shabby Victorian, while the real chaos lies where no one is looking.

Endings can be a problem. It is paramount that narrative demands are satisfied, yet what explanation can there be? Ghost writing is in many ways the opposite of crime or detective fiction, whose worlds are more logical than real life – you find out who did it – whereas the supernatural can have no straightforward point of revelation to work towards. So there is a necessary ambivalence. I firmly believe in tying up narrative strands, so while every human story must be followed to its conclusion, the reader must be left plot-satisfied but intentionally uneasy, the paranormal at play in the margins.

If visions and voices are rationally explained, it's not a ghost story; if they're not, incredulity can set in. And again, Freud's influence can muffle the shivers: if a ghost is a mere psychological delusion, the gleam of the supernatural is dulled. Apparitions cannot be mere symbols, metaphors or projections: the characters, however warped, must experience them as hauntings, the reader on side.

The conventions of traditional ghost stories are there to play with, and, for the modern writer, there is pleasure to be had in hidden rooms, with resistant houses and barely heard sounds. Tropes can be ignored or upended, and chilling child patterings and mysterious stains are an enticing part of what Henry James called "the strange and sinister embroidered on the very type of the normal and easy".

This is an era conversant with extreme horror and increasingly successful crime genres, with console games that scatter images of blood on the screen. Yet we still seem to desire less definable hauntings in the form of the gothic, vampiric and ghostly. France leads the way, with its hit supernatural series Les Revenants, while ghost writer Marc Levy is now the most read living French writer in the world. The truth is an audience can be deeply scared by the very phenomena they don't believe in, haunted as they are by childhood reading or by that primal fear of the noises outside the cave. Or, worse – inside it.

Above all, ghost writing is about atmosphere. The mood and resonance, the sounds, scents and tense awareness that here is a place where anything could happen. Even the most sceptical can be seduced by it. What has always appealed to me is the modern gothic, the unsettling and even the unsavoury in literature. It's the glimmer of another presence that lies just outside our normal understanding that intrigues.

• Joanna Briscoe's novel Touched is published this week by Hammer/Arrow.