Anyone who has read Shirley Jackson’s The Haunting of Hill House will find a couple of details of its 1959 reception almost too neat to be true. Jackson had been writing novels and stories for nearly two decades before embarking on her tale of Hill House, a mansion set under a hill where visitors can turn up any time they like but find it rather harder to leave. These earlier works were striking, wrote Jackson’s biographer Ruth Franklin a couple of years ago, not only because they were such accomplished contributions to the strain of American gothic that includes Nathaniel Hawthorne, Edgar Allan Poe and Henry James, but because they foregrounded women – single women desperate for the social acceptance of marriage, or married women trapped in domestic situations so stifling they were (often malevolent) characters in their own right. Jackson herself was increasingly desperate in her marriage and in the imposed role of homemaker.

The Haunting of Hill House was her first book to earn its advance, and more: Franklin notes that Jackson used the surplus to pay off her mortgage. It was optioned, and then filmed by Robert Wise, who had just finished making West Side Story and would go on to make The Sound of Music; Jackson used that money to remodel her house, buying sheets in such vivid colours that the small Vermont town in which she lived remembered them for years. The book was a finalist for a National book award along with novels by Saul Bellow, John Updike and Philip Roth (who won with Goodbye, Columbus) – writers who became household names synonymous with seriousness, while Jackson’s oeuvre was dismissed as middlebrow thrills, skilfully produced by a housewife who, unhelpfully to herself, sometimes claimed to be an amateur witch. Shortly after The Haunting of Hill House was published, Jackson became so ill with agoraphobia and colitis that she barely made it to the premiere of the film in 1963. “I have written myself into the house,” she said to a friend, and it was true in many more ways than one.

Facebook Twitter Pinterest Family affair … the Crains in The Haunting of Hill House. Photograph: Steve Dietl/Netflix

Hill House, “not sane, stood by itself against its hills, holding darkness within; it had stood so for eighty years and might stand for eighty more”. Dr Montague, a doctor of philosophy, wishes to investigate what constitutes the darkness, which has led to the house being shunned by all who live nearby. He intends to reside there for a summer, with as many people as he can find who have a sympathy for the paranormal. Two women – Eleanor, who as a child once seemed to activate a poltergeist, and Theodora, whose empathy is such that she can effectively mind-read – answer the letters he sends out. Along with a young man called Luke, who is to inherit Hill House, they form a cosy party of four (not counting a housekeeper who resembles an automaton, and her grouch of a husband, who cares for the grounds). Stephen King once called the novel “as nearly perfect a haunted-house tale as I have ever read”, but it is much more than that. (Wise’s film, called simply The Haunting, is now acknowledged as a classic; a 1999 remake, however, was roundly condemned.)

The book is thrillingly well written, full of complex, economical, vivid insight. Dr Montague’s letters are described as having a certain “ambiguous dignity”; a couple of boys Eleanor passes on her way to the house stare, “elaborately silent”; a marble cupid on a Hill House mantel “beamed fatuously” down at the two women, whose talk is punctuated by “the comfortable small movements of Luke and the doctor taking each other’s measure” in a game of chess. Even seemingly throwaway descriptions carry weight and set off echoes, and the dialogue dances with wit and subtext, making sudden shifts in tone from sunshine to the chill of cloud passing over. So the doctor, telling the tale of the house – built, its angles deliberately off-kilter, by a man called Hugh Crain, whose young wife died in an accident as she entered the grounds, and whose daughters then fell out over it – notes: “It was said that the older sister was crossed in love … although that is said of almost any lady who prefers, for whatever reason, to live alone”.

Jackson's book is thrillingly well written, full of complex, economical, vivid insight

For this, in the end, is the crux and power of the novel – the ways in which judgments and expectations dictate and trap and make vulnerable the outer and inner lives of women. Eleanor’s existence so far has been “built up devotedly around [the] small guilts and small reproaches, [the] constant weariness, and unending despair” of caring for a seriously ill mother; her trip to Hill House is the first thing she has done entirely for herself and she is giddy with freedom and the sense of new beginnings. “I am going, I am going,” she thinks, driving away, “I have finally taken a step.” Theodora is different, more worldly, charismatic, a bit spoilt. Jackson is brilliant on the shifting sands of female friendship, the psychological edifices built at first on laughter and blind trust – which then provide the weapons that can wound most deeply.

Jackson herself had a more outwardly privileged and urbane life. The daughter of well-to-do Californians, she married a literary critic called Stanley Hyman while young; they eventually settled in an academic town in Vermont where their circle included JD Salinger, Bernard Malamud and Ralph Ellison. But she too was escaping a mother she felt criticised everything about her, and her husband turned out to be flamboyantly patriarchal and controlling, retrograde even for the time. In a 2016 piece in the New York Review of Books, Joyce Carol Oates, another fan, quotes an essay, Memory and Delusion, in which Jackson writes: “I am a writer who, due to a series of innocent and ignorant faults of judgment, finds herself with a family of four children and a husband, an eighteen-room house and no help, and two Great Danes and four cats ... It’s a wonder I get even four hours’ sleep, it really is.” However, “all the time that I am making beds and doing dishes and driving to town for dancing shoes, I am telling myself stories. Stories about anything, anything at all. Just stories.”

Facebook Twitter Pinterest Cri de coeur … Shirley Jackson in 1951. Photograph: AP

Do not be fooled by the comic tone, Oates warns, this is a cri de coeur. And it comes out as such early in The Haunting of Hill House, where Eleanor, stopping at a cafe on her way, watches a little girl being persuaded to drink milk out of an ordinary cup, rather than her own cup with its motif of stars at the bottom. “Don’t do it,” she thinks, “insist on your cup of stars; once they have trapped you into being like everyone else, you will never see your cup of stars again; don’t do it; and the little girl glanced at her, and smiled ... and shook her head stubbornly at the glass. Brave girl, Eleanor thought; wise, brave girl.”

The house behaves as haunted houses will, but the tension increasingly comes from wondering whether Eleanor’s hope of independence, of chosen friendship and love, will be fulfilled. Does she have a chance at happiness, her own cup of stars? Moments of pure fear are followed by a sense of excitement, a kind of hectic community and camaraderie. “I have somehow earned this joy,” she thinks, “I have been waiting for it for so long.” But the laughter is ever more hollow; the jokes and teasing reveal their edges and begin to draw blood; Eleanor laughs along but she feels her consciousness splitting between public amiability and the darker knowledge that “that isn’t what Theodora meant at all”. “I am learning the pathways of the heart,” Eleanor thinks a bit later. There is nothing straightforward or even necessarily hopeful about where this education might lead.

These eddies and shifts are far harder to pull off on screen than shrieks and groans and things that go bump in the night. The creators of Netflix’s 10-part retelling of the novel have decided to go back in time and tell the story of what it was like for the original Crain family to live in what they are calling “the most haunted house in America”. They have decided, also, to expand the family, including Theodora and Eleanor as two of five siblings, with the narrative flipping back and forth between the children and their modern-day grown selves. Insofar as the psychic ramifications of family are at the heart of the book’s considerable power, this makes some sense. Franklin’s biography recounts Jackson’s claim that at one point in writing the novel she went to her desk in her sleep and wrote across a piece of paper “DEAD DEAD”. But when Franklin investigated, what she actually wrote was “FAMILY FAMILY”. It’s clear from the initial episode that the programme makers intend to plumb all the ways in which Jackson suggests a haunting can be just as much about trauma, about memory and the unquiet unconscious as about the paranormal.

• The Haunting of Hill House is on Netflix.