“One who raises demons,” the writer Shirley Jackson once observed, “must deal with them.” Whether fiction could raise demons and deal with them remained for her an open question, one that drives Ruth Franklin’s comprehensive new biography. In 1948 the 31-year-old Jackson, who had been publishing magazine fiction for a decade without attracting much notice, produced a very brief story for the New Yorker that would change everything. “The Lottery”, a simple tale of villagers fulfilling a summer ritual that ends with one of the most shocking twists in modern fiction, became an instant sensation, catapulting its author into literary celebrity.

Jackson’s lifelong interest in rituals, witchcraft, charms and hexes were, Franklin convincingly maintains, metaphors for exploring power and disempowerment. (She pretended to be a witch, and was once labelled “Virginia Werewoolf”.) The demons in Jackson’s fiction might be social, as in “The Lottery”; or they might be personal and psychological, as in The Haunting of Hill House, which writers including Stephen King and Joyce Carol Oates have hailed as one of the greatest ghost stories of the 20th century. We Have Always Lived in the Castle, Jackson’s final novel, brings social and psychological demons together in an unforgettable gothic space. “The relationship between a person’s surroundings and his or her mental state was one she understood well,” Franklin somewhat understatedly observes. In Jackson’s work the house of fiction is always a haunted one, and usually it is women who are being disturbed, in both senses of the phrase.

Franklin situates Jackson’s conflicted relationship with coercive postwar US domesticity within the context that would give rise in 1963 to Betty Friedan’s attack on “the feminine mystique”. It was a time of rigid and rigidly enforced gender roles, as many women struggled to reconcile their own ambitions with society’s expectations. A 1950s woman was, Franklin writes, “pressured by the media and the commercial culture to deny her personal and intellectual interests and subsume her identity into her husband’s”, and Jackson’s fictional exploration of this conflict, she argues, “constitutes nothing less than the secret history of American women of her era”. Without wanting to dispute the essential point, this seems a little overstated: as secrets go, the idea of the 1950s as a restrictive era for women is rather an open one. This was the argument not only of The Feminine Mystique, but of classic contemporary novels such as Sylvia Plath’s The Bell Jar, which also pointed out that the impossibility of living up to the contradictory ideals of womanhood in the repressive 1950s was driving women mad.

In Jackson’s work the house of fiction is always a haunted one, and usually it is women who are being disturbed

In The Bell Jar, Plath’s heroine observes that if it is neurotic to want two mutually exclusive things, then she will be neurotic for the rest of her life. Jackson similarly commented: “I wrote of neuroses and fear and I think all my books laid end to end would be one long documentation of anxiety.” It was an anxious age, as Franklin notes: the cold war meant that fears of apocalypse were ambient, making it an easily available metaphor in novels such as Jackson’s The Sundial; with typical humour, she set the apocalypse on 30 August because that was her deadline. “The Lottery” would go on to inspire such dark classics of social satire as The Wicker Man and The Hunger Games, but most of Jackson’s fiction belongs in the tradition of the female gothic, in which madness, repressed rage, doubles and other images of psychic conflict haunt domestic spaces.

A line can be drawn from Jane Eyre’s madwoman imprisoned in her husband’s house, through Charlotte Perkins Gilman’s “The Yellow Wallpaper” – in which being similarly imprisoned drives a woman mad – to bestselling 1950s books such as The Three Faces of Eve, in which a woman with multiple personalities is “cured” by two male psychiatrists with the authority to determine which facets of her identity are “normal”. In Life Among the Savages, her comical account of domestic travails, Jackson describes going to the hospital to give birth to her third of four children. Asked by a clerk to state her occupation, she says that she is a writer. “I’ll just put down housewife,” the clerk replies.

Facebook Twitter Pinterest Catherine Zeta-Jones, Owen Wilson, Lili Taylor and Liam Neeson in The Haunting, a 1999 adaptation of The Haunting of Hill House. Photograph: Allstar/DreamWorks

Humour can be another way of coping with disempowerment, of course, and by the end of her career Jackson was a master of black comedy. The deadpan opening of We Have Always Lived in the Castle is deservedly famous: “I dislike washing myself, and dogs, and noise,” a young woman says, introducing herself to the reader. “I like my sister Constance, and Richard Plantagenet, and Amanita phalloides, the death-cap mushroom. Everyone else in my family is dead.” Many consider it Jackson’s masterpiece. Reviewing it in Esquire, Dorothy Parker wrote: “There is still sunshine for us. The miracle is wrought by Shirley Jackson, God bless her, as ever unparalleled … [a] leader in the field of beautifully written, quiet, cumulative shudders. This novel brings back all my faith in terror and death. I can say no higher of it and her.” After it was finished, Jackson had a breakdown and suffered for months from severe agoraphobia and panic attacks.

Franklin’s ambition in this sympathetic and fair-minded biography is to elevate Jackson to the status of a major 20th-century writer, rather than the minor author of “The Lottery” and a handful of enjoyably eerie novels. Jackson was raised in an affluent middle-class household, first in San Francisco and then in upstate New York; her parents were conformist and conservative, with a tendency towards social climbing. Jackson did not share their values and would not adhere to their expectations. Geraldine Jackson, who was herself frustrated by becoming pregnant too soon with Shirley, constantly criticised her daughter’s appearance, especially her weight; Franklin does some fault-finding of her own, dismissing Geraldine’s judgments as superficial and calling her “clueless”.

In adolescence Jackson was socially awkward, as so many exceptionally bright young women can be until they settle more comfortably into their identities and find a milieu that accepts their talents. That moment seemed to come when as a student at Syracuse University she met the brilliant Stanley Edgar Hyman, two years her junior, an aspiring critic of spectacular ambition, an IQ of 180 (which put him in the genius range), and a tendency to be arrogant and controlling. Even while they were dating, he informed Jackson that his philandering was a philosophical principle, and expected her to listen with docility to the details of his flings. Docility doesn’t appear to have been in her repertoire, however: Jackson tried to suppress her rage, and nearly had a breakdown before they were married. The pattern, including Hyman’s unapologetic and all-but-constant affairs, would fatally repeat itself.

She began taking a frightening array of prescription medications, including Thorazine and Valium

They married in 1940 (she was 23, he was 21), over the objections of both sets of parents: Hyman was Jewish, and mixed marriages remained uncommon. Jackson quickly became pregnant, and would have four children over the next 11 years. Money was a serious problem for that first decade, as the couple struggled to establish literary careers, and Jackson tried to submit to her competitive, complicated relationship with a husband who emerges from Franklin’s portrait as emotionally abusive. At first things seemed promising: Hyman was hired as a staff writer at the New Yorker and began teaching at Bennington College, but he never published as much in the magazine as he hoped, and his first foray into teaching ended abruptly after he offended his colleagues. His ambitions were focused primarily on the major works of literary criticism he planned to produce, projects of such scope as to make Middlemarch’s Casaubon quail. His tomes took decades to complete, while Jackson’s magazine fiction sold quickly, if erratically.

They kept afloat by borrowing from her parents and against his salary at the New Yorker, while running a chaotic household, which exasperated Hyman. (As Franklin pointedly observes, his mania for order did not contribute to his ability to publish, but it did give him reasons to criticise Jackson for her failures of housekeeping.) Once “The Lottery” became a hit, however, Jackson began to command much higher prices, and before long was the family’s primary breadwinner. Hyman still arrogated to himself all the male prerogatives of the era, expecting her to care for the children and keep the house to his meticulous, not to say cranky, standards: he refused to enter her study because her books were not alphabetised.

Facebook Twitter Pinterest Jackson in later life. Photograph: Courtesy of Laurence Jackson Hyman

After a few brief stints in Manhattan and Westport, Connecticut, the family settled in Vermont, where Hyman eventually was rehired at Bennington, becoming in the end a popular and successful teacher. Their friends included JD Salinger, who played catch with their son on the front lawn in Westport; Bernard Malamud; and Ralph Ellison, who credited Hyman with making Invisible Man possible. The liberal Hymans did not feel at home in Westport or North Bennington, and unkind villagers who ostracise the inhabitants of chaotic homes became something of a feature in Jackson’s fiction. At the same time, she was contributing comic pieces about the travails of domestic life to Good Housekeeping and Woman’s Home Companion. Hyman insistently pressured her to write more, complaining whenever she wasn’t working and attaching a price-tag to every letter she wrote to friends or family. Once “in a fury” he “figured out that considered in terms of pure writing time my letters are worth $40 a page”. In truth, the fury was mutual, but only expressed in one direction: Jackson mostly channelled hers into her fiction.

Unsurprisingly, perhaps, Jackson was plagued by anxiety throughout her life, and during a particularly bad episode in the early 50s was prescribed Dexamyl, a mixture of amphetamine and barbiturate that was used as a combination diet-pill and antidepressant miracle cure; she was a heavy drinker as well. She began taking a frightening array of prescription medications, including Thorazine and Valium, while her weight soared to dangerous levels. Hyman’s efforts at control may also have taken the form of making him what we would now call a “feeder”, encouraging Jackson to overeat as a means of keeping her dependent on him: “he would … urge food on her. Thick cream pies … I had to watch him stuffing her like a goose,” said her agent later.

Jackson’s mother, across the country in California, was concerned at her daughter’s appearance in publicity photos, and wrote in censorious terms. Certainly she can be faulted for being critical, and comes across as an awkward fit for her talented daughter. But Franklin tends to flatly blame Geraldine for her daughter’s anxieties in ways that can be too simplistic; the book’s clear feminist sympathies do not seem to expand beyond Jackson to her mother, who was presumably a product of her own upbringing. Similarly, when Geraldine objected to the photos of her now obese daughter, Franklin doesn’t give her the benefit of the doubt, although she was expressing concern: “I know excess weight is hard on your heart and your blood pressure and I hated to see you using yourself so badly.” It’s rather unfair to suggest, as Franklin does, that an unkind letter about her daughter’s appearance was more to blame for Jackson’s breakdown and acute agoraphobia than the assortment of prescription medications she’d been taking for a decade, or the destructive jealousy corroding her marriage.

In 1965, at the age of just 48, Jackson died in her sleep of heart failure. She had been writing stories and letters about escape; Franklin suggests she was ready to leave Hyman and make a bid for freedom. In the end that final conjuring trick was beyond her, but her fiction remains to haunt us with phantoms of female power, and female defeat.

• Shirley Jackson: A Rather Haunted Life is published by Liveright. To order a copy for £25 go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.