The false distinction between genre and literary novels is increasingly a matter of marketing rather than critical inquiry. John Banville can moonlight in crime as Benjamin Black; Jonathan Lethem uses aliens to explore humanity in Girl in Landscape, just as Nnedi Okorafor does in her SF novel Lagoon. Percival Everett has given us a multicultural western in Wounded, Lucy Ellmann has turned the medical romance inside out with Doctors and Nurses and Helen Oyeyemi consistently reinvents the ghost story.

"Horror", though, still has a vaguely disreputable tang. There have been notable successes – Colson Whitehead's Zone One, Robert Shearman's Remember Why You Fear Me – but stereotypes of the video-nasty in print still prevent it being taken or deployed seriously. And horror does indeed have issues. There's a certain laziness about gender questions that can easily slip into outright misogyny. There is the difference between being shocking and playing for shock value. Since horror deals with primal fears, it has to deal with fearful things: just as it is difficult to write a novel about pornography without being pornographic, or boredom without being boring, too often horror has confused the sublimely horrific with the simply horrible. If you're going to handle pitch, it's wise to wear sturdy gloves.

It is interesting, therefore, that much of the best contemporary horror is being written by women – Sarah Lotz, Kelly Link, Gemma Files, Sara Gran among them. Best known in the field, perhaps, is Lauren Beukes, after her previous novel, The Shining Girls, became mainstream enough to catch the attention of Richard and Judy. That novel was, I think, underestimated at first. It is about how misogyny adapts to and exploits every freedom that feminism achieves. Some reviewers were wrong-footed by a perceived lack of explanation about how the malign drifter Harper Curtis was connected to the supernatural House that allowed his time-travelling. Yet as Harper literally bleeds into the House, we are shown that hatred of women is the architecture, the framework, the structure of this horror. The House is not unheimlich or eerie. It is the default position of reality.

There are similar ideas at work in Broken Monsters. Set in Detroit, the Louvre for aficionados of ruin-porn, graffiti and hipster art, it opens with the discovery of a body. "The body. The-body-the-body-the-body, she thinks. Words lose their meaning when you repeat them. So do bodies, even in all their variations. Dead is dead. It's only the hows and the whys that vary. Tick them off: Exposure. Gunshot. Stabbing. Bludgeoning with a blunt instrument, sharp instrument, no instrument at all when bare knuckles will do. Wham, bam, thank you, ma'am. It's Murder Bingo! But even violence has its creative limits." Unfortunately, for Detective Gabi Versado, this proves not to be true. There is no limit on the creativity of violence, particularly when a shunned and lonely man has a dream lodged inside him. The body in question is a teenage African American boy, but the legs that are curled in the foetal position are those of a deer.

The narrative switches between Detective Versado, her teenage daughter – who is engaged in a dangerous game of paedophile-baiting with her best friend from school – a washed-up online journalist looking for a scoop, scavengers of Detroit's derelicts and the murderer. The last of these, horrifically, is the most sympathetic of the characters. The detective is hard-bitten; her daughter tight-lipped, unless online, and quick-fisted; the "asset reclamation and redistribution" experts smart-assed; and the journalist gimlet-eyed and cold-hearted. The murderer, and the dream that both inhabits and controls him, is pitiable, vicious, utopian, needy and deplorable. As with Harper in The Shining Girls, Beukes never indulges in the idea of the serial killer as genius. No, he's an inadequate, maladjusted and disappointed individual – human, not superhuman, let alone a Nietzschean Übermensch.

The search for the killer, the cyber-stalking-and-shaming, the hunt for copper wires and left-behind microwaves, the hunger for fame and the quest for a dreamlike entangling of reality and desire unite the stories, albeit in a way that, from page one, the reader will realise is unlikely to end wisely or well. The choreography of The Shining Girls is as evident in Broken Monsters, but Beukes has honed her prose. Sometimes sentences in her previous work were perfunctory; here she achieves a quiet precision. "'So drive, Layla. I love you.' It's a heavy load for those three words. Because what she means is I'm sorry." The plain language and the complicated emotions are balanced expertly. The technical device that impressed me most is the drifting between the killer's conscious mind, alert and in control, and his dream. The latter point raises some of the moral imperatives of this novel. Is the killer morally culpable if, in a town of repossessions, he is uniquely "possessed"? The dream requires exposure, prominence, networks and lacks flesh, language, possibility. The monster, as the book makes clear through its subtle mirroring of major and minor plots, is a manifestation of our desire to be in the world and yet free from its constraints.

The satirical shafts that punctuate the narrative act as ways of deflecting the reader's anxiety – Beukes is especially good at parodying mediocre art. Nevertheless, she genuinely conjures up horror in its purest, most sincere form. There is real grief when characters we have come to identify with are subjected to all the banality and monstrosity of evil, and a goose-flesh sensation as we realise how similar the evil's values are to our own beliefs in art, in opportunity, in wanting the world to be a better place. Never exploitative, never superficial, never uncomplicated: Beukes shows how horror can be the best way to explain our unbelievable reality. She uses the mode like the knife that opens the oyster.

• To order Broken Monsters for £10.39 with free UK p&p call Guardian book service on 0330 333 6846 or go to guardianbookshop.co.uk.