For several months, when I was 10 or 11, I avoided windows at night, because I didn't want to see a hideous child vampire staring back at me. I had read Stephen King's 1975 'Salem's Lot, and it had really screwed with my tiny mind. (There's another horrible moment to do with reflections in this new novel.) At that age I was reading all the Stephen King I could get my hands on, a fact that might have worried my parents more had I not also been reading anything else I could get my hands on too. In those days, King's reputation was that of a pulp master, with the emphasis firmly on "pulp". Good for cheap thrills, but not literarily improving, except in the way that completely indiscriminate and insatiable reading might eventually turn out to be.

Two things have happened since then: first, in the wake of nonsupernatural and so more culturally acceptable novels such as the claustrophobic masterpiece Misery, King has grudgingly been admitted by lit-crit folk into the ranks of "actually good writers" as opposed to mere megaselling dimestore artists. It happened to Elmore Leonard, and it looks as though it is also currently happening, with justice, to Lee Child. (It will never happen to Dan Brown.)

Second, an entirely new literary genre, or at least trick of demographic marketing, has been invented: that of "young adult" fiction. In my day the choice was between children's books (Willard Price) and adult books (Stephen King). Now The Hunger Games is the tweenie juggernaut on which everyone wants to hop. I went to a few Australian literary festivals earlier this year, and every second writer I met confessed that they were working on a "dystopian Young Adult" novel. I don't know whether the idea is that adolescents are too stupid to understand grown-up dystopian novels such as Nineteen Eighty-Four, or simply that they won't be interested in expending their precious media time on a book unless there is a teen romance at the centre of things. There is something terribly condescending in this industrial discovery or invention of the "young adult", but you can't argue with the sales figures. Presumably teenagers loading up on this stuff are not reading as many books by old hands such as King.

One doubts a writer of King's enormous success is too worried about this, but there are some ironic swipes at YA fiction in his latest book. At one point a character thinks: "It was sort of like being in one of those love-and-horror supernatural novels, the kind Mrs Robinson in the school library sniffily called 'tweenager porn'." There are also allusions to Twilight, Game of Thrones and the Bruce Willis movie The Sixth Sense – all of which is strangely reassuring, since in most genre fiction the characters have never read or seen any genre fiction. This is a shame because it might help them deal with the vampires or werewolves or zombies they have to contend with.

Instead, the anxiety of influence here works inwardly. This novel is a sequel to one of King's most famous works, The Shining. In the concluding author's note, King admits that sequels are almost never as good as the originals, and "nothing can live up to the memory of a good scare". The Shining, of course, was also made into a film by Stanley Kubrick, with the celebrity of which King wrestles a bit grumpily: it's a movie "which many seem to remember – for reasons I have never quite understood – as one of the scariest films they have seen". The novel, not the film, he insists, is "the true history of the Torrance family".

After an expertly compressed "previously on The Shining" prologue, we discover in Doctor Sleep that Dan Torrance, the little boy in The Shining who sees dead people, is now all grown up. Unfortunately he has become a boozehound and drifter who gets into bar fights and steals from women he sleeps with. The moral torpor is leavened with some amusingly disgusting hangover descriptions and plangent existential comedy – as drunk Dan settles down for a night sleeping rough, he reflects: "He supposed he wasn't leading what straight America would call an exemplary life."

Dan eventually decides to stick around after getting off the bus in a small town in New Hampshire, and starts going to AA meetings and working as a night porter in the local hospice. It's a building with self-conscious horror-movie character: "There was a turret at the top of the mansion on the left side, but none on the right, giving the place a queerly unbalanced look that Dan sort of liked. It was as if the big old girl were saying Yeah, part of me fell off. What the fuck. Someday it'll happen to you."

Dan's supernatural talents, long suppressed by the drink, now enable him to help people die peacefully, which leads to him being nicknamed Doctor Sleep. (The hospice cat singles out those on the brink, curling up with them on their beds.) Meanwhile, Dan also starts to receive telepathic messages from a young girl, whose psychic abilities are much stronger than his – in the lingo, her "shining" is far brighter. When the girl, Abra – whose gifts include Carrie-style telekinesis – reaches the age of 13, it is discovered that she is on the tasting menu of a group of very bad people.

The bad people are the True Knot, a roving gang who live in motorhomes and torture child psychics to death so they can eat their shining. (They call it "steam"; it's their only nourishment.) Introducing them, King casually flicks a technical switch, addressing the reader for the first time:

"How many times have you found yourself behind a lumbering RV, eating exhaust and waiting impatiently for your chance to pass? … You might have seen the True's rolling motorhomes parked in that lot … You hardly see them, right? Why would you? They're just the RV People, elderly retirees and a few younger compatriots living their rootless lives on the turnpikes and blue highways."

Those "RV people" are actually evil supernatural beings, "empty devils" who are very long-lived if not actually immortal: "Once they rode camels in the desert; once they drove caravans across eastern Europe. They eat screams and drink pain." (It is always satisfying when a newly minted myth can be shown to retrospectively explain earlier superstitions.) Dan and Abra at one point explicitly discuss the generic taxonomy of these enemies: they are a bit like vampires, but not really. Such namechecking of generic reference points helps to add the dimension of popular art to King's brilliantly simulated normality.

Another globally bestselling writer who is good at simulated normality is Haruki Murakami, upon whom one is tempted to speculate that King was an early influence. Both writers are able to describe, at leisure and in loving and apparently gratuitous detail, scenes of perfectly humdrum quotidian action (someone blowing leaves off a lawn or cooking rice). The longer this goes on, the more it generates its own kind of weird suspense, because you know this is a Stephen King or Haruki Murakami novel, and something very weird or horrible must happen soon. But it is only because this patient groundwork has been laid so slowly that the eventual irruption of something abnormal is so powerful.

Like Murakami, too, King finds a concretely physical, matter-of-fact way of describing eldritch events, which hypnotises the reader into accepting them quickly. At one point the leader of the True Knot, a beautiful Irishwoman named Rose the Hat, is able to suck Abra psychically into her own mind. This is described, from Abra's point of view, thus: "The world beyond her window began to turn, as if it were mounted on a gigantic disc." Another woman who is captured by the gang feels like this: "Andi tried to turn, but hands seized her head. The terrible thing about them was that they were inside." When someone else dies, the simplicity – even near-banality – of King's simile is arrestingly vivid: "The life ran out of her face like water from a basin with a hole in it."

Even when things are going completely nuts – as in a surreal bad-clown dream-scene of Abra and Rose fighting in Abra's head, which turns out to be a room stuffed with file drawers, while Abra is mounted on a horse and wielding a lance – King keeps the action logical, comprehensible and swift. He is even able to build dramatic tension in a scene in which people in a car are fighting telepathically for control of a gun. This is just the kind of thing that can happen in his world.

Such moments are certainly eerie, but is the book scary? Well, either I am now less easily scared than my child self – who was reduced to gibbers by the apparition of a balloon in a library in one of King's weirdest and most frightening novels, It – or Doctor Sleep does not actually deliver the "good scare" of yore. There is nothing to match one of the images from The Shining that Dan here helpfully remembers: "Those hedge animals. The ones that sometimes moved around when you weren't looking." Another problem is that while Rose the Hat is a decent villain, the rest of her gang seem more like comically bumbling rednecks.

What the novel lacks in brute fright, though, it makes up for with more subtle pleasures. The scenes where Dan accompanies elderly hospice residents in their final moments are tonally very well judged: here King finds a mode of the supernatural that has a melancholic beauty while avoiding spiritualist blather. (These moments are punctuated by quotations from TS Eliot. Ezra Pound and WH Auden, as well as lyrics from the Village People, crop up elsewhere, making an unshowy point about literary inclusiveness.) And there are some wonderful secondary characters, including Abra's great-grandmother Concetta, an Italian-born poet, and the twinkly old dude Billy, who runs the local tourist railway.

This is also (or perhaps even firstly) a novel about alcoholic excess, and King's tenderly sympathetic but no-bullshit approach to the subject is in a way more authentically disturbing than any pseudo-vampire. "Your mind was a blackboard," Dan remembers at one point. "Booze was the eraser." King refers to the earlier self who wrote The Shining as a "well-meaning alcoholic", just like Dan's father in that novel. Doctor Sleep traces a fuller arc in which Dan gets hooked on drink, hits rock bottom, and drags himself up again. The characters in the novel exhibit gruff scepticism about some parts of AA lore, but the book is prefaced by two epigraphs from The Big Book of Alcoholics Anonymous, followed by an "old AA saying" on the next page: "FEAR stands for Fuck Everything and Run."

It's possible to interpret the True Knot themselves as allegorical alcoholics, mirroring the novel's merely human drinkers: after all, the villains too are substance-dependent drifters, wrecking children's lives. But while the surface story of supernatural derring-do is never less than a superbly well-engineered ride, full of satisfying twists and switchbacks, the novel's deepest shiverings depend on no made-up devils. At one point in his early desperation, Dan reflects on how other people's well-meaning advice to "Give it time" is misplaced: "Time changed. That was something only drunks and junkies understood. When you couldn't sleep, when you were afraid to look around because of what you might see, time elongated and grew sharp teeth." Time with jaws: now that really is a scary monster.