Stephen King has written a lot of books – at 56 novels, he's closing in on Agatha Christie – some of which have been great, some of which less so. Still, he says, when people say, "Steve, your books are uneven", he's confident "there's good stuff in all of 'em". Now and then, a story lingers in his mind long after it's published. When fans ask what happened to Charlie McGee in Firestarter, for example, King isn't interested. But when they ask what happened to Danny Torrance, the boy from The Shining, he always found himself wondering. Specifically: what the story would have looked like if Danny's father – mad "white-knuckle alcoholic" Jack Torrance – had "found AA. And I thought, well, let's find out."

At 65, King is a big, shaggy presence, towering despite his slightly stooped shoulders and with an air of affable amusement at the vastness of his success and all that comes with it. We are at a house in Maine that his assistant, opening the door, drily refers to as "spare"; it's one of several King properties in the area, on a lake, and designated vaguely as a summer house. It is at the end of a long, deserted road, surrounded by woodland and in a GPS dead zone; this, after a week of rereading King novels, is unrelaxing. Rather than spend the night in a remote B&B near his house, as his publisher suggests, I stay in Portland, 100 miles away, in a hotel where there are lights and cars going by, and people to hear you scream. "Really?" King says when I mention my unease, and grins. "Good."

Doctor Sleep, his 56th novel, revisits Danny in adulthood, when he has become an alcoholic drifter haunted by the memory of his raging father. The Shining had such resonance – in part because of Kubrick's film, which King disliked – that one returns to the characters with a sense of deep familiarity. In the sequel, Wendy, the mother, is dead from lung cancer and Danny is alone, working in a hospice in a small town, where his paranormal talents help people towards a peaceful death. When Abra, a telepathic child, pushes into his consciousness asking for help, Danny gets sucked back into the terrain of his childhood, battling a bunch of centuries-old serial killers disguised as RV-driving pensioners (it is sometimes easy to overlook how slyly funny King is) who literally feed off the pain of others. "When the disaster was big enough," King writes, "agony and violent death had an enriching quality." They get a big kick out of 9/11.

It is scary, of course: a woman with a tusk instead of teeth pops up periodically to hang at second-floor window level and startle the bejesus out of you. And without labouring the point, it has good allegorical bones: the sick buzz one gets from consuming the grisliest news stories. It also captures the reality of a recovering alcoholic, a state with which King is intimately familiar. "The hungover eye," he writes, "had a weird ability to find the ugliest things in any given landscape." Danny turns his life around and starts going to AA meetings, where, King writes, he discovers that memories are the "real ghosts". It is a book as extravagantly inventive as any in King's pantheon, and a careful study of self-haunting: "You take yourself with you, wherever you go."

King has been sober for decades, ever since his family staged an intervention in the late 1980s. If he hesitated to write in this much depth about AA, it was only because he wanted to get it right. "The only thing is to write the truth. To write what you know about any particular situation. And I never say to anybody, 'This is all from my experience in AA,' because you don't say that." It was King's 36-year-old son, Owen, who, after reading the first draft of Doctor Sleep, told him there was something missing. "He said that the scene he remembered best from The Shining was the one where Jack Torrance and his friend are out drunk one night and they hit a bicycle and think they've killed a kid. And they say, 'That's the end; we're not going to drink any more.' And Owen said, 'There's no scene that's comparable to that in Doctor Sleep. You ought to see Dan at his worst.' And, as usual, Owen was right."

The scene King put in, would, in subsequent drafts, go on to drive the whole story: Danny waking up next to a one-night stand, stealing her money and leaving her infant son wandering about with a full nappy, reaching for drugs on the coffee table. "And I think every alcoholic has a story comparable to that. Something where you actually hit rock bottom."

In his case?

"I don't have anything as dramatic. Of course, in a novel, you're looking for something that's really harsh. Harshly lit. For me, when I look back, the thing that I remember is being at one of my son's Little League games with a can of beer in a paper bag, and the coach coming over to me and saying, 'If that's an alcoholic beverage, you're going to have to leave.' That was where I said to myself, 'That's something I'll never be able to tell anybody else. I'll keep that one to myself.' I drew on that memory."

In Doctor Sleep, Danny fights his past with a more profound sense of terror than anything the woman with the tusk can bring on. The tentacle reach of history has always interested King – "What's inside your head grows. And you don't have any sense of proportion until you see how other people react to it" – as has the futility of trying to escape it. "Take Dan Torrance, who is the child of an alcoholic, child of a dysfunctional family, abusive father, and he says, as people do, 'I'm never going to be like my father; I'm never going to be like my mother.' And then you grow up and find yourself with a beer in one hand and a cigarette in the other, and maybe you're walking the kiddies around. And I wanted to see what would happen with that."

For a while, King would write sober during the day and edit what he had set down, while drinking, at night. "As time went on, I started to fumble a lot of the balls. I had a busy public life and a lot of those things got a bit ragged by the end." Did he, like Danny, go to bars and get in fights?

"No. I didn't go out and drink in bars, because they were full of assholes like me."

Stephen King at his home in Maine, US. Photograph: Steve Schofield for the Guardian

Stephen King has been at the top of his game for a very long time, but his quirks still give him a quiver of outsidership. Before he became fashionable, he was deeply unfashionable – a nerdy guy writing in a nerdy genre, married with three kids when everyone else in his generation was raging through the 1960s. For a while, he and his wife, Tabitha, lived in a trailer in Herman, Maine (as King once put it, "If not the asshole of the universe, then at least within farting distance of it"). In literary circles, it is a more outlandish background than the most lurid of King's horror stories: Tabitha worked in a Dunkin' Donuts and King supplemented his high school teaching income at a laundry and a filling station. He felt under such tremendous pressure during those years, he says, that it was as if "battery cables were hooked up to your head. Like your brain was a battery."

He was a good teacher – the kids enjoyed his classes – but he felt trapped in the wrong life. "I would teach, and I would come home tired, like I'd been on stage. And then I had to correct papers – more of the same. And there was very little time left for my own work. I can remember thinking, 'Two or three more years of this and I won't be able to write at all.' Because they wanted to give me the debating club, and the play, and stuff like that. There was no discussion of me quitting. We would have had nothing to live on. We were barely making ends meet, living in crappy apartments."

His wife encouraged him to keep at it, and in those early days, King says, he was highly motivated by "this gush of image and story and words. It was like somebody yelled, 'Fire!' in a crowded theatre and everybody's trying to crowd through the door at the same time – that was ideas and work." By their mid-20s, they had two children and were very stretched.

Most of his friends weren't even married. Why have kids so young?

He bursts out laughing. "Because they came! Naomi was about nine months old when we got married. Tabby was 21. And then it seemed like a great idea to give Naomi a brother or sister. I can remember being home one day and there was a knock at the door – a guy selling something – and he said, 'Hon, is your mommy home?' And Tab said, 'I'm the mommy.' We had two kids by then."

Every scrap of free time had to be put to good use. During one particularly frenzied period, King bashed out The Running Man in a week. A week! "February vacation week. I was white hot, I was burning. That was quite a week, because Tabby was trying to get back and forth to Dunkin' Donuts and I had the kids. I wrote when they napped or I would stick them in front of the TV. Joe was in a playpen. It seemed like it snowed the whole week, and I wrote the book. Couldn't sell it."

In King's 1986 novel It, the character Stuttering Bill stands in for the author as a highly successful horror writer, who corrects journalists when they ask where he gets his ideas from. The better question, Bill says, is why do they come in that particular form? Why horror? King has always recoiled from glib readings of a childhood rift in his psyche: a father who left when King was an infant, never to return. But it was through his father that he discovered writing: a book he found in the attic, which his father left behind. It was a collection of short stories by HP Lovecraft called The Lurker In The Shadows and had a demon on the cover. King read it as a boy and something pinged in his brain.

Everything changed with Carrie, the story of a telekinetic teenager and her sublime rage at her fundamentalist mother and bullying schoolmates. It was picked up in 1973 by Doubleday, for an advance of $2,500. That was enough for the Kings to buy a new car. A year later, when the paperback rights went out for auction, King expected to make something in the region of $60,000, half of which would go to his publisher. Since $30,000 was more than he earned in a year as a teacher, he planned to take a sabbatical and write two more books. "But the advance turned out to be $500,000."

King notes with some amusement that he has been around so long that kids who read and loved him in the 1970s now run publishing houses and newspapers; he is revered, these days, as a grand old man of American letters. The experience of reading King young – "Under the covers with a flashlight at summer camp," as he puts it – doesn't leave one, and although he says, casually, that "it's pretty easy to scare a 14-year-old", the pleasures of his books endure. There is a lightfootedness to King's prose, a quickness of thought and expression that over the course of decades has somehow always seemed modern. It's partly a function of speed: his books err on the long side – a casualty of quick turnaround – but the action rattles along at a pace that is mimicked in the snap and verve of his language. In Dr Sleep, the serial killers appear with "faces like old apples and the moon shone right through". Danny, in the hospice, observes of a dead patient, "inside was all the clockless silence of death". It's what King does best, matching dim fears with indelible images, and it is recognised these days as a rare talent.

For a long time this wasn't the case. Condescension and a certain amount of hostility used to mark the critical reaction, and because of that, perhaps, he is belligerently against what he sees as rarefied writing types. Stuttering Bill Denbrough in It storms out of a college writing class when the instructor sneers at his horror story. "Why does a story have to be socio-anything?" Bill asks. "Politics… culture… history… aren't those natural ingredients in any story, if it's told well? I mean… can't you guys just let a story be a story?"

The biggest beef King has with mainstream literary culture is one of productivity. He was recently asked by the New York Times to review Donna Tartt's new novel, The Goldfinch. "And Donna Tartt is an amazingly good writer. She's dense, she's allusive. She's a gorgeous storyteller. But three books in 30 years? That makes me want to go to that person and grab her by the shoulders and look into her face and say, 'Do you realise how little time you have in the scheme of things?' "

It is 11 years since Tartt's last book, and King says, "I looked at it and thought, 'God help you, Donna, this better be interesting.' " And was it? He smiles. "It's very good." When people ask why he is so prolific, he smiles and tells them: "I'll stop soon enough."

Sissy Spacek in the movie version of Carrie, Kathy Bates in Misery, Tim Curry in It and Danny Lloyd in The Shining. Photograph: Everett Collection/Rex

Almost all his books have been turned into movies, the bulk of which have been successful, although King doesn't bite his tongue when something isn't to his liking. He enjoyed Brian De Palma's 1976 adaptation of Carrie, starring Sissy Spacek. But he "hated" what Stanley Kubrick did to The Shining in 1980: the film turned his novel into "a domestic tragedy with only vaguely supernatural overtones", he said at the time. He also thought Jack Nicholson hammed it up appallingly, and Shelley Duvall as Wendy was "insulting to women. She's basically a scream machine."

It has been frustrating, he says, when he has tried to bust out of his genre and been largely dismissed or misunderstood – primarily with his novel Needful Things, a satire of Reagan-era materialism that baffled the critics. "They read it and said, 'This is just peculiar.' " He has a lot of sympathy for JK Rowling, who was spectacularly mauled for her first non-Harry Potter novel, The Casual Vacancy, which King is reading at the moment.

"Man, this book is like… Do you remember Tom Sharpe? It's a bit like that. And it's a bit like Who's Afraid Of Virginia Woolf? It's fucking nasty. And I love it. The centre of the book is a dinner party from hell and you say to yourself, 'These little people in the town of Pagford are a microcosm not just of British society, but western society as a whole, of a certain class.' The fact that she set it around this little election that nobody cares about in a shit little town is fabulous. She's a wonderful storyteller and the writing is better than in any of the Harry Potter books, because it's sharper."

King is not so successful as to be above the fray, and he is sceptical about some of his more direct rivals in the mega-selling horror and fantasy categories. Contrary to popular opinion, he says, this is not a golden age of horror. What about the Twilight franchise? "I agree with Abra's teacher friend [in Doctor Sleep] who calls Twilight and books like it tweenager porn. They're really not about vampires and werewolves. They're about how the love of a girl can turn a bad boy good."

Sweet Valley High with teeth?

"Yeah. Pretend I said that."

Does he read them out of professional interest?

"I read Twilight and didn't feel any urge to go on with her. I read The Hunger Games and didn't feel an urge to go on. It's not unlike The Running Man, which is about a game where people are actually killed and people are watching: a satire on reality TV. I read Fifty Shades Of Grey and felt no urge to go on. They call it mommy porn, but it's not really mommy porn. It is highly charged, sexually driven fiction for women who are, say, between 18 and 25. But a golden age of horror? I wouldn't say it is. I can't think of any books right now that would be comparable to The Exorcist."

When his children were little, King didn't stop them from reading his own books, or watching the film versions. (In fact, when his oldest son Joe was 12, he overheard him explaining to a friend that his seven-year-old brother Owen was allowed to read and watch these things. "Joe said very seriously, 'You have to understand, my father writes scary stories. Owen has lived with horror his whole life.' " King laughs uproariously.)

In 1982, during an air traffic control strike, King was making a film called Creepshow and, unable to fly, drove 600 miles every week back and forth to Pittsburgh from Maine. To pass the time, he asked his daughter Naomi to record audio books for him of titles that didn't exist in that format - Wilbur Smith novels, for example; he would pay her $5 per cassette. "She was all over that like white on rice. She did it and then Joe got on board, and finally Owen, too."

They are a close family. Both boys are now writers and Naomi, who also writes, is a Unitarian minister. They have come a long way since the bad old days when Tabitha threatened to leave him if he didn't stop boozing and taking cocaine. Owen, his youngest, was 10 then and Naomi 17. It is better to be frank about these things, King believes, since people always find out about them anyway. But memories of the intervention are still painful.

"There's a thing in AA, something they read in a lot of meetings, The Promises. Most of those promises have come true in my life: we'll come to know a new freedom and new happiness, that's true. But it also says in there: we will not regret the past nor wish to shut the door on it. And I have no wish to shut the door on the past. I have been pretty upfront about my past. But do I regret? I do. I do. I regret the necessity."

Still, he feels pretty lucky, not least after a near-death experience in 1999. King was walking down a road near his house when he was hit by a truck and thrown 14ft in the air. There were no white lights, but it did get him thinking seriously about death. "Our body knows things, and our brain knows things that don't have anything to do with conscious thought. And I think that it's possible, when you die, that there is a final exit programme that goes into effect. And that's what people are seeing when they see their relatives or a white light or whatever it is. In that sense, there may really be a heaven if you believe there's a heaven, and a hell if you believe there is one. But there's some kind of transitional moment. That idea that your whole life flashes before your eyes." He smiles. "Of course, they say about co-dependents – people who grow up around alcoholics – that somebody else's life does."

It is this moment of transition that Doctor Sleep deals with and the idea, like so many of King's, came from an incidental story in a newspaper. This one was about "a cat in a hospice that knows when people are going to die. He would go into that patient's room and curl up next to them. And I thought, that's a good advertisement for death, for the emissary of death. I thought, 'I can make Dan the human equivalent of that cat, and call him Doctor Sleep.' There was the book."

When King wakes in the night, he is not preoccupied with thoughts of death. He worries about his grandchildren, or turns over new ideas. His writing habits have changed over the years. "As you get older, you lose some of the velocity off your fast ball. Then you resort more to craft: to the curve, to the slider, to the change-up. To things other than that raw force."

He is as successful as ever, with a hit TV show, Under The Dome, about a town cut off from the world by the sudden descent of a large, overturned fish bowl, a clever twist on the locked room scenario. He has also just had the earliest peckings of a new book idea. It came from a news story that was big in the US last year, about a Brooklyn woman who drove the wrong way down a motorway with a car full of children, killing them all. It wasn't the horror of the incident that interested King, but the unanswered questions. "I have stories that ask to be written. And the thing that interested me about that crash on I-95 was that her husband swears up and down that she wasn't a big drinker, and that she wasn't drunk when she left with them. But there was a bottle of vodka in the car. She was high as a kite. So I'm saying to myself, there's a real mystery here." A pause. "It's the kind of mystery that only fiction can unravel." And so it begins.

• Doctor Sleep is published by Hodder & Stoughton at £19.99. To order a copy for £15.99, including free mainland UK p&p, go to theguardian.com/bookshop or call 0330 333 6846.

• This article was amended on 23 September 2013 to correct the amount of money King paid his daughter to record audio books for him.