The Sandman author reveals how he lied his way into journalism and why he could write a sequel to almost everything he's published

In one of the earlier stories in Neil Gaiman's hugely popular Sandman graphic novel series, a writer is keeping the muse Calliope imprisoned – "demeaned, abused, and hurt" – to fulfil his need for ideas. She is rescued by Morpheus, Lord of Dreams, who visits a curse of "ideas in abundance" upon the writer. He ends up grovelling on the street, clawing out his stories in blood: "a man who falls in love with a paper doll … two old women taking a weasel on holiday … a rose bush, a nightingale, and a black rubber dog collar … make them stop."

It's hard not to wonder if Gaiman himself ever feels the same way. Already this year, he has published the children's book Chu's Day, about a panda with a big sneeze, and his first adult novel since 2005's Anansi Boys, The Ocean at the End of the Lane. He's also written a Doctor Who episode, and has another kids' book, Fortunately, the Milk, out this autumn, along with his much-anticipated return to Sandman. I emailed him earlier this year to ask if he'd have a chat about Sandman. "Can't happen," he replied. "Six short stories to get written today." OK, then.

Gaiman, chatty and warm despite a hectic schedule on a trip to the UK from his Minneapolis home, admits he "will die with books unwritten". "I think everything I've written with the exception of Ocean has a sequel I could start tomorrow."

Not even tomorrow: he's currently deep in his Sandman prequel, the subject of much excitement among fans because it's the first Sandman story in over 10 years. And it's "horrible", says Gaiman. "It's going really well but it's the least pleasant writing experience of my life because I feel like I'm writing with a million people looking over my shoulder, and I've never felt like that before."

He's also about to start work on a sequel to American Gods, his epic road novel about the clash in America between the old gods and the new – this time the gods of social media will play a part (Gaiman is a prolific and popular tweeter). And he's just written a "long short story" set in the world of Neverwhere, his first venture to "London Below" in 16 years.

His new novel The Ocean at the End of the Lane, meanwhile, which Gaiman calls his "weird little book" and which some admirers are calling the best thing he has ever written, was only meant to be a short story. But it took hold of him, and from a 15-page tale in which a seven-year-old boy and his father discover their lodger's body at the end of their lane, it became a novel in which the boy goes on to befriend a family of 1,000-year-old witches, and is drawn into a battle with one of the scariest monsters created for some time.

Gaiman has dedicated the book to his wife, the musician Amanda Palmer, and he says he wrote it for her while she was making an album in Australia, and "checking in once a week to make sure I was still alive".

"I wasn't thinking of it particularly in terms of publication. Every other book I've done I've said, 'Right, I'm now writing a novel'. With American Gods, I said, 'Now I'm going to do one of those books which is kind of like a brick, a big meandering road novel, it will take two years of my life to write,' and that's what I'm doing now."

Anansi Boys, about the sons of the trickster spider god Anansi, "was mainly written, I think, looking back on it, because I got so grumpy with the people who said: 'Ah, American Gods, shapeless, formless, meandering, road novel, proof that Gaiman cannot control form, although he's an interesting writer.' I read these reviews and I thought, 'No, no, it's exactly what it was meant to be!' I wanted to write a road novel, they meander, that's the whole point. You want me to write a classical novel, beginning, middle and end in that order? Right, I'll do it. And it won't just be a classical novel, it'll be a proper farce, it'll be a funny book which means all of the pieces are going to have to fit together, and I will do this. And I wrote Anansi Boys."

Ocean, though, was "like driving at night through the fog" – he knew "three or five pages ahead what would happen", but no further.

"On the one hand I love that, on the other hand it looks like it's not something I know how to do again. I don't know that I could ever write another book by saying: 'I will tell my wife, by making stuff up, kind of what it was like to be me when I was seven, from the inside of my head, not in the real world, then put it in the actual landscape that I grew up in.'"

There genuinely was a farm down his road that he was told was in the Domesday Book, and in 2003 his father told him that their lodger had killed himself in their car. "There was me as a kid going: 'Why does nothing happen in life like it does in books?' And actually there is all this stuff going on and nobody tells you," he says, gleefully, recounting stories of his bookish childhood when he'd climb drainpipes because that was what kids did in the stories he read, and felt "a failure for not having discovered something with my chemistry set".

But although the novel is the story of a child – narrated by his adult self years later – Ocean is most definitely a book for adults, "not because kids wouldn't enjoy some of the fantasy and the magic … but it's much more about helplessness and powerlessness, that point when you're screwed. Hello, you're seven, you're absolutely fucked," says Gaiman.

It's different, he says, from Coraline – his acclaimed 2002 children's novel in which a little girl finds herself in an identical house to her own, where the new version of her mother has buttons for eyes and won't let her leave.

"With Coraline it's absolutely about slaying dragons. Yes, there are monsters out there, and if you're plucky and you're smart and courageous and tricky you can deal with them," says Gaiman. "This is saying: 'Yeah, but you know what? Some shit goes wrong, you can make mistakes and bad things can happen.'"

Gaiman grew up in the Sussex countryside where Ocean is set, where his family were active in the Scientology community. Periodically, Gaiman says, "bonkers" rumours spring up online "saying I'm a secret Scientologist and I've donated $29m, and Amanda and I are both secret Scientologists and we were ordered to marry by the church", but he himself believes only "in the power of stories".

"Because I'm so good at making things up, I look at everything else people have made up and say, 'You know – people made that up.' So I believe in the redeeming power of stories, I believe that stories are incredibly important, possibly in ways we don't understand, in allowing us to make sense of our lives, in allowing us to escape our lives, in giving us empathy and in creating the world that we live in."

A scholarship boy at school, Gaiman didn't bother with university because he was so sure he was going to be a writer. "The terrible thing, and it really is terrible looking back on it, is I was sure I was brilliant," he recalls. His early writing, he says with a grin, certainly wasn't. "But I was convinced I was smart, I was convinced I had something to say, and I did the right thing accidentally."

This thing, after writing a lot and getting rejected a lot, was to become a journalist. "I remember with glorious, 22-year-old arrogance saying: 'Either I'm not any good, which I do not choose to believe, or I need to know how all that works'," he says. "I thought: 'Good, I'll specialise in publishing, I'll specialise in authors, I'm now a journalist, look at me, world.' And that really worked astoundingly well. I phoned editors, I pitched them things, they said yes, I went off and did them. And for the first couple of months when I was asked who I'd written for I just lied."

He ended up with a couple of book deals – for a 1984 biography of Duran Duran, and for 1985's Ghastly Beyond Belief, a collection of quotes from sci-fi and fantasy books and movies. "It was what I was good at; somebody would sit down with me and next thing you'd know I'd got a contract," he says. "I was charming and plausible, fortunately not dangerous."

What he really wanted to do, though, was write comics. So he harnessed his chutzpah, and, at a horror convention, asked the legendary Alan Moore what a script looked like. "He said: 'Page one, panel one, you describe everything you can see … if it's a sound effect you write: Fx: 'Argh' or Fx: 'Thwack', and he did a little script, and I went away and wrote one," says Gaiman.

The Sandman series followed, and it won him awards and critical acclaim – Norman Mailer called it a "comic strip for intellectuals". Plus millions of fans. "I had these mad grandiose schemes of making comics that were art. And I did … I genuinely have no idea how many million copies of Sandman have been sold in the last 25 years," he says, hazarding a guess at 20m.

Because he writes genre fiction – comics and fantasy – Gaiman is often given the label "cult". And although he's also described as a "ridiculously bestselling author" on the cover of Fortunately, the Milk, and has millions of followers on Twitter and writes a phenomenally popular blog, he thinks cult is still the right word.

"I don't think I'm mainstream. I think what I am is lots and lots of different cults. And when you get lots and lots of small groups who like you a lot, they add up to a big group without ever actually becoming mainstream."

Partly thanks to The Graveyard Book, his Jungle Book-inspired story of a boy who grows up in a graveyard, which became the first book ever to win both the Carnegie and the Newbery medals, he found himself "in a weird kind of position where it was suddenly safe to like me" even – shock horror – "to admit liking me from a literary perspective".

He points to a review in the Times of Ocean which said that if Gaiman were South American, the novel would be seen as magic realism rather than fantasy. "I thought that's kind of – is that nice? I don't know – but the idea is it's obviously literary fiction," he says. "I mean yes, I've won Hugos and Nebulas and Stokers and all these kind of things, but suddenly you've got two fundamentally very mainstream literary awards, and after you've won them people would actually say: 'Yes he's a good writer, he's a real writer.'"

"It's now got to the point where I'm kind of at the intersection of a lot of different things," he continues. "So I got big in comics, then I went over to books, and I got big in books, and then went off and did children's books, and that sort of happened and then Doctor Who came in and kicked the whole thing up another level, and you've now got to this sort of weird point where I never turned into Jeffrey Archer in any specific field, but essentially an awful lot of people read me and know who I am and like my work and like it a lot."

For years, says Gaiman, he would say he was "like sushi, I'm something only a few people like". And then "gradually you get to this point where an awful lot of people like sushi, and suddenly, sushi kind of works".