If the name John Darnielle doesn’t ring any bells, that may be because his band, The Mountain Goats, has been one of indie rock’s best-kept secrets for at least half of its 30-year existence. Now a four-piece, their early albums were often made using only the most rudimentary methods. Their brilliant 2002 album, All Hail West Texas, was largely composed and taped at home over one solitary week with an acoustic guitar and Darnielle’s beloved Panasonic RX-FT500 boom box. The resulting songs are shrouded in tape hiss, giving the impression that Darnielle is singing about “The best ever death metal band in Denton” from inside a cardboard box with a gas leak.

Early albums such as Sweden and The Coroner’s Gambit sold in small enough numbers to keep Darnielle in his day job as a psychiatric nurse. But their necessary primitivism couldn’t obscure Darnielle’s gifts with melody and lyrics: “We were the one thing in the galaxy God didn’t have his eyes on,” he writes of a heady motorbike ride with his beloved (“Jenny”). Fans were few but devoted, creating an intense bond with the band that persists to The Mountain Goats’ more commercially successful present. “We hit this point of communion [in concerts] where we say: this may not be the fullest room in town but there is a tighter connection between the stage and audience than you get with other bands,” he says.

Darnielle’s fortunes changed on two fronts. The Mountain Goats entered a proper studio and recorded breakthrough albums Tallahassee (2002) and The Sunset Tree (2005). Diehard “Goat-heads” might have grumbled about the more polished sound, but it won Darnielle a broader fan-base, including Lou Reed and Stephen Colbert.

Then in 2014, Darnielle published the novel Wolf in White Van. “I have been writing songs with varying degrees of success, but enough success to make a go of it for 30 years,” he says. “I have been writing prose my whole life, but I never really tried to write a novel until Wolf in White Van. It is all very new terrain.”

In contrast to his sluggish climb up rock music’s greasy pole, Darnielle’s literary impact was instant. After rave reviews, Wolf in White Van was nominated for America’s prestigious National Book Award and was named one of the best debuts of the year by Britain’s The Independent. Donna Tartt is a fan, a friend and, apparently, a source of invaluable advice. “We did an event in Brooklyn and somebody asked what I was writing next. [Donna] physically reached out to make sure I wouldn’t start talking,” says Darnielle. “Guarding the subject matter is always the way. If I say I am writing about Vikings, people will ask: Is it some totally hilarious thing where Vikings play heavy metal? You don’t need that on your mind when you are making it.”

I spoke to Darnielle when he was sandwiched between his two artistic selves. His second novel, Universal Harvester, has just been published. At the end of this month, The Mountain Goats release their 16th album, Goths. Together they should cement Darnielle’s status as what Rolling Stone magazine called “rock’s greatest storyteller”.

Tartt did well to stop Darnielle from talking. By turns erudite and witty, thoughtful and loquacious, he comes across as a pleasing embodiment of his lyrics and prose. The differences between writing songs and fiction are distilled into a few pithy lines: “When you sit down to write fiction, your space is infinite. When you are writing a song, your space is vast.”

In full flow he is hard to stop, skipping from one subject to the next. One moment Darnielle is assessing the merits of Megadeth’s first three albums, the next he is unpacking the experimental nature of seminal English novelists such as Henry Fielding, Laurence Sterne and Samuel Butler. “Do you know Thomas Nashe’s Unfortunate Traveller?” he asks excitedly. “It’s the weirdest stuff. There’s a post-modern book if ever there was one, but he doesn’t know it’s weird because the rules are not really in place.”

When I ask why the new Mountain Goats album is called Goths, I am treated to a history lesson about the term’s problematic musical provenance. “Goth is one of those movements where everyone says: we are not a Goth band. I am sure if you use the word in an interview with Peter Murphy [of Bauhaus] he will assault you. I know Nick Cave hates it. Andrew Eldritch [of The Mission] hates it. Everyone hates it except for the fans who used it to identify their own people.”

The fans included an adolescent Darnielle. “I am writing about a music that was really important to me as a teenager,” he says. “If you were lucky enough to be turned onto something by somebody, it might be a life-changing moment of identification. Here’s music that speaks to my situation and to what I like to think about.”

Here, perhaps, is a clue to Goths defining theme, what Darnielle calls “the appeal of dark things”. The song Unicorn Tolerance asks what it means “when what you want is the stuff about the grave and death”. Darnielle has experienced his fair share of dark stuff, which possibly explains the appeal of Goths music and the grimmer honesty of his own writing. Memories of Darnielle’s youthful drug addiction are dredged up on We Shall All be Healed – “And the headstones climbed up the hill”, he sings on Palmcorder Yajna.

The finest example of Darnielle’s lyrical sober-sweetness is 2005’s The Sunset Tree, in which he confronted the physical abuse doled out by his stepfather. “I am going to make it through this year/if it kills me”, Darnielle declaims on the rousing This Year.

“That was an autobiographical exercise,” he says. “I was processing that weird grief you have when a person who has done you wrong but is also important to you has died.”

Darnielle is also quick to put his suffering into context. “I’ve had my struggles. But I have worked [as a psychiatric nurse] with children who would have traded their lives for my young life any day of the week. All abusive households should not be abusive. That is an obvious truth. But it is all quite obviously true to say, someone who gets beaten every day has it worse than someone who gets beaten once a month.”

The appeal of dark things – death, loss, grief, suffering – is writ large throughout Universal Harvester. “Harvesting is a great metaphor. You are going to cut down all these plants that once enjoyed life as much as a plant can. You are going to kill them all,” he says with a laugh. “But then everybody will sit around tables and eat the fruits of these plants and have communion around these good things.”

The novel’s premise draws on horror tropes contained in The Blair Witch Project or The Ring. A series of unsettling, lo-fi home movies are discovered embedded on videotapes of mainstream Hollywood films. Are the figures glimpsed in the grainy footage victims of torture? Or are they complicit in a surrealist conceptual art project? The footage fascinates everyone who sees it, prompting several characters to try to find the mysterious filmmaker.

The eventual revelation turns everyone’s preconceptions on their head, including those of the reader. What proves ultimately significant is the film not the director, the song and not the singer. “I think what the artist makes can be very special, but the artist himself isn’t. We know artists who think they are special. They are insufferable,” says Darnielle, laughing. “You want to stay as far from them as you can.”

This scepticism about the cult of the artist reminds you that The Mountain Goats was forged in the indie-rock wars of the 1990s, when musicians such as Kurt Cobain yearned for stardom only to agonise about selling out their punk-rock ideals. “I have a dream of publishing anonymously,” Darnielle tells me. “Not from any reasons of shame. Just so that the work would be forced to stand on its own two legs. This very romantic idea makes my heart swell. An artist that denies his work even though he knows it’s good. That to me would be a real artist.”

If a similar “romantic idea” informed Darnielle’s lo-fi work in the 1990s, he admits that even then he was not immune to the allure of selling records and playing concerts. “Back then I really wanted more people to hear [The Mountain Goats], although I was very obstinate about it. I went out on tour. When you go out on tour, you are saying look at me.”

There will be plenty of people looking at John Darnielle and his work in 2017, whether this is in a concert hall or bookshop. His present success means Darnielle can support his family (he has two young sons) by doing what he loves, even if it means long stretches away from home. After 30 years of patient and, frequently brilliant, striving, Darnielle sounds content with his place in the creative universe. “What I do was never going to appeal to the boss at Capitol Records. I am never going to play The O2 arena in London. So, we focus on our craft. We focus on playing for the people who are looking at us. If it goes beyond that, it’s nice. If it doesn’t, that’s fine too.’

James Kidd is a regular contributor to The Review.