Extraordinary stories tumble out of Richard Stanley’s mouth – absurdist adventures from far-off lands, anecdotes involving ghosts and warlocks – all delivered with a mischievous matter-of-factness. One minute, he is talking about a short film he directed after a glow-in-the-dark Ouija board from Toys “R” Us dictated the plot to him; the next, he is remembering a friend who died after doing a ritual to placate “a seemingly fictional deity”. We are ostensibly discussing his new film, Color Out of Space, but it is understandable that he might head off on the odd extended detour; it has, after all, been 28 years since he directed a feature.

His last attempt was his 1996 adaptation of HG Wells’s The Island of Dr Moreau: a dream project for Stanley that turned into a disaster. The daughter of its star, Marlon Brando, killed herself the day before the Australian shoot, leaving the production in temporary limbo in his absence. Co-star Val Kilmer repeatedly disrespected Stanley. A hurricane almost obliterated the set. Engulfed in chaos, Stanley was fired after just three days of shooting, and replaced by John Frankenheimer. Stanley’s contribution to the film was uncredited. (The whole episode was revisited in the terrific 2014 documentary Lost Soul: The Doomed Journey of Richard Stanley’s Island of Dr Moreau.)

His career in tatters, Stanley retreated to Montségur, an isolated mountain village in southern France. He has lived there ever since, writing scripts, making obscure documentaries and working as a mountain guide, taking tourists to caves and Neolithic remains. Our Skype interview happens two weeks later than scheduled – he didn’t materialise the first time, and I never get to the bottom of his disappearance – but he has finally been pinned down in LA, where he is promoting his film. “I’m in the midst of madness here in Hollywood,” he says, surrounded by crates, in a shack in the hills. The South African-born 53-year-old smokes cigarettes throughout our conversation, his raven-coloured mane spilling out of an Australian Akubra hat. A gothic cowboy of sorts, he is rarely seen without such headgear, looking like a character from any one of his films.

If you go down to the woods today... Nicolas Cage in Color Out of Space. Photograph: Landmark Media/Alamy

Color Out of Space, adapted from horror writer HP Lovecraft’s 1927 short story, is as eccentric as its director: in rural New England, alpaca farmer Nathan (Nicolas Cage) and his family come undone after a meteorite lands in their front yard, precipitating much strange behaviour. Time begins warping, a colour never seen before by human eyes subsumes the environment, and the film hurtles headlong into nightmarish psychedelia, with Cage going full, well, Cage. It’s been a stealth word-of-mouth hit in the US, which is why Stanley is there, still promoting it – “Basically on a revenge tour,” he says, in a nod to his years in exile. The Dr Moreau situation had labelled him an unreliable director, and many comeback attempts had failed. But now, with Color Out of Space’s box-office success, “Hollywood’s been doing the come-hither thing,” he says, not entirely trusting the industry’s overtures. “I’ve been trying to separate dream from reality.”

Those borders have blurred throughout his life. His mother, Penny Miller, was an English anthropologist who moved to Africa. After he was born, she spent years travelling the continent, researching her book Myths and Legends of Southern Africa. She took her young son with her, meeting witch doctors and watching goat sacrifices. In his late teens, on the verge of being conscripted, Stanley fled to London, became a British citizen and began directing music videos. Then things took a left turn: in 1989 he moved to Afghanistan to film the Soviet-Afghan war, joined a fundamentalist guerrilla group, and got caught up in the siege of Jalalabad. Eventually he returned to London to direct his first film, killer-robot sci-fi Hardware (1990), which cost $800,000, made a few million, and instantly gave him cult kudos. He followed it with 1992’s pagan horror Dust Devil.

Disaster artist... Marlon Brando in The Island of Dr Moreau. Photograph: Photo 12/Alamy

Then Dr Moreau happened. After the calamitous shoot, Stanley fled to the rainforest, where he lived on coconuts and cannabis. Later, with Frankenheimer’s production in full flow, some loyal extras ran into Stanley and smuggled him back on to the set, where he disguised himself as one of the story’s dog-people and cavorted about in the background as Frankenheimer filmed. “I was on the island as a dog, trapped in a place which I’d designed,” he reflects. Was there a perverse poetry in that? “Yeah, there seemed to be some kind of Conradian sense of it being appropriate,” he says. “It felt like I’d somehow earned that fate.”

The past few years have been rough for him. Having been evicted from his home after a quarrel with a mining company that was “vaporising the mountain”, he moved into “a cabin on a crag” with no electricity. As a Brit in France, his future post-Brexit seems uncertain. “I’ve been facing a few existential challenges. It was feeling very dark at the point when Color Out of Space came over the horizon.”

Stanley has been a fan of Lovecraft since he was a kid: by the time he was eight, his mother had read him all of the author’s work. He wrote a script (with co-screenwriter Scarlett Amaris, as he wanted the film to have a female voice) and it was picked up by Elijah Wood’s production company SpectreVision, who had just made trippy 2018 horror Mandy, also starring Cage. The actor was keen to star in this, too; a fellow Lovecraft fan, he had almost worked with Stanley on Dust Devil, and thought he would be perfect to attempt to create the story’s alien colour on screen. And so, last year, Stanley found himself directing again.

In trying to visualise the extraterrestrial colour, Stanley experimented with hot air, water and temperature changes, “because the monster is neither a solid nor a liquid, it’s like a gas with fingers, it’s something from another dimension.” A VFX house then digitally augmented it. Meanwhile the sound design, along with Colin Stetson’s ear-shattering score, “pushes into ultrasound and infrasound – it goes beyond the human auditory level,” says Stanley. “The score really does push into forbidden frequencies in the last third of the movie.” The result is a bizarre and entertaining experience that is attracting repeat viewings in US cinemas. “People are going back to see the movie, realising they can take edibles with them.”

His aim was to convey Lovecraft’s sense of cosmic horror, which, Stanley says, is “to see the sheer insignificance of one’s place both in the universe, the infinite cosmos that surrounds us, and in deep time, the fact that we are just a split second surrounded by billions of years on either side of us, and our lives will truly be forgotten”. He laughs. It’s a funny film too, in its extremity. “Good!” he says when I tell him so. “Everything I do is pretty much a deadpan apocalyptic black comedy.” Such has been the film’s success, he is now writing a follow-up, an adaptation of Lovecraft’s The Dunwich Horror. His version will be about “backward Maga-era hillbillies who have interbred with ultra-dimensional demons from beyond space,” he says with a glint in his eye. “It’s very juicy.” Meanwhile, returning to Hollywood after two decades in the remote Pyrenees mountains, he’s “back into the mixing board of human experience,” he says. “I still feel as if I’m visiting this time period. I don’t exactly feel like I belong here. But I think I’m gonna get a decent third act.”

For now, he has to get home to his cat, which is being looked after by some witches, naturally. His current Montségur home, on a river, is in an even stranger locale than the cabin on the crag, he says. “In recent summers, there was a coven of witches using the house. Which is not an experience I’d like to repeat. They did leave the downstairs in a bit of a mess.”

Chaos, it seems, is never far away.

Color Out of Space is in cinemas from Friday 28 February