David Lynch seldom smiles in photographs. His etched Easter Island statue of a face doesn’t glower so much as brood; lips pursed and eyes hooded, he looks every inch the auteur in winter. The quiff completes the effect, its lush swirl seemingly frozen in place by alarming, Lynchian thoughts.

In his 40 years of film-making, the director has taken audiences from sunlit American idylls to surreal dimensions populated by demons, doppelgangers and psychotic killers. His are scenes you can’t forget: the whimpering, deformed baby in Eraserhead, the severed ear in Blue Velvet, the blood-spattered, skull-crushing violence of Wild At Heart, the nuclear explosion in Twin Peaks: The Return. Google “David Lynch creepy”, and you get 5.5m results.

Lynch works from a studio on a slope above one of three adjacent homes he owns in the Hollywood Hills, just a stone’s throw from Mulholland Drive. He is reclusive and seldom leaves this little realm, let alone grants audiences to journalists – but he is prepared to now, on the eve of publishing an unconventional memoir-cum-biography, Room To Dream. An assistant escorts me through the house (sleek concrete walls and surfaces, floor-to-ceiling shelves of VHS cassettes and CDs) and up through the garden to the studio. A snake slithered across the path earlier this morning, the assistant says.

Lynch sits in a corner, hunched over a lithograph. The tabletop brims with paint pots, lotions, chemicals, gel formula cement, lithographic paper, pneumatic drills, cables, wires and paintbrushes. There is a mug of coffee, a pack of cigarettes. The director wears cracked, ancient boots, ragged chinos and the remnants of a black buttoned-up shirt that looks to have been shredded by a badger.

It’s a scene that might be intimidating were it not for a great, largely untold secret: David Lynch is a cheery, congenial soul who rivals the Simpsons character Ned Flanders for howdy-doody niceness. “Hey, bud!” he greets the assistant, offering me a handshake and wide smile. He laughs, often, and yearns for peace on Earth. “I love my life and I’m a happy camper,” he says. “It would be nice if we were all able to fulfil our desires and live good, long, happy lives.”

The studio is a bunker-like structure of concrete and glass that overlooks a panorama of trees, bougainvillea and villa rooftops. You can feel the morning sun and hear birdsong. “I like it up here, the trees,” Lynch says, speaking with the halting lilt of Gordon Cole, the FBI agent he plays in Twin Peaks (only without the deafness). “It’s a feeling I get in LA, a feeling of freedom. The light, and the way the buildings are not so tall. You can do what you want.” Would he like a coffee, his assistant asks? “Yeah, I’m about ready for a hot one, thank you.” There is no toilet up here, so to save trekking down to the house the boss pees into a sink built into the wall. “See that thing with the handle? It pulls out,” Lynch explains. “You can pee right in there. Then you run the faucet.”

A large canvas standing in the middle of the studio, an unfinished work, depicts a tree with children. Closer scrutiny reveals that the boy standing at the base is holding a knife. A girl on the branch above is cowering. Another girl is dangling from a noose. Neither look like candidates for good, long, happy lives.

You can find suffering and death anywhere if you look, Lynch says. The other day, a spider’s web next to his desk snagged a bee. The bee broke free, only to get snagged again. “The spider came out, started wrapping him, and pretty soon the spider had him wrapped completely. And I think bit him, too. Then he undoes the packaging and drugs him and drags him.” Lynch smiles at the memory. “Man, that is a violent thing.” LA is glorious, he says, “but you see things.”

What Lynch sees, and then puts on screen for us to see, is one of the great enigmas of cinema, one that has launched a thousand film studies PhDs. When the director looks at a manicured lawn, his mind’s eye tunnels beneath it to hidden mystery, mysticism and depravity – visions he has turned into mind-bending television and film. It’s an oeuvre people tend to love or detest, and even devotees don’t claim to fully understand. With the exception of The Elephant Man (1980) and The Straight Story (1999), his films are oblique and non-linear. Who is the chipmunk-cheeked Lady in the Radiator who crushes sperm-like worms while singing to the father of the mutant baby in Eraserhead? How much of Mulholland Drive is real, or dreamed by Naomi Watts’ aspiring actor? And is the ending really the beginning?

He also believes in reincarnation, which keeps him Zen about growing old. 'Life is a short trip. We’ll all meet again'

Lynch is the last man to provide any answers but now, at the age of 72, he has opened up – sort of – in his 577-page doorstopper of a book. Room To Dream declares itself at the outset to be a chronicle of events, not an explanation of their meaning. Chapters alternate between Lynch’s own recollections and those of his friends, relatives and collaborators, as told to his co-author Kristine McKenna. There is his boyhood in 1950s middle America, his early painting career, his four marriages – to Peggy Lentz, to Mary Fisk, to his longtime editor and producer Mary Sweeney, to the actor Emily Stofle – as well as his relationships with other women. What emerges is not so much Lynch, but two very distinct Lynches: the single-minded and reclusive visionary who is also a savvy media player; the hermit who very consciously nurtures his brand.

He is at his most animated when discussing ideas. “They’re like fish. If you get an idea that’s thrilling to you, put your attention on it and these other fish will swim into it. It’s like a bait. They’ll hook on to it and you’ll get more ideas. And you just pull them in.”

When he puts a story on screen, he does not think in term of beats or plot points. “No, it’s a feeling, more of an intuition. It’s the idea that you’ve fallen in love with, and you try to stay true to that. You see the way that cinema can say that idea, and it’s thrilling to you.”

In his studio in the Hollywood Hills. Photograph: Dylan Coulter/The Guardian

He is not about to make a big-screen film any time soon: while he loves the superior sound and picture quality of cinema, Lynch thinks theatrical releases have become too short. “I would not make a feature film in today’s world because the kind of films I make couldn’t be on the big screen for very long.” Better, then, to make shows for television, a medium that is more comfortable with loose ends and narrative culs-de-sac, where he can tell a “continuing story”.

There is a theory that Lynch himself doesn’t always know what is going on in his stories. He shoots this down. “I need to know for myself what things mean and what’s going on. Sometimes I get ideas, and I don’t know exactly what they mean. So I think about it, and try to figure it out, so I have an answer for myself.”

Audiences, however, must do their own figuring out. “I don’t ever explain it. Because it’s not a word thing. It would reduce it, make it smaller.” These days he rarely gives interviews, not even during the hugely hyped return of Twin Peaks last year – a show that is still debated as either the best or worst TV of 2017. “When you finish anything, people want you to then talk about it. And I think it’s almost like a crime,” he explains. “A film or a painting – each thing is its own sort of language and it’s not right to try to say the same thing in words. The words are not there. The language of film, cinema, is the language it was put into, and the English language – it’s not going to translate. It’s going to lose.”

Kyle MacLachlan and Naomi Watts in Twin Peaks: The Return (2017). Photograph: Suzanne Tenner/Showtime

Lynch takes a cigarette, lights up and inspects his work table, perhaps wishing he could express himself with the gel formula cement rather than these slippery, inadequate words.

A film or TV show is like a magic act, he continues, “and magicians don’t tell how they did a thing”. He hates the idea of behind-the-scenes footage or making-of films about special effects. “People do it for sales, for money. But the film is the thing and should be protected.”

So no guidance on Twin Peaks: The Return’s finale, which fleetingly resolved some storylines only to fracture them and create new puzzles? Even diehard fans yearned for clarity.

Lynch smiles. “No way.”

He does, however, squash the theory, much loved by some Peakers, that the last two parts of the 18-hour series should be watched simultaneously on two screens, with dialogue overlapping. “Yeah, I heard that. It’s bullshit. See, it’s beautiful that someone came up with this. You could double-expose scenes in lots of films and it could conjure some fantastic thing.”

I heard plenty of other theories at Twin Peaks: The Return’s gala premiere last year, a glitzy affair in downtown Los Angeles where the guests drank cocktails and wore tuxedos and gowns. Lynch and his co-creator, Mark Frost, had such tight control over the script and the edit that the audience had no idea what to expect – not even the Showtime studio executives who had paid for the entire series. Tight smiles masked a collective thought: please God, don’t let it suck.

Lynch made a gnomic speech about trees, then the curtains parted and the first two hours rolled, reuniting Agent Cooper, the Log Lady, Laura Palmer and other characters last seen in 1991. Kyle MacLachlan’s Dale Cooper was still trapped in an extra-dimensional Black Lodge, amid an even stranger universe; a school principal was arrested on suspicion of beheading a librarian; a young couple were violently killed by a ghostly entity which sprang from a glass box. There was also a talking tree.

When it ended there was a hush, followed by thunderous applause. The most popular adjective at the after-party was “mind-blowing”. Nobody understood the plot or seemed to like the characters, but it didn’t matter: Lynch was a genius.

MacLachlan with Sherilyn Fenn in the original Twin Peaks series (1990). Photograph: ABC/Getty

I found one heretic, a Dutch journalist, who confessed he had been bored. Other guests bristled: could he not see the brilliance? The Dutchman filed a negative review but his editors ran a different one, proclaiming a television milestone. The Lynch train was rolling.

As Twin Peaks: The Return progressed, even fans acknowledged there was a kind of masochism to watching it – witness a two-and-a-half-minute scene in which a man does nothing except sweep a floor. When Cooper finally escapes the Black Lodge back into the real world, the FBI agent is reborn as a dullard insurance salesman – a character so slow and monosyllabic it is sometimes physically painful to watch his scenes opposite onscreen wife Naomi Watts.

“This withholding of pleasure is Twin Peaks’ new secret weapon, and it works,” wrote the Guardian’s Stuart Heritage. Others disagreed, the Hollywood Reporter branding the reboot a “self-indulgent, pointless, meaningless… exercise in misplaced nostalgia and auteur idol-worship”.

When I ask Lynch if this kind of thing stings, he says he tends not to read reviews. “The good ones aren’t good enough, and the bad ones will depress you.”

Nor does he see much other TV or cinema. When I ask what he likes to watch, Lynch says crime shows and car shows, but declines to elaborate. His favourite film of the past year? He goes silent and scrunches his face, thinking, thinking, thinking. The silence stretches. He puffs on the cigarette. Later I check the tape. Not a word for 46 seconds.

“Um,” he finally says. “I saw my son Austin’s movie [Gray House, a documentary] last year and I really liked it. I don’t think I’ve seen any other films.”

He must have seen some. The Shape Of Water?

“No.”

Dunkirk?

“No.”

Black Panther?

“No.”

Does he have any curiosity about them?

“Not really. I never was a movie buff. I like to make movies. I like to work. I don’t really like to go out.”

But these days you can watch them at home.

“Uh-huh.”

Trump could go down as one of the greatest presidents in history because he has disrupted the thing so much

In Room To Dream, Lynch says he used to be angry and thought this gave his work an edge. Then he took up transcendental meditation (TM), lost the anger and gained better focus. He has evangelised about TM ever since, dedicating his book to the late Indian guru Maharishi Mahesh Yogi “and the world family”. Sceptics take note: this highly caffeinated, pack-a-day smoker is possibly the least jittery person in Hollywood.

He also believes in reincarnation, which keeps him Zen about growing old and losing friends and colleagues, including Twin Peaks: The Return collaborators Harry Dean Stanton, Miguel Ferrer and Catherine Coulson (the Log Lady), all of whom have died since filming. “Life is a short trip but always continuing,” he says, tapping ash. “We’ll all meet again. In enlightenment you realise what you truly are and go into immortality. You don’t ever have to die after that.”

Politically, meanwhile, Lynch is all over the map. He voted for Bernie Sanders in the 2016 Democratic primary and thinks – he’s not sure – he voted Libertarian in the presidential election. “I am not really a political person, but I really like the freedom to do what you want to do,” says the persecuted Californian smoker.

He is undecided about Donald Trump. “He could go down as one of the greatest presidents in history because he has disrupted the thing so much. No one is able to counter this guy in an intelligent way.” While Trump may not be doing a good job himself, Lynch thinks, he is opening up a space where other outsiders might. “Our so-called leaders can’t take the country forward, can’t get anything done. Like children, they are. Trump has shown all this.”

Lynch and his grandfather, Austin, in Sandpoint, Idaho. Photograph: Courtesy of David Lynch

Room To Dream contains no Rosebud revelation, no key to unlock Lynch, though it does offer clues in his childhood. A happy boy, he grew up in a loving extended family and made friends easily. They biked, hiked, built backyard forts. His mother, an English language tutor, would not give him colouring books in case they restricted his imagination. With his father, an agriculture research scientist, he inspected diseased and decaying trees; they also hunted deer, skinning and eating them. Lynch recounts a memory of flies crawling on a porcupine they shot.

There is another striking scene from childhood. One night, Lynch writes, he encountered a beautiful naked woman walking down the street, bruised and traumatised. “It was so incredible. It seemed to me that her skin was the colour of milk, and she had a bloodied mouth.” He was too young or too transfixed to find out who she was before she vanished.

After art school, Lynch hustled for years to make Eraserhead, widely believed to be a response to the birth of his first child, Jennifer, who had club feet. Cineasts still debate what the onscreen infant was made of: skinned rabbit, lamb foetus? But when I ask Lynch he bats it away. “I don’t talk about the baby.”

Eraserhead landed Lynch a job directing The Elephant Man. His attempt to personally mould and apply John Hurt’s makeup failed, forcing him to hire a specialist and triggering an extraordinary bout of self-reproach. “I thought it would be better to kill myself because I could hardly stand to be in my body,” he writes. He recovered only to clash with Hurt’s co-star, Anthony Hopkins, who deemed the young American out of his depth and tried to get him fired. Mel Brooks, who produced the film, recalls an angry phone call from Hopkins but not an explicit demand to fire Lynch. In any case, he defended his director and shepherded The Elephant Man to eight Oscar nominations in 1980. Brooks remains a fan, telling Lynch’s co-author: “He’s all screwed up, too, of course, and he projects his own emotional and sexual turmoil into his work and assaults us with the feelings he’s being assaulted by.”

One of the great puzzles about Lynch is his relationship with women. He writes strong female leads, and has worked for years with many of the same actors (Naomi Watts, Laura Dern, Sheryl Lee) yet the frequent onscreen violence has led to accusations of misogyny. Offscreen, he can be brutal when it comes to the end of a relationship. In Room To Dream, Isabella Rossellini that Lynch laughed while shooting her rape scene with Dennis Hopper in Blue Velvet, and “I still don’t know why.” Rossellini went on to have an affair with the director, a relationship that ended his marriage to Mary Fisk. “My heart was truly broken, and I was walking around like a lost person with blood dripping from every pore,” Fisk tells McKenna. She has since forgiven him.

It was Rossellini’s turn four years later, around the time Lynch was directing her in Wild At Heart. “David has this incredible sweetness but [he] completely cut me out of his life,” she recalls. “I didn’t see it coming.” Devastated, she wondered if it was because she did not meditate, before realising he had fallen in love with his editor Mary Sweeney, who became Lynch’s third wife. The marriage lasted a matter of months.

With his daughter Lula. Photograph: Courtesy of David Lynch

Throughout the book, and in his own telling, it is clear that Lynch’s most enduring passion is work. When his current wife, Emily Stofle, an actor who appeared in Inland Empire and Twin Peaks: The Return, became pregnant, he warned her that film would still come first. Their daughter was born when Lynch was 66, and Stofle 35. “After I had Lula, he disappeared into his work, which is what he does… he works and that’s where he gets his joy,” she tells McKenna.

I ask Lynch how he manages to inspire such loyalty, despite such strict rationing of human contact – from his collaborators, friends, even his exes. He drops his cigarette on the floor and stubs it out with a boot before answering. “I like to have some people around. If I was totally alone I think I’d get funny, and not in a humorous way.”

As a father and husband he has often been absent, he concedes. “You gotta be selfish. And it’s a terrible thing. I never really wanted to get married, never really wanted to have children. One thing leads to another and there it is.”

That sounds like regret, until he elaborates. “I did what I had to do. There could have been more work done. There are always so many interruptions.”

• Room To Dream, by David Lynch and Kristine McKenna, is published by Canongate Books at £25. To order a copy for £18.99, go to guardianbookshop.com or call 0330 333 6846.

Commenting on this piece? If you would like your comment to be considered for inclusion on Weekend magazine’s letters page in print, please email weekend@theguardian.com, including your name and address (not for publication).