One wintry evening in London, Josh Tillman, the American singer-songwriter better known as Father John Misty, is sitting on a hotel balcony and talking about Leaving LA, the intensely personal 13-minute-long song he performed on BBC 6 Music a few hours earlier. To be precise, he is talking about the ninth verse, which he wrote after trying to identify the source of his tragicomic worldview. It’s a true story, or as true as any childhood memory can be.

He remembers being six, growing up in a repressive, evangelical family in Rockville, Maryland. He remembers choking on a piece of watermelon candy in the department store JC Penney. He remembers his emotionally distant mother holding him tight, for once, and screaming for help. And he remembers Fleetwood Mac’s Little Lies playing over the store’s PA.

“This is where it all comes from,” he says, 30 years later. “Wanting love, getting it in this traumatic moment and feeling like, ‘If this song is playing at this moment, then life is a joke.’” He frowns and blows smoke into the damp night air. “My wanting love from her affects everything about me: the stupid way that I ramble, the stupid five-dollar words that I use, the stupid humour. All of it. The main criticism people have of me is, ‘Why are you trying so hard? This guy’s exhausting.’ Yeah, I am! That’s my way of trying to get love.” He snorts a laugh. We have been talking for just seven minutes. “This is a hell of a way to start this interview.”

This combination of candour and self-mockery is typical of Tillman, who reminds me a little of David Foster Wallace, as portrayed in the movie The End of the Tour. It’s the way he interrogates his words as soon as they come out of his mouth, peeling back layers to find the thought within the thought, never living up to his own expectations.

Tillman spent his 20s holding back. Fleeing his family and religion, he worked in construction, made several sparse, morose albums as J Tillman, and landed a cosy but unfulfilling gig as the drummer in Fleet Foxes. A few years ago, he quit the band, upended his life and reinvented himself as Father John Misty with the notion of using an exaggerated version of himself as a Trojan horse to tell the whole truth for the first time. His 2012 album Fear Fun was a gruesomely funny account of a period of self-loathing debauchery in Los Angeles, from which he was rescued by meeting and marrying photographer and film-maker Emma Garr. I Love You, Honeybear, his 2015 breakthrough, was a warts-and-all song cycle about allowing yourself to fall in love when you are a wounded cynic. The music had the confident lushness of 70s Los Angeles while the lyrics’ brutal, hilarious honesty drew comparisons to Leonard Cohen and Philip Roth, two of Tillman’s heroes.

Father John Misty’s Pure Comedy.

Tillman thinks those albums “weren’t particularly zealous for people’s admiration”, but they made him moderately famous (his admirers include Beyoncé and Lana Del Rey) and frequently misunderstood. He keeps having to explain that his stage name is not an ironic alter ego and that his dry, absurdist sense of humour – on record, on stage and online – doesn’t mean he isn’t serious.

“It drives me insane to see people say: ‘Josh Tillman, the person behind the Father John Misty persona,’” he says, jabbing the air with his cigarette. “People can either accept that I mean what I’m saying or think that it’s some kind of mumblecore, beta-male, self-aware trickery. The truth is somewhere in the middle. All of my music exists in the middle. People who want to see things in stark dualities are not going to get much out of my music.”

He is aware that this tightrope-walk is not for everyone. A few weeks after we meet, his performance on Saturday Night Live will prove typically polarising, not least because the song Total Entertainment Forever features the red-rag-to-an-online-bull line: “Bedding Taylor Swift every night inside the Oculus Rift.”

“I’m not bamboozled by the fact that people are disgusted by me,” he says. “I’m not my biggest fan either. The criticisms that people have of me now are the same criticisms that my teachers, parents and pastors had of me as a child: that I talk too much, that I’m loud, that I’m self-obsessed. It’s the same shit.”

The current internet culture rarely rewards the qualities Tillman is interested in – ambiguity, nuance, the tap-dance between irony and sincerity – and he has discovered that if something can be misread for the purposes of click bait, then it surely will be. “Outrage is the new entertainment,” he says. “That’s what Twitter is for.” Last September, he abandoned social media altogether, for his own sanity. “As someone who has depression, there are days when I’ll wake up and say: ‘I’m not going to allow myself a single happy thought today.’ I get sick pleasure out of going on the internet and reading about how much people hate me. That’s deranged. That’s the very dark underbelly of what people see as my fanciful social media presence.”

Father John Misty performing in Leeds last year. Photograph: Andrew Benge/Redferns

Occasionally his discomfort got the better of him in public, like the stomach-clenchingly uncomfortable interview in May last year with Mark Radcliffe and Stuart Maconie on 6 Music, which was widely reported using the words “car” and “crash”. He winces when I mention it.

“Sometimes I just suck,” he says, sighing. “The truth of the matter is it has not been a good year. I have substance-abuse problems. That morning, I was completely fucked up. This was Fear and Loathing level. I’ve got these lizard men with English accents doing this Laurel-and-Hardy act on me and I just couldn’t deal with it. It’s not as if those guys were so horrible. It’s me. It’s my fault.”

When I first met Tillman, in New York two summers ago, he was in a rigorously abstinent phase, but tonight he is smoking American Spirits with a vengeance and steadily depleting a bottle of tequila. I begin to ask what changed. Last time we met you were …

“In a better place? Yeah, my life is a mess.”

What happened?

“Well … it’s really personal. There was something that happened where I was like, ‘fuck this’, and I gave myself permission to go back into self-destruction.”

He drums his fingers on the table and gazes into the darkness. “This is so hard because I hide in interviews. I’m not sure I have ever given an honest interview. I’m not like this when I’m just hanging out, I’m actually kind of funny. But when the gaze is turned on me I freak out, and that’s the public gets to experience; the person who is squirming under the microscope.”

While I Love You, Honeybear’s popularity brought Tillman an uncomfortable degree of attention, it also opened up new opportunities. He cowrote songs for Lady Gaga and Beyoncé because he wanted to “know how this sausage is made, just out of morbid curiosity”, although he has turned down every offer since. He also met with several major labels, making promises to take his career to “the next level”, a phrase he calls “a Jedi mind trick”. He decided to stick with the independent labels Sub Pop and Bella Union, and his producer Jonathan Wilson. Whatever awaited him on the next level, he didn’t want it.

“Tony at Sub Pop says: ‘People can sense that you could take it or leave it.’ And that’s true,” he says. “This is the point when most indie bands sign to a major label and put out a record that disappoints everybody. I’m just doubling down. There’s a detachment in me which turns people off because it comes off as arrogant, but that’s the very thing that allows me to say what I need to say. That’s what freedom is: to be able to take it or leave it. I passed what I wanted in terms of success miles and miles ago. It’s all gravy now.”

Leaving LA, which took Tillman three years to write and features a startling string arrangement by the composer Gavin Bryars, illustrates the scale of new album Pure Comedy’s artistic, rather than commercial, ambition. It’s an extraordinary, panoramic tour of human nature whose cast includes an angry deity, Ballardian revolutionaries, virtual-reality addicts, narcissists, utopians, cynics, lovers, fighters and a dying man who “first checks his news feed to see what he is about to miss”. It’s both scathingly satirical and profoundly compassionate. He doesn’t want to give the impression that he is “wagging his finger at people from atop Fuck Mountain”.

“I’m not some smart guy who is observing the world and dismantling what’s wrong with it,” he says. “That’s not the truth. Life is pure comedy? That’s not reality. That’s some hurt child that lives inside of you.” A beat. “You can put ‘laughs bitterly’ after that.”

Father John Misty’s Two Wildly Different Perspectives.

It’s getting cold on the balcony, so we adjourn to Tillman’s room. He pours a mug of tequila, lights another cigarette and talks about the messy conclusion of the I Love You, Honeybear tour. At a festival in New Jersey on the day after Donald Trump accepted the Republican nomination, he played just two songs and delivered a long, despairing monologue about the price of apathy. In a tweet afterwards, he alluded to Amusing Ourselves to Death, Neil Postman’s classic 1985 book about the relationship between entertainment and oppression.

He coughs a bleak laugh. “When I was writing this album, I thought Hillary Clinton was going to be the president and I was going to be this little shit-kicker saying, [whiny voice] ‘Well, think about it in a macro sense.’ But even if Clinton had won, Trump was the other candidate. Something is wrong if that can happen. It’s too easy to say that racism and misogyny made Trump. There’s some other variable, and I think it’s about passivity and entertainment and oblivion. And I’m an entertainer, so I’m complicit. That kind of self-satisfied, self-aware apathy about the state of affairs in America somehow helped this whole thing go down.”

On election night, Tillman was sitting on the fire escape of a bar in Los Angeles when he heard a voice from the television inside. It was Mike Pence introducing “the next president of the United States of America: Donald Trump”. Tillman mimics a horrified double-take. “In that moment, it was like all of the Gen-X humour that I was weaned on had this very cruel orgasm in my mind. In that moment, satire died. We’re now in a post-satire world because this is the stupidest thing that could ever happen. It’s like bad comedy. I just can’t totally verbalise how tragic I think it is. I feel like the boy who cried wolf. All this scepticism and cynicism that I have felt my whole life became so literal.” His eyes are glistening slightly. “People are really scared. It’s not theoretical any more.”

Given this alarming new context, Tillman is now working out how to translate Pure Comedy to the stage. “I have spent the last six months on cocaine in my underwear writing the most indulgent musical – I’m being colloquial – with dancers and costumes and sets. The show started with six Father John Mistys dancing around a bonfire.” He shakes his head. “Completely insane.”

Now he is changing tack. “This thing needs to be direct. Pure communication. Look at my last album cycle, which started with me with a fresh haircut, yukking it up on David Letterman, and ended with me bedraggled, out of my mind with despair and panic, yelling at an audience about entertainment. I would rather take that guy out into these shows than the guy at the beginning.”

Creatively, Tillman is on a roll. He has already written his fourth album and sketched out his fifth. “When you add melody you change the type of thoughts that you are capable of having,” he says. “That’s what I hear in this record. I’m starting to think in song.”

Throughout our two-and-a-half-hour conversation, Tillman is constantly pausing, doubling back, amending, apologising, except when he talks about songwriting. Then he sounds unstoppable. Music is the place where his whirring brain can find optimism, clarity and faith. It’s what enables him to tell the truth.

“It’s depression,” he says. “It’s despair. The music is the times I can get my head above the water and make something out of it. All I can do is quote my own lyrics. Those are the most true things I will ever say. Everything else will just be bullshit.”

Pure Comedy is out on 7 April on Bella Union.