“You think you’re the only person who wants to get a face-to-face interview with me? Get in line. I’m the best person you never met and one day, if you ever meet me, you’ll probably want to have my baby.”

That’s Mark Kozelek, via email, which is the way he chooses to conduct his interviews (“So I don’t get quoted with words like ‘dunno’,” he told Uncut last year, explaining his refusal to meet interviewers in person). He was – via his PR – replying to my questions about Universal Themes, his new album under the name Sun Kil Moon. Three weeks ago, I had tried to arrange a face-to-face meeting for this piece. As expected, the answer was a polite no: “He’s too scattered and distracted on tour to conduct an in-person interview and he will have to pass, but he said to tell you thank you for asking.”

Instead, I emailed my questions and approached his peers for supporting interviews, only later discovering that Kozelek has previously requested that journalists should not interview his associates for pieces about him.

Last Monday (1 June), Sun Kil Moon played at the Barbican in London. During the encore, Kozelek introduced a snippet of a new song he had apparently been writing. I wasn’t there, but a friend/colleague was, and phoned me after the gig. I’ve since heard the audio and it made me feel sick. “There’s this girl named Laura Snapes, she’s a journalist. She’s out to do a story on me, has been contacting a lot of people that know me,” he told the sold-out, 1,900-capacity room. Then he started repeating the line: “Laura Snapes totally wants to fuck me / get in line, bitch … Laura Snapes totally wants to have my babies.” The audience clapped and cheered. He played another song, then said he’d only been kidding around before: I was “cute”, “sweet”, “a good kid”. “She’s written some nice things about me.” Then he sang it a few more times before chiding himself. “Better stop before I make Pitchfork headlines for myself again.”

In 2011, Kozelek – after more than 20 years of making music, with Red House Painters, Sun Kil Moon and under his own name – fundamentally changed the way he wrote songs, introducing this kind of unfiltered immediacy to his lyrics. Backstage before a gig at Chicago’s Lincoln Hall, the Ohio-born songwriter came up with Sunshine in Chicago, a lovely, metred stream of consciousness about how the city reminded him of his dad, and all the Red House Painters had played there in the 1990s: “When we had lots of female fans, and fuck, they all were cute / Now I just sign posters for guys in tennis shoes.” It kickstarted 2012’s Sun Kil Moon album Among the Leaves, on which he ditched the heavy metaphor of his previous few records for off-the-cuff narrative sung over lilting classical guitar. Funny and self-deprecating, it transcended the tennis-shoe crowd and became his best-received album in a decade.

“His arrival at this very effortless-sounding flow is master-level songwriting,” says Low’s Alan Sparhawk. “I find it reassuring that it comes full circle to this place where you can’t even see the process and structure any more, and all you see is the moment, no matter how banal or sublime.”

The distance between art and artist narrowed again on 2014’s album Benji, which challenged the image of the confessional singer-songwriter. Kozelek was then 47 years old, having made a life in music since Ivo Watts-Russell of indie label 4AD seized on Red House Painters’ demo in 1989. Based in San Francisco, the Painters left 4AD in 1996 and split in 2001, but Benji was a partial tribute to Watts-Russell, whose patronage was Kozelek’s salvation. The musician had entered rehab for drug abuse at the age of 14; several of his schoolfriends died in senseless accidents; as an adult, his cousin and uncle were killed in separate explosions caused by burning rubbish in their yards. Escaping Ohio saved his life, and Watts-Russell transformed it. Benji was a heavy-hearted tribute to those who didn’t make it, to small but not insignificant lives. “She was only my second cousin,” Kozelek sang on the track Carissa. “But that don’t mean that I’m not here for her, or that I wasn’t meant to give her life poetry / To make sure her name is known across every sea.” It appeared in lots of end-of-year lists for 2014, the Observer’s Kitty Empire saying: “Kozelek … has nothing left to hide, or lose: the effect is utterly riveting”.

Sun Kil Moon, AKA Mark Kozelek: ‘I see myself more like a novelist or polygamist than a musician in terms of my output.’ Photograph: PR

“Benji was the first thing of his I heard and I was blown away,” says El-P of Run the Jewels. “There was something really refreshing about hearing such a direct writing style. No pretence.”

Kozelek’s candour wasn’t the only reason that Benji met with more attention. His live banter has often been antagonistic, but last September, a cruel streak emerged. He told a North Carolina crowd, “All you fucking hillbillies, shut the fuck up,” and repeatedly berated the War on Drugs after their sound from a neighbourig stage dominated his set at the Ottawa folk festival, calling them “beer commercial lead guitar shit” and introducing an old song as The War on Drugs Can Suck My Fucking Dick. A week later, he actually released a song called War on Drugs Suck My Cock, in which he also attacked the “spoiled bitch rich kid blogger brat” female journalist who reported his “hillbilly” comments. In another song, Adam Granofsky Blues, Kozelek read and laughed at the War on Drugs frontman’s bewildered comments about the affair.

The headlines churned. Was it “just a joke”? A performative response to the absurdity of becoming a minor indie celebrity after years of relative obscurity? Or did it betray uglier prejudices? The think-pieces about Kozelek’s violent language only irked him further. It’s the Jack White paradox: a musician rails that the press highlights trivialities rather than their pure art, while wholly aware that their comments perpetuate the gossip cycle, cultivate their persona and probably don’t hurt sales.

“I’m not going to defend him,” says Jimmy LaValle of the Album Leaf, who provided the melancholy electronic textures for their full-length collaboration, 2013’s Perils from the Sea. “He is who he is. There’s a lot that I’d like to say about things – what’s popular, EDM, people watching a show through their cell phone, talking over quiet music. But I very seldom address things that bother me. He does.”

This week, Kozelek released Universal Themes, Sun Kil Moon’s seventh album, recorded in San Francisco with Sonic Youth drummer Steve Shelley. Here, the music takes a heavy, electric turn and Kozelek’s stream of consciousness becomes even more abstract and intricately personal. Any given song might encompass his isolation acting on the Swiss set of Paolo Sorrentino’s new film, Youth, visiting a sick friend in Ohio, his HBO habits, cats, gigs, his waning fitness versus his rich appetite. The album’s sole references to 2014’s headlines come on Cry Me a River Williamsburg Sleeve Tattoo Blues, where Kozelek adopts the mindset of a bummed-out fan before breaking character to point out how much worse life could be, and This Is My First Day And I’m Indian And I Work at a Gas Station: “Some people love what I do and some get fuckin’ pissy / but I don’t give a fuck, one day they’re all gonna miss me.”

Universal Themes documents several broken teeth and Kozelek’s attempts to do more exercise, which we discussed in our email exchange. Was he feeling any fitter? “Yeah. Wait ’til you see me up there [on stage]. You’re gonna be like a cat in heat.” Apropos of nothing, he repeatedly issued instructions to watch for traffic. “Listen to your elders. I’m 48 and I have wisdom. I’ve seen girls laid out on the street with an ambulance picking them up because they are crossing the street with those stupid headphones on.”

In his few remarks about the record, Kozelek said he doesn’t reflect on the bigger picture created by any body of work: “I see myself more like a novelist or a polygamist than a musician, in terms of my output.” He describes Universal Themes as “more gratitude – a nice year in my life and a shout-out to friends I spent time with along the way.” As he sings on Garden of Lavender, the first song written for this album, “My heart is drawn to the small, out-of-the-way things that I can’t help but to give my focus and attention and care, ‘cause they shut off what hijacks my brain and help to tune out what can cause me pain.” He stays optimistic “because I got friends who I meet for dinner, and I love those dinners. And I’ve got a girlfriend who makes me happy, and I love her company, and I love to make music. And I got people I need to be here for.”

Kozelek is a noted boxing fan – on This Is My First Day … he recounts spending $17,000 on tickets for the recent Mayweather/Pacquiao fight, the result of Benji selling, he says, “enough copies that I’m living good!” Birds of Flims lists different boxers’ survival techniques: “Some win matches by throwin’ bombs like Mike Tyson, and some like Pernella are slippery and they win cleverly, some are fearless like Arturo Gatti.” Which does he identify with? “All of them. The confidence, the elusiveness, the vulnerability, the discipline, the focus. All of them. I was a fighter in another life.”

But in this life, Kozelek trades in sucker-punches. He impugns online “bitching and whining”, but hides behind one-way email exchanges, balks at the idea of his peers speaking about him and issues tirades (and sometimes, sexual advances) from the cowardly remove of the stage, with the get-out clause that it’s a performance. He can use sexually violent language to reduce female critics to the status of groupies, knowing that while male musicians’ misogynist acts are examined for nuance and defended as traits of “difficult” artists, women and those who call them out are treated as hysterics who don’t understand art. “The world don’t owe us shit, I learned that real fuckin’ young,” he sings on Universal Themes’ Little Rascals. If anything remains to separate Kozelek from his work, it’s that his music preaches that the least we owe one another is decency.

• Universal Themes is out now on Rough Trade.