In 1965 Robbie Robertson was living in the room next to Bob Dylan’s at New York’s Chelsea hotel. This was when Dylan was writing Blonde on Blonde. “The television was on. There was music playing. The phone was ringing. There were people coming and going – and he was writing away on his typewriter. I thought, ‘I don’t even understand how somebody can close off the outside world like that and concentrate. This guy is from another planet,’” Robertson says. But for a while he shared that planet, or came as close to sharing it as any musician did at the time.

Robertson was the lead guitarist of the Band (then known as The Hawks), the five-piece group that backed Dylan in his early days as an electric act: essentially, they supplied the noise that the acoustic-loving crowds booed on tour. But while the collaboration changed the course of music history, it had another, quieter and more personal effect on the Band, shifting the dynamics of what Robertson calls their “brotherhood”, the way the five of them related.

This “brotherhood” is clearly on Robertson’s mind, because after a lifetime of writing steadfastly unautobiographical songs, the most powerful track on his new solo album, Sinematic, mourns the demise of his band family – three of whom are dead. His new documentary, Once Were Brothers, which shares its title with that song, sees Robertson pottering around his music room, looking back on what made the Band tick as a collective, and what he lost in losing them. It is fair to say he is in a memoiristic phase.

Facebook Twitter Pinterest The Band at Woodstock, 1968 (from left): Richard Manuel, Robbie Robertson, Levon Helm, Garth Hudson and Rick Danko. Photograph: Elliott Landy/LandyVision, Inc

“For a long time I thought I wanted to be private,” Robertson says, on the phone from Los Angeles, where he lives. “But at this stage” – he is 76 – “it all comes flowing out and I can’t stop it. I would put our story up against any musical group in the history of the world.”

Eric Clapton, Bruce Springsteen, Van Morrison, Martin Scorsese (the film’s executive producer) and George Harrison – as well as Dylan – are among the greats happy to say how great the Band, and Robertson in particular, were. Clapton even disbanded Cream – whose sound had become increasingly strident – after he heard their debut album, Music From Big Pink, in 1968, with its quiet, homespun roots air. But while Robertson’s music with the Band has touched the work of so many important musicians, he is still mostly regarded as interesting for whom he influenced rather than in his own right, like a person who has many claims to fame, but somehow isn’t a celebrity himself.

I wonder if he minds being of interest primarily for music that he made so long ago, for his relationship to, and impact on, other, more famous, musicians. “It’s a big part of the journey and a big part of the story, and I completely understand that,” he says. “But it’s not what I do. It’s what I did.”

Robertson was born in Toronto. His mother, who was Mohawk, had grown up on the Six Nations of the Grand River Reserve, and as a child he used to visit family there. “It seemed to me that everybody played a musical instrument or sang or danced. I thought, ‘I’ve got to get in on this club!’ I said, ‘I think a guitar looks pretty cool.’” So his mother bought him one – with a cowboy painted on it. “I thought it was very ironic that Indians would teach me to play guitar with a picture of a cowboy on.”

By 15 he was proficient enough to be hanging around rockabilly Ronnie Hawkins when he played in Toronto, captivated by Levon Helm on drums, sticks flying, white hair blazing and only three years Robertson’s senior. “He seemed to glow in the dark,” Robertson says. Robertson was an only child; Helm became a kind of sibling. “With Levon I was the younger brother, learning and growing.”

Facebook Twitter Pinterest Bob Dylan and Robbie Robertson go electric at the Academy of Music, Philadelphia in 1966. Photograph: Charlie Steiner/Getty Images

Fellow Canadians Rick Danko, Richard Manuel and Garth Hudson joined the band to complete the brotherhood, and after they split from Hawkins, they continued to play as Levon and the Hawks. Helm was the nominal leader, Hudson was classically trained and reclusive, Danko good-looking and soulful, and Manuel from the start was cherished as sensitive and vulnerable: “From a very young age, we understood that Richard had a difficulty with alcohol and addiction,” Robertson says. (Manuel killed himself when he was 42.) And what of Robertson? What part did he play in the brotherhood?

“Everybody did something that raised the level of what we were doing to a stronger place,” he says. His “something” was “the main part of the creative direction of where this group would go.” He became less of a brother and more of an adult in the room. He negotiated with the band’s manager – Albert Grossman, whom they shared with Dylan – took a growing charge of the songwriting and, as Danko, Helm and Manuel became increasingly interested in heroin, he tried to keep them focused on music.

It must have been tough to play both parent and brother. Didn’t it create a sense of difference between him and his bandmates? After all, Helm was critical of Robertson in his 1993 autobiography, This Wheel’s on Fire, for the way he distributed songwriting credits (Robertson got most of them). “I didn’t feel any separate feeling,” he says. “What everybody gravitated to was what they could bring to the table. So they leaned on me, in the best sense of the expression, to really take care of business.”

In his 2016 memoir, Testimony, Robertson writes about his parents’ alcoholism, and that of his wife Dominique (now ex-wife, but they remain close and both live in LA, as do their children). He also logs the uses and abuses of his bandmates, especially Danko, Manuel and Helm’s use of heroin. Did he try it? “Yeah, it wasn’t for me,” he says. “But if you smoke grass or take a psychedelic or if you did coke or anything, you were in no position to be scolding anybody.

Early starter … Robbie Robertson.

“And I didn’t know anybody that didn’t do drugs. It was such a normal. But what I did discover was that when it got in the way of what I wanted to do, when [I felt] ‘Holy – ! I’m going to die if I keep living this way,’ I went in the other direction. My addiction was work.” He pauses. “And still is,” he adds. He has a habit of bringing his recollections back to the present, and for someone who celebrates a love of storytelling, his enjoyment seems rather ambivalent. I can’t help feeling there is only so far he wants to go.

Robertson also regards Dylan as “still part of the brotherhood”. He was 22 when they met. They became so close that Robertson was the only witness to Dylan’s wedding to Sara Lownds in 1965. And then, of course, there were the electric tours, all those angry audiences. “It was like we went through the war together,” Robertson says. “We’re going to be in it for ever, just because of what we went through.”

“When people boo you night after night, it can affect your confidence. Anybody else would have said, ‘Well, the audience isn’t liking this, let’s change what we’re doing.’ We didn’t budge. We went out there and just played this music and the more they booed, the louder we got. Inside of us we felt, ‘This is a revolution. And we’re part of this revolution, and we’re going to go through with it.” Later, of course, came The Basement Tapes – weeks of laidback, lumberjack-shirted recordings with Dylan from the Band’s Woodstock home.

In this light, Dylan’s contribution to Robertson’s documentary – the Band were “gallant knights” for standing behind him – seems slight. But Robertson says it is “extraordinary” that he contributed at all. “Bob doesn’t do anything like this, ever.”

The two are no longer close. Robertson was the only member of the Band not to feature in Dylan’s 30th anniversary celebration in Madison Square Gardens in 1993. He was “in somewhere else”, he says, as if that were a place.

And although a few years ago he and Dylan ran into each other at an event and “caught up a bit”, they have less in common now. “He’s always on the road, and I’m never on the road. He’s out there, and I think that keeps the blood flowing for him, and I admire that very much. But that’s not what I’m interested in … I haven’t talked to him in a long time.”

After Dylan watched Once Were Brothers, so he could sign the release, “he called me and said, ‘My God, I started watching this, and I got completely hooked. I love this film!’” He especially liked the part where Robertson talks about his mother, whom Dylan also knew (the two musicians lunched with each other’s mothers: “I know!” Robertson says. “It doesn’t seem like we would be the type!”) Now he and Dylan “have got an idea that we might pursue together, so we’ll see what happens”.

Facebook Twitter Pinterest ‘I said, guys, I’m not going back on tour with you. I had this fear inside’ ... Robbie Robertson. Photograph: David Jordan Williams

Robertson has just finished the music for Scorsese’s new film, The Irishman. Sinematic was released this month, he is writing the second volume of his autobiography and assembling the 50th anniversary collection of the Band, out in November. “I’m not lazy!” he exclaims, a realisation he describes as “a wake-up call” but he sounds happy to be busy, and I’m not convinced it’s news to him.

“I am so into what I am doing now,” he says. “I do revisit special things in my heart about this brotherhood. But I’m very much, like, 90% of it is what I’m doing today and what I need to do tomorrow. I’m always wanting to move forward.”

It was Robertson who organised the Band’s farewell concert, The Last Waltz, in 1976, partly, he says, “because of my deep concerns for Richard [Manuel]’s wellbeing. It was shattering to me when Richard died. I was afraid of this for many years. The idea was to be protective. Because although we were all riding in the caravan, some people were much more vulnerable to the disease of addiction than others.”

Dylan, Joni Mitchell, Neil Young, Van Morrison and others contributed to the show, which Scorsese also filmed – the start of a long working relationship. Robertson hoped that afterwards the Band would regroup in the studio. But he was the only one to turn up. He has always depicted the end of the Band as a surprise – but this seems at odds with the concept of a supremely elaborate farewell concert.

“Our plan was we were going to put the road aside, to take a break. And then, as I said in the film, when I realised everybody forgot to come back, I had to read the writing on the wall,” he says.

That word “forgot” feels like a bit of a narrative liberty. After all, the other members of the Band reunited in 1983. Even after Manuel died in 1986, Hudson, Helm and Danko carried on – without Robertson. So didn’t he leave them? “They got in touch and said, ‘We want to go out and do some playing, and is it OK that we use the name the Band?’ Because they didn’t have a creative outlet like me.” By then Robertson was writing music for film.

“I said, ‘Guys, go ahead. If you want to go in and make a beautiful record, I’m first in line. But I’m not going back on the road with you.’ I knew what was going on with Richard, Rick and Levon, still. And I had this fear inside. It could be dark. And it could be dangerous. And as it turned out, it was.”

I can’t help wondering whether memoir is a way to self-soothe, to sort through the difficult circumstances that led to the Band’s demise, the pain of the distance from Helm in particular towards the end of his life. But Robertson sounds perplexed by the question. He says he is only celebrating their amazing journey.

The other living member of the brotherhood is Hudson, who is 82 and lives in Woodstock. Archive audio of his voice features in this documentary, but no new interview. “Garth is a recluse and he doesn’t talk,” Robertson says. “He has a health issue. I don’t think it would be respectful to Garth to show that he is not feeling that well, and to not be able to show him in a shining light.”

So it falls to Robertson to tell the Band’s story and, by telling it, to own it. Does this inequality with his brothers trouble him, especially since at its heart the Band was a celebration of the collective?

“Inequality to the other guys?” he asks, sounding puzzled. “Like I said, early on they just kind of pushed me out there and said, ‘Tell them what we’re trying to say.’ And because I was the creative writer in this group, and I also have the ability to … whatever you call it ... lead the charge, everybody enjoyed that I would take that responsibility. That I would do the dirty work in that way. So it fell on me. It fell on me and I respect it and I try to tell the most honest version of the story I can.”

It’s a fair point, yet for a memoirist, he seems pretty averse to reflection. “I’m not going to get moopy or soppy or mushy,” he says. “It’s just not the cloth I’m cut from.”

• Robbie Robertson’s album Sinematic is out now. Once Were Brothers premiered at the Toronto film festival in September.

• This article was amended on 29 October 2019. An earlier version said the Hawks backed Dylan “when he first went electric”. To clarify, Dylan’s first electric appearance, at the Newport folk festival in 1965, was with members of the Paul Butterfield Blues Band.