Beck Hansen has been sitting on his new album – his gleeful, giddy, what-a-wonderful-world-we-live-in album Colors – for at least a year now. The Californian, who records as Beck, began writing tracks for it a while back, “when the biggest song in the world was Happy by Pharrell Williams”, he recalls. As the veteran composer of a dozen albums since the 1990s, some of them classics of their day, Beck was aware that his modern output had tended towards the downbeat, the inward-looking. It seemed an obvious next move to make a super-duper-happy record. Time to unleash the fun! Colors was written and recorded and ready to go by November 2016, just as Donald Trump secured his place in the White House. Recalling the timing of this now, while sitting in a London bar, Beck frowns: “It was coming out literally the month of the election. And it just felt, uh... It didn’t feel like the right time for those songs.” Colors was slipped into the industry equivalent of a dusty cupboard, its release held off indefinitely – until this month.

Is the world any more cheerful than it was this time last year, though? “I hear what you’re saying,” Beck says. “I sort of had the sense [when I started writing Colors] that we were going into challenging times, but I really worked hard to stay away from anything that was too down, or negative. I guess I could have gone and made a different record, but we put so much work into these songs, it had to see the light of day.”

He is 47 now, a married father of two, but Beck retains the loose and laconic manner of the twentysomething he was back in 1994, when his breakthrough single Loser became an anthem for those Generation X-ers who were oddly proud to define themselves by their inadequacy. He still expresses amusement in textbook Gen X fashion, not so much laughing out loud as drawing in a sharp, reluctant breath: Huh. Time has passed, but Beck can still pop a crowd, and the night before, at an intimate London venue, I watched him play for a group of British fans. Fresh off a plane, he eased himself into proceedings with some of his gentler folk songs, then shook off his jetlag and ratcheted up the tempo for a wild hour of discography-trawling that put this musician’s vast catalogue (two parts rock, two parts folk, one part dance, one part rap) on splendid display. By the gig’s end, Beck had the crowd roaring such loud approval that, 12 hours on, his ears haven’t stopped ringing.

It must be gratifying, I say, to be 23 years into a career and still able to electrify a room like that.

Beck smiles. “I think of last night as what I’d hoped or imagined performing would be like when I was starting out – which it wasn’t. Because at the time, my music was so confusing to people that they didn’t know how to react. Unless it was to laugh, or throw things.”

Should we go and hide in our corner? As time goes on I’m more interested in being in the place you’re not supposed to be

As fans might have guessed from his lyrics, which can recall the smooth, rhythmic nonsense of Edward Lear poems, Beck is a gently peculiar guy. One of his hobbies is Instagramming random appearances of his own name, on billboards, on passing trucks, in celebration of what he calls “gratuitous namesakes”. He’s a Scientologist, something I’ve been forbidden to ask about. (When Beck has spoken about his religion in the past, he’s described it as “something personal”, inherited from his father.) A messily handsome man in general, he is oddly sensitive about his profile, which is unexpectedly triangular and about which he says: “Photographers always ask to do some from the side. Until they see how I look from the side, and then they ask to do some more from the front.” In his 20s, Beck looked barely teenaged and now, as he pushes 50, his age is only really evident in the way he moves, especially on stage, where his dancing (once legendarily silly) has become more restrained and limited to basic arm swipes.

When I ask about this, Beck speaks, obscurely, about a recent accident. A fall, he says, without offering more details: “Hopefully it gets better.”

He can be like this to talk to, veering, like his lyrics, between the cryptic and the explicit. Our conversation doesn’t really start to flow until we get talking about Bono, of all people. At his gig the night before, Beck took a good deal of teasing about the fact that he’d recently supported U2 on a tour of American stadiums. “Fuck Bono!” someone shouted from the crowd, and Beck, not quite hearing, replied, “Yeah!... Wait... What did you say?” When the shout was taken up by multiple voices, Beck seemed a bit flustered and tried to defend the U2 frontman. But: “Fuck Bono! Fuck Bono!” Eventually, he had to sigh into his mic and say, “I mean, I’m not gonna argue with you.”

Facebook Twitter Pinterest Photograph: Rachael Wright

Today, he seems baffled by the exchange. But this was a room full of his diehard fans, people who, I suggest, might prefer to imagine he exists a little farther out on the cultural fringes than Bono. Beck takes the point, but isn’t especially pleased about it. “So, so, so, should we go and hide in our little corner? Or should we go and engage with the world? As time goes on, I’m more interested in being in the place you’re not supposed to be. I don’t know what the rules are.” He sighs. “Believe me, I definitely spent a lot of time in my music-making feeling very removed or separate from mainstream culture. But then, after a while, I realised, y’know, why marginalise an idea? Let’s just bring it to the centre. Let’s engage.”

***

Beck was born in 1970 and raised in east Los Angeles, which in the 70s and 80s could be a forbidding place: “I grew up in the Area That Nobody Drives Through.” There were palm trees near his home, but they were infested with rats. He had a fake ID he used not to get into clubs, but into ceramics classes at an adult community college, to avoid unwanted attention on the streets. “In that environment you’re pretty – what’s the word? – disconnected from things. There wasn’t, like, a sports centre where you could go and join a team and meet a bunch of kids and make friends. It was more like growing up in a warzone.”

And you’re a slender guy, I say. You must have been tiny as a kid.

“Yeah. So, I mean, it was just survival, really.”

His parents are well-known, to a degree, but not in a way that brought any glamour to Beck’s early years. His mother, Bibbe Hansen, is a former performance artist who as a teenager worked with Andy Warhol. And though his father, David Campbell, has gone on to make a successful career as a composer, when Beck was young he was a jobbing guitarist. “Anybody who’s lived in poverty [would recognise it],” Beck says of their life then. “No space. Sometimes food is there and sometimes not. Living in an area that’s very dangerous, and the stresses of that.” His parents separated and resettled with other people, a stepfather later describing Beck, as a youth, as less like a little boy and more like a “little artist”. When I mention this, Beck says, “Within those very narrow options, it was probably the only thing you could really do: create your own little world.” David Bowie was one of his heroes, and later in life Beck went to see an exhibition that featured a reconstruction of Bowie’s childhood bedroom. It prompted a pang of recognition. “It seemed like he’d created this whole little fantastical world for himself, as a way of transcending his limitations.”

Facebook Twitter Pinterest Beck performing at the SXSW festival in Austin, Texas, 1994. Photograph: Catherine McGann/Getty Images

Beck dropped out of school and in 1989, aged 18, rode a bus across the country to New York: a multi-day journey during which a passenger advised him, bluntly, he would be “cut” for his Walkman if he ever fell asleep. He lasted a year or so on the east coast, plugged into the Manhattan punk-rock scene and, in the early 90s, returned to LA, where he supplemented the beginnings of a music career by working in a video shop, sorting porn, he once claimed. He was with a serious girlfriend at the time, a fashion designer called Leigh Limon, and he had a certain fearlessness that let him get up in front of crowds and sing the kooky, incomprehensible songs he was writing at the time. But Beck once referred to this period as a time of “zero money and zero possibilities”. Change when it came was rapid. “Early ’94’s most unlikely overnight-success story!” Rolling Stone wrote of him after Loser went top 10. He signed to a major label, DGC, and released his first LP, Mellow Gold.

Beck has been a hungry and prolific worker ever since, not only as a musician but as a creator-for-hire, publishing sheet music, composing tracks for PlayStation games, designing spectacles. Though he comes across as someone who would be happy in his own company, he has also been a tireless industry mingler over the years, making pals of Will Ferrell, Winona Ryder, Spike Jonze, Dave Eggers, Christina Ricci. He once traded memories of a misspent LA youth with Tom Waits, befriended Tom Petty, and duetted with Taylor Swift (his two kids, Cosimo and Tuesday, were excited by that last one). Meanwhile, he concocted a dozen albums, as well as losing more songs in his day than most have ever written. On a tour in the mid-noughties, he managed to leave the tapes of a record-in-progress in a dressing room; the music was never recovered.

Facebook Twitter Pinterest Beck with Taylor Swift in 2015. Photograph: Christopher Polk/TAS/Getty Images for TAS

When you have written so many, are some albums more memorable than others? Beck shakes his head: “I remember them all. I can put myself there like I can put myself back to last week. Easily.”

We talk about his break-up album, Sea Change, which Beck wrote and released in the early 2000s after his separation from Limon. He played it straight for once – to profound and wounding effect. “The producer, Nigel Godrich, had just broken up with his girlfriend [as well],” Beck says. “So he was really devastated and heartbroken, too, and we were making the record together in that sense.” It was Beck’s intended follow-up to Sea Change that he mislaid in a dressing room somewhere and, gutted as he was about that creative loss, he wouldn’t try to make a record in the same mode for more than a decade. Then, around 2013, at a time when Beck was happily married to the actor and writer Marissa Ribisi and living in Malibu with their two kids, he wrote Morning Phase, a companion piece to Sea Change that became its equal in both assurance and beauty. That album’s reception, Beck says, “took us all by surprise”.

More often than not, over time, his work has been nominated for Grammy awards. Beck is only slightly exaggerating when he tells me that, “I’ve sat there and lost probably 16 times”: he’s actually sat there and lost 11 times. With Morning Phase, nominated for a Best Album Grammy in 2015, things played out slightly differently.

Facebook Twitter Pinterest Beck collects his Grammy award for best album from Prince in 2015 Photograph: Frank Micelotta/REX/Shutterstock

Prince was on stage that night to reveal the winner, and dressed for the occasion in shiny orange pleather and carrying a silver cane. Beck was back in the stalls. He’d read all the internet commentary in the build-up, “saying there was no way, they didn’t even know how I’d got in the category”. And no wonder, Beck acknowledges, given the competition: Pharrell Williams (“untouchable”), Sam Smith (“the most popular record that year as far as sales”), Beyoncé (“people loved her record”). So no, Beck says, he was not expecting to win. Which seemed a shame on the night, if only because he would have liked the opportunity to talk to Prince. When the winner was announced – Beck – he stood, clearly stunned, and hugged his wife. Pharrell clasped his hand on the walk to the stage, and Beyoncé applauded and smiled. Then she was seen to mouth: “Oh no... Not again.” As Beck was receiving his award, Kanye West invaded the stage in protest, just as he’d done six years before when Taylor Swift won. (On both occasions, West felt Beyoncé should have won instead. “Come on, man, I love Beck,” West said later, “but he didn’t have the album of the year.”)

Beck and I discuss his memories of the quick, confusing moment that played out on live TV. He says he didn’t really register what was happening: that West came and went before he could properly react. “Afterwards, I thought if I hadn’t been so nervous, I could have had more fun with it.”

What kind of fun?

“I dunno, it could have played out a lot of different ways. We could have had a little conversation. Talked about life.”

Prince could have mediated, I say.

“Yeah! Exactly!” Beck looks off into the distance. “That would have been interesting.”

I wanted to tell Prince how much he meant to me. But when I turned back around, he was gone

It gets him thinking about Prince, an artist who had always been huge in Beck’s life. “It’s weird. While it feels incredibly presumptuous, I do feel that certain artists are like family, in a way. I never knew Prince, but he was so much a part of my world. There’s a lot of musicians like that.” Beck talks about some of them: Bowie, Leonard Cohen, Tom Petty. “I’ll never forget this conversation I had with my band a couple of years ago on the tour bus. I was saying, ‘We’re so lucky! We still have all these people!’” Not long after that chat, Beck recalls, they started to go like dominoes. “It just gets you in the heart.”

I ask if the night at the Grammys was the last time he saw Prince, and he grimaces. He had just received his award, he recalls. The speeches were over and the two men were walking together towards the wings. Finally, Beck thought, here was a chance to speak with an idol. “We’re walking off... I was trying to talk to him. Prince was listening. I wanted to tell him how much it meant to me that he was up there. Like, how much he meant to me. And it all happened so quickly. Somebody grabbed me to take a photo, because they always want to get your photo when you’re coming off stage, and when I turned back around, Prince was gone.”

Facebook Twitter Pinterest With wife Marissa Ribisi in 2008. Photograph: Katy Winn/Getty Images for IMG

Please tell me you caught up with him, I say.

Beck shakes his head: couldn’t find him. “And that was the last time. He died a year later. So.”

Beck isn’t sure what lesson he learned from this moment, except, perhaps, not to stop for photographs again. “But now? After the last few years?” he says. “If any artist ever says it’s their last tour? I’m going to be there.”

Will Beck have another run at the Grammys? After a year such as this one – with its political infamy, civic protest, natural disasters – will the blissed-out Colors have enough grit to it? Beck acknowledges there had at one stage been a more cynical version of Colors, a version with “a lot of things that I wrote that were maybe a little bit darker, or, y’know, were kind of commenting on things that were negative. But I would take all those lyrics out.”

Why?

“It’s probably my natural place, things that are Not Quite Comfortable, y’know? But I wanted to make something that felt good.”

He is very aware that right now things seem “untethered”. We talk about the mass shooting at a concert in Las Vegas earlier this month (“I think we’re all still processing it”), and about the attack on the Bataclan in 2015. (Beck shares a manager with Eagles Of Death Metal, who were on stage in Paris that night: “It felt close to home.”) Of this uncertain period in time, Beck says: “There’s apprehension. There’s a hoping-for-the-best. And then every month – right? – something else happens. But I like to think it’s something that as a society we can grow beyond.” He believes a musician’s role during difficult times is to continue trying to unite people. “[We’re in] this chaotic period, which I think has been growing for years. It’s like there’s been a storm coming, and it’s here, and if anything, I feel like it’s important not to take for granted these moments we can play music and come together.”

Beck: Colors review – a hymn to hedonistic pop Read more

Doesn’t a musician also want to document those difficult times?

“There is that,” Beck says. He talks about artists he thinks are doing a good job capturing the current mood, including Kendrick Lamar. But, Beck shrugs, it isn’t his style. “Some people are really good at taking diaristic things and weaving them into a piece of art. For me, I’m just trying to get a song to work on its own terms.” Each song he writes has its own needs, “and sometimes what it needs is something throwaway”.

He tells me a story about an upsetting experience he had in 2005, when he was living with Ribisi and their newborn son in central LA. “I got jumped at gunpoint. Two young guys. I didn’t have any money on me and they were going to beat me. Not having any money was a big offence, which I know because I’ve been in that position when I was younger – if you don’t have money, they just beat the shit out of you. That happened to me several times. But I told these two guys I’d just had a kid, that I had to look after my kid. And they shoved me around a bit, then they let me go.”

The experience meant something to Beck, though he couldn’t quite articulate what. He might have tried to weave this diaristic moment into song, only it felt too heavy.

“Sometimes, something very innocuous just feels better,” he says. “You might think, ‘It’s about nothing.’ But, in a way, the songs that we love are kind of about nothing. There’s power in a song being non-profound, throwaway. These songs can have a sort of magic. They can exist beyond the artist, beyond the genre, beyond the era.” Beck raises a hand, as if he’s launching a balloon. “They float”.

• Beck’s new album Colors is out now on Virgin EMI.

• Commenting on this piece? If you would like your comment to be considered for inclusion on Weekend magazine’s letters page in print, please email weekend@theguardian.com, including your name and address (not for publication).