His self-destructive urges have more than once threatened to completely derail his career – but the comedian turned Hollywood star says he has never been more focused and driven

"What's your life like?" Without waiting for an answer, Russell Brand channels the monotone certainties of a fairground psychic and hazards a guess. "South London, no proper boyfriend-y person at the moment – difficulty with relationships, I think, cos you're an interesting combination of sort of strong, and then I think you get a bit groundless. That's my guess about you. I bet you've got good sexual energy, probably not too difficult to bring about an orgasm in the right hands. Relatively satisfied with your work, but beginning to have an inkling that there could be some unfulfilled project, something that you could write that would be brilliant but you've not got round to yet. Curious, probably, about spirituality, but not entirely committed yet."

Brand is very good at accelerated intimacy, even if accuracy isn't his strong point. He's wrong about me on just about every count, I tell him, laughing, but then he clearly stopped caring about being right in the conventional sense a long time ago. Having got so many big things famously wrong – from coming to work dressed as Osama bin Laden the day after 9/11, to making offensive prank calls to an elderly actor – the comedian has broken almost every professional rule in the book, and critics have been predicting his terminal disgrace since before he was even all that successful. And every single time, Brand has proved them wrong. The Sachsgate scandal in 2008 was supposed to be the final straw – since then he has become a Hollywood star, married and divorced a pop princess, and now here he is, the very acme of respectability, promoting Comic Relief concerts to raise money for abstinence-based addiction treatment.

Brand in London in 2001. Photograph: PA Archive/Press Association Ima

Brand spent most of his 20s addicted to more or less everything – crack cocaine, heroin, prostitutes. The chaos provided some remarkable material for his memoir, My Booky Wook, full of anecdotes, including one about introducing his crack dealer to Kylie Minogue. That same chaos is now providing the basis for his latest incarnation. Having made a BBC documentary about addiction last year, debated the war on drugs with Peter Hitchens and given evidence to the home affairs select committee, Brand is establishing himself as a public-health campaigner. He may look like a pantomime pastiche of a rock star (the rubber trousers he's wearing when we meet would be be useful as a puncture repair for a JCB), and he sounds weirdly like Ricky Gervais (if you close your eyes the effect is disorientating) but his argument is very simple. We're wasting our time trying to punish junkies with laws that don't work, and wasting their lives by dishing out methadone, which simply substitutes one drug for another. Addiction is a disease – not a criminal offence, but a medical condition – and should be treated with compassion.

Brand knew he was an addict long before he ever took his first drug. "It was obvious from the way I watched TV as a kid – too much television, too many Penguin chocolate biscuits. I think what addiction is, for me, is it's a behaviour that's indulged continually, despite detrimental consequences. Some people can eat chocolate – and some people keep going 'til they get fat then they make themselves puke up. Some people can look at a bit of pornography, other people are looking at increasingly deviant pornography, watching it all night wanking themselves senseless."

He thinks roughly one in 10 of us are susceptible. "And if you have this condition, and I call it an illness, then drugs will address it really well, because they create a physical craving to accompany the psychological malady." That doesn't mean, he's quick to clarify, that prohibition is the answer. "It's not a moral or judgmental thing about drugs and alcohol; I don't give a fuck if people drink or take drugs if they've got no drug or alcohol problem. But for those people who become a menace to society, and a pain in the arse to the people that love them, there is a solution. And that's why I'm frustrated. 'Cos the solution's very, very obvious."

Brand's solution is abstinence-based treatment, along the lines of the programme his manager eventually frog-marched him into after catching him smoking heroin in the loo at a Christmas party. Brand is basically talking about 12-step fellowship programmes – Alcoholics Anonymous, Narcotics Anonymous, Cocaine Anonymous and so on – only he can't say so exactly, being "bound never to relinquish their anonymity", which leaves him in the complicated position of campaigning for something he can't name. "These programmes are highly successful, but more people need to be directed towards them. So without compromising the anonymity of the fellowships, we want to direct people towards these abstinence-based recovery programmes."

But isn't AA the one thing everyone already knows about? "You say that, but people don't know really. People don't know it's available to them. The mindset of an addict is trenchant in its very nature; you're in a circuitry of neurosis that's very, very difficult to break. Impossible, almost. So you're not going to just wander in off the street. People don't do that. So with Comic Relief we want to raise awareness that it exists, then fund further treatment centres and counsellors and different ways of directing people towards it. 'Cos people don't know it's an option."

To Brand's opponents such as Hitchens, the flaw in his thesis is obvious. A true desire to get better is the first principle of AA – but if addiction is really a medical disease, how can resolve be a necessary condition of recovery? "There are a lot of discrepancies," Brand airily concedes. "If you leave someone on a desert island with cancer, they're going to die. If you leave a heroin addict on a desert island, they are not going to die. There are loads of distinctions. To tell you the truth, the rationale, the semantics and the intellectualisation of the condition aren't particularly helpful."

Given Brand's own semantic insistence upon addiction's status as an illness, that seems a bit rich, and you can see why he drives Hitchens mad. "Doing standup comedy is a paradigm with which I'm familiar," as Brand says, so he tends to see off counter arguments as if they were heckles, deploying charisma in place of analytic discipline, which is effective and funny, if not always strictly legitimate. But he is absolutely sincere when he says: "All I care about is getting this help to people who need it."

The success rate for abstinence-based fellowship programmes is better than most, but still only about 25%. Brand knows he's one of the lucky ones – but sobriety is unlikely to deliver the same happy ending to someone still stuck in a tower block, unemployable and destitute. So although he's the perfect role model for abstinence in one way – glamorous, seamy, the very opposite of a prig – his own story is so unusual that you wonder how much can be usefully extrapolated from it.

"In the end not everyone is going to go off and be in movies and live some superficially sequined glitzy little life," he agrees. "But to tell you the truth, and this is easy for me to say, sat here in the monkey-themed offices of Comic Relief, but that's fucking bullshit anyway. You know that, I know that, everyone who's worth anything knows that there is no real satisfaction or gratification in, 'Oh look, you've got a nice car,' if you're not happy inside. So the thing that's valuable to me is that I now have the tools and facilities for dealing with life. And that would be beneficial to me if I was still living in a one-bedroom flat in north London, or of no fixed abode."

Brand with then-wife Katy Perry in 2011. Photograph: Rex Features

Nine years clean now, at 37 Brand's life today is a novel mixture of new-age spiritualism and the gonzo broadcasting on which his career was first built. Since his divorce last year from Katy Perry he lives alone with his cat, Morrissey, in a splendid LA mansion, and begins each day with transcendental meditation, followed by Kundalini yoga. "It changes consciousness, it's really good if you've had addiction issues. It's highly psychological, and very beautiful, and overwhelming, and real, and trippy." After that he goes to his office on the Warner Bros lot, "To have some meeting about a film that'll never get made, and then I go home and do a meeting about the TV show, then have a meeting about a film that might get made, so there's a lot of meetings."

The weekly cable-TV show, BrandX – part standup comedy, part talkshow, notionally topical but fundamentally anarchic – is much more recognisable to his early fans than his recent roles in big budget Hollywood comedies, being based around improvised riffs and stunts, for instance inviting the audience to vote on whether a porn star guest should fellate a drunken audience member. Opinion, as ever, divides on whether it's puerile or inspired, but either way it's an unusual package from someone who gives every impression of having just returned from an extended stay in an ashram.

"I don't want to get too high-minded," he hesitates. Go on, I say – this is the Guardian, after all. "All right, I'll give it a go. Here's a bit of bargain basement metaphysics and new-age rhubarb. The idea that we live our lives in the past, the present and the future is illusory. We only live moment to moment. But we forget that." He talks earnestly about humanity's essential oneness, about attuning into different frequency bandwidths of consciousness, eschewing superficiality and so on, all of which he puts down to his spiritual awakening. "Sometimes," he has often liked to joke, "I think I'm Jesus."

Being of a more prosaic disposition myself, I wonder if it's got a lot to do with having confounded so many predictions of his professional demise. Reading back through years of newspaper cuttings covering his career is not unlike reading a hormonal teenage girl's diary – Love him! Hate him! Love him! Hate him! – and if that was me they'd been writing about, I tell him, by now I'd be feeling pretty supernaturally invincible.

"One of the things I've learned is not to live my life through others' perspective on me, as it is irrelevant. My experience of being alive ain't contingent – thank fucking God – on what people think of me. Now when you first get famous, there's nothing more gratifying or exciting than reading that people like you. What one quickly – or in my case slowly – learns is that it's irrelevant what other people think of you. It's none of your business. Now of course I require a certain number of people to like me for my livelihood. But I'm beyond the point where I need to do a head count. All I care about now is having an intrinsic relationship with what I do, as a performer, that's legitimate and real and authentic.

"It's not like I don't care, in some super aloof cool way. It's the same reason why I don't go to nightclubs. Not because," and he adopts a comic posh drawl, "'Oh maa-an, this is so trash.' No, because I know what it will do to me, I know what it will awaken in me and stir up. So I don't look at newspapers no more. We've only got a short time here and I can spend my time stimulating my mind however I want. I can read whatever I want. I ain't ever reading another copy of the Sun until I've read the complete works of Goethe."

Few A-list celebrities have written so candidly about their own ravaging hunger for fame – but now that victory turns out to be on the pyrrhic side, where does that leave his ambition? "I'm still ambitious, I'm very driven, but all I want to do is carry on doing things like this. Make people feel better in a way that is truthful and legitimate. Not temporary, in a sugar rush of nonsense." What does that mean? "To tell you the truth," he jokes, "I'm primarily working on my career in the world of psychic readings."

He's presumably not planning on any more prank phone calls, and when I ask about the Australian DJs' hoax call which led to a nurse's suicide he looks suddenly bleak. "Poor sods. Everyone involved. Poor sods. Those poor DJs just mucking about. That poor nurse. Again, it's the duplicity of these media confections in which both you and I participate. The faux shock, the pseudo concern, it's all a pose, it's all a construction. It's bollocks, the whole thing's bollocks. That woman obviously wouldn't have committed suicide unless there was a great deal of other stuff going on in her life. Those DJs wouldn't have done it if they'd imagined those would be the consequences. It's just bollocks, isn't it? And I'm reluctant to invest any more ink in it."

And yet, when he appeared in the Olympic closing ceremony last summer, dressed as Willy Wonka astride a psychedelic bus, singing I Am the Walrus, for a brief, heady moment the old impulse for catastrophic self-sabotage came over him again.

"It's kind of like a psychological vertigo – the knowledge that you can jump off an edge makes you want to do it a little bit. Like, what will happen to reality if I do that? Just in that moment, when I had that live mic in my hand, and I could say anything, and the knowledge that I could say anything to a billion viewers – and I think, oh my God, if I could do something like that, it would be almost just to watch the consequences – to see it all unfold. It interests me."

What would he have said? "Well what would be the worst thing you could say in that situation? The worst thing would be, given the climate of fear that exists in public events of that nature, something that would stimulate those fears." Something along the lines of, "I'm wearing an explosive suicide belt"? He grins enthusiastically, nodding. "That would be the kind of area, wouldn't it?"

So what stopped him? "'Cos I sort of think that when I have caused great big social explosions in the past, what I've most regretted is that it's either been flat-out wrong, or it's been kind of pointless. And I thought, oh, it would just be another one of those."

Clearly this was a good decision, but I wonder if the serene and spiritual Brand ever misses the metaphorical leap into pandemonium. "No, 'cos I'm going to do it again. But next time it will be worthwhile. I don't mind a little bit of chaos. I'm just waiting until something's worthwhile." He smiles knowingly, teasingly.

"What did Dorothy Parker say? 'They tire of quiet, that have known the storm.'"

• Russell Brand is appearing at Give It Up for Comic Relief at Wembley Arena on 6 March. More information: rednoseday.com/giveitup