From crack addict and alcoholic to the highest-grossing film star of all time, Samuel L Jackson is well aware his astonishing story could have had a very different ending. Here, as two new movies hit our screens, he talks about being recruited by Prince Charles, the optimism of Obama and – of course – the joy of golf

Samuel L Jackson, who was once a crack addict and a heavy drinker but is now Hollywood's highest-grossing actor, is telling me why he doesn't attend AA meetings in Los Angeles.

'It's just too weird,' he says. 'You hear guys saying stuff like: "I've been hitting the red wine too heavy and I need to stop, but I want to keep smoking reefer and doing cocaine."'

He lets loose that high, mocking laugh that is one of his onscreen trademarks. 'In New York, rehab is for real,' he continues, getting into his stride. 'You sit next to guys who were IV users, guys who stole shit, guys who sold their bodies. In Los Angeles you're sitting next to a guy who wants to go easy on the fine wine. Man, that's a symptom of something right there.'

In person, Jackson is engaging company, self-confident and voluble. He doesn't so much talk as hold forth. In full he can sometimes come across like a character from one of his films - maybe Jules, the fast-talking hitman from Pulp Fiction, albeit without the biblical quotations. Sprawled across an armchair in a suite in Claridge's, he is never less than utterly at ease, and belongs to that rare breed of celebrity - though he is uncomfortable with that reductive description - that seems to enjoy the interview process. Put simply, he likes to talk about himself. 'For a long time, when I was doing drugs and stuff, nobody knew who I was,' he says at one point. 'Now I'm sitting here talking to you and people are interested in what I've got to say.'

Jackson is trimmer than I expected, and a whole lot younger looking than his 59 years. He has dispensed with the Kangol hat that, worn backwards, became a sartorial signature of sorts for a while back there as his hairline rapidly receded. Now, proudly shorn and looking dapper in a bright-blue sweater and designer jeans, he sips on tea and iced water throughout, exuding a degree of self-confidence that, in a less charming man, could be mistaken for arrogance. There is an edge to him, too, though - an almost imperceptible flicker of some darker, nervier energy that pulses just beneath the cool, calm exterior.

'I'm not as angry as I used to be,' he once told a reporter. 'But I can get in touch with that anger pretty quickly if I feel my space is being invaded or somebody is not treating me with the respect that I think I want.'

These days, though, people tend to treat Jackson with respect. He commands a reputed $10m a movie, and with almost 80 films to his name, was recently named the highest-grossing actor of all time in the Guinness Book of Records, displacing Harrison Ford in the process. When you consider that he was a late starter, crossing over into the mainstream only after Pulp Fiction, which was released in 1994 when he was 45, the figures are even more phenomenal. 'I like to work,' he says, 'and when I'm not working, I like to play golf.' He is in town for a celebrity tournament and once claimed that he has a clause inserted in every contract allowing him to play regular rounds of golf during breaks in filming.

Since becoming clean and serene in 1991, the year he famously played a crack addict in Spike Lee's Jungle Fever, Jackson has been nothing if not prolific. He has starred in a string of big box office hits including Die Hard: With a Vengeance, Shaft and Star Wars: The Phantom Menace, where he insisted on having a purple light sabre. What, one wonders, would Freud have said about that?

There have been countless reasonably good films in which he shone - Coach Carter, A Time to Kill - and one or two turkeys, too, including SWAT, a formulaic action pic, and the frenetic 51st State, in which Jackson managed to maintain a consistent state of hyperactivity while playing a drug dealer in a kilt. Better by far are the character roles that he seems to relax into: Ordell Robbie, the affable arms dealer in Tarantino's underrated Jackie Brown; Doyle Gipson, the desperate and vengeful insurance man in Changing Lanes.

'I tend to play characters that I can infuse with certain kinds of humour,' he says. 'Even the baddest guy can be funny in his own particular way. I want the audience to engage with the character on some deeper level so that they leave the cinema still thinking about him. Those are the kinds of characters I empathised with when I was a movie-mad kid.'

In the next month Jackson stars in two new films that attest to both his range and his willingness to go out on a limb. In Frank Miller's much-anticipated take on The Spirit, Will Eisner's legendary comic strip, Jackson plays an evil mastermind known as the Octopus. 'It's a cartoon brought to life,' he says. 'The violence is straight out of Bugs Bunny and Wile E Coyote. At one point I hit the Spirit with a 12ft wrench and you can almost see the birds whistling and flying around his head.'

In the original comic version, the Octopus was simply a pair of gloved hands. In Miller's version he's a mad scientist on a mission to remake the whole world in his evil image. 'I'm looking for the Blood of Heracles, which will make me totally indestructible,' elaborates Jackson, gleefully. 'Basically, Frank [Miller] gave me the licence to be as maniacal and larger-than-life as I wanted to be. I mean, who could refuse?'

In Lakeview Terrace, out this weekend, Jackson plays another kind of maniac - one who operates under the veneer of law and order. The film is a tense racial drama that never quite lives up to the promise of its audacious premise. Directed by Neil LaBute, the playwright-turned-filmmaker whose peculiar talent is the detailed delineation of misanthropy, it explores the racial insecurities of a middle-class America by making Jackson's troubled cop, Abel Turner, a right-wing, bullying racist who wages a vindictive war of attrition on the mixed-race yuppie couple who move in next door to him. It is Jackson's darkest role to date, and until the film loses its way in the final half-hour, his most nuanced and powerful. How did he feel playing a racist?

'Well, the first thing is that Abel is more than that,' he replies, fixing me with a withering look that suggests I have misread the film's subtext. 'It's not crazy, blatant racism. He's an intensely troubled man who has a set of rules that he lives by and expects everyone else to live by. That's his strength and his weakness. Right off, I knew he had to have a sense of humour so that you can relate to him on some level even though he is a detestable kind of guy. There's a personality in there that can be disarming just with a smile, but then before you know it, he's said or done something that makes you want to go have a shower. He's a tricky individual.'

In Lakeview Terrace, Jackson's Abel Turner looks all of 70 years old, a man whose tense and troubled features convey not just his current state of mind but all the tribulations and injustices that have formed and eventually warped him. In one pivotal bar-room scene, where he attempts a rapprochement with his younger, white neighbour Chris (Patrick Wilson), he moves from contrite to reflective to volcanically angry. It is a study in subtle character acting of the highest degree, but there are moments, too, throughout the film where that kind of subtlety gives way to the broadest of brush strokes: the familiar raised eyebrow, that sardonic grin, those badass soliloquies that helped make Jules in Pulp Fiction such an iconic character but have become shorthand for a certain kind of swaggering, street-smart machismo ever since.

For all that, Jackson is one of the most commanding presences in contemporary film. When he steps up to the plate, there is little room on the big screen for anyone else. Sometimes, as is the case with the denouement of Lakeview Terrace - the point where it stops being an edgy racial drama and becomes a ham-fisted revenge thriller - you can even sense his frustration with the conventions of his trade.

'Well, to be truthful, there is a point where that film breaks down,' he says, shaking his head ruefully, 'and that's because there is a studio mechanism and a set of rules that say the audience has to be satisfied in a certain way. The last 20 minutes are problematic because they insist on some kind of closure that, most times, life does not give you. I have a problem with that, but I got to work within those conventions as best I can.'

Jackson, one senses, is an outsider who has made himself fit in. He was born in Washington DC in 1949, and grew up an only child in Chattanooga, Tennessee, looked after by his grandparents after his father fled familial responsibility soon after his birth. His mother continued to live in Washington until he was nine, visiting him at Christmas and in the summer holidays. He once said that he 'learned not to cry' when she left and that as he grew older his 'separation anxiety became easier'.

That sense of utter self-containment endures. His wife, the actress LaTanya Richardson, with whom he has a grown-up daughter, Zoe, once described him as 'emotionally disconnected'. Often, she said, when she would ring him on set to ask if he was missing her, he would simply say no. He likes, he says, to 'keep busy' and 'not brood on things'.

Jackson is old enough to have experienced segregation first hand in the Deep South, though he has said more than once that he was not scarred by it. His grandparents were strict to the point of being authoritarian. 'They knew where I was every hour of the day,' he told an interviewer, 'and I was very afraid of their retribution.' They also instilled in him the rigorous work ethic that has made him the actor he is today, telling him constantly that he had to work 10 times harder to survive in a world run by white men. Acting, you feel, was his salvation, and his escape: 'I never wanted to do anything else except give people that feeling of satisfaction I felt as a kid when I watched a great actor in a great picture.'

Jackson attended a segregated school in Chattanooga, where he was a model student, playing the French horn in the orchestra. As a teenager attending the prestigious Morehouse College in Atlanta, Georgia, though, he seems to have suddenly lost his way, ditching his family's traditional values for the cause of black radicalism. He was suspended after leading a protest demanding a black studies course. It culminated in a standoff with the local police after several faculty members were taken hostage, including Martin Luther King's father. Jackson was suspended for two years and convicted on a charge of unlawful confinement. Soon afterwards he befriended Stokely Carmichael, the Black Power leader, and was monitored by the FBI, which informed his mother that he would most likely be killed if he continued to embrace radical black politics. That onscreen swagger is not all front, then. I ask him if he has any regrets about how he lived his young life.

'Well, I made a few movies I shouldn't have,' he deadpans, before cracking up with laughter. 'But. You know, so what? As for the other stuff? It was all part of growing up in an extraordinary time. That shit helps you mature if you come through it and I'm here, right? Even the drugs and shit - I came through. I turned myself around.'

Last week, the news broke that Jackson has been recruited by Prince Charles's charity, the Prince's Trust, as a kind of universal mentor for young people with ambitions. You can see why he fits the bill. This guy believes.

'The best advice that was given to me was that I had to be 10 times smarter, braver and more polite to be equal,' he said in his first statement for the charity. 'So I did.' Like everything he does, he makes it sound easy. His life, though, pre-fame, was anything but.

Jackson's long dalliance with drugs and alcohol began soon after he graduated from Morehouse with a degree in drama. He moved to New York City in 1976 and began working as a relatively successful stage actor. 'I paid my dues,' he says, 'and that apprenticeship was invaluable to me. From the start of my movie career, I never looked at how Brando did this or Gregory Peck did that. Instead I always looked at what I learned in the theatre. What is the character? What does he do? And if there is not enough information in the script, you create a biography for him. When and where was he born? What did his parents do? Was he in the military? Has he been to college? On and on, slowly building a character. The lessons I learned back then I held on to even in the drug fog I was in for years. And I still ask all those questions of every character, whether it be for Star Wars or Shakespeare.'

By 1990, though, that drug fog had all but enveloped him. He was working as an understudy to a stage actor called Charles Dutton in a Broadway play called The Piano Lesson and found himself doing more and more crack cocaine just to 'drown out' Dutton's voice, which was echoing in his head even in his sleep. When Dutton was nominated for a Tony award for his performance, Jackson went on an almighty binge that culminated in his wife discovering him unconscious on their kitchen floor. I read somewhere that he had to be dragged into rehab 'kicking and screaming'. In most cases, that means the cure doesn't take.

'Yeah, all the experts say that you've got to surrender,' he says, nodding, 'but fortunately I was tired. I had reached that place where I could honestly say to myself: I've tried everything else, may as well give this a shot. The irony is I never got to taste Cristal. Back then, Moët was my champagne of choice. Now I get sent crates of Cristal and I ain't never tasted the stuff. Ain't that a bitch?'

Joking aside, does he ever feel like having a drink, maybe raising a glass of champagne or two to his own extraordinary success? 'Hell yeah, there are days when I feel like that, but I don't do it. I ain't the kind of guy who can have one drink. I never could. That's what I have to remember. I never had one drink in my whole life. When I bought a six-pack, I didn't drink a couple of beers and put the rest in the fridge for later in the week. I drank the lot, then went out and bought another one. I was compulsive.'

Does he think that, in some way, that compulsion now drives his work? 'Yeah, I guess so. I read six to eight scripts a week. I want to get up every day and act. The work defines me now.'

That has been the case since Jungle Fever, the Spike Lee-directed film that first signalled the depth and breadth of Jackson's then burgeoning talent. In it, he played a raddled crack addict. The shoot began a week and a half after his stint in rehab ended, perhaps the most vulnerable time for a recovering addict. Did it ever cross his mind not to accept the role? 'No. Soon as I read the script, I wanted to do it.

Plus by then I was angry at the rehab people. They kept on saying: "Don't do it, man - you're going to have pipes in your hand, lighters, all these triggers. You're going to pick up again and be right back there." Man, they did not want me to do that movie. In the end, I was like: "I will not pick up again for no other reason than I do not ever want to see you in my life again."'

He creases up, then turns suddenly serious. 'When I'm in New York, I still go to meetings. I go right up to Harlem to that same place I got clean. I can walk in the door and see the same old faces, and they're glad to see me. It still feels good not to be one of those eight-days-clean stories.'

One of the most refreshing things about Jackson's fall from grace and his amazing recovery is that he doesn't make a big deal about it. He doesn't even seem that regretful. 'That's true. See, I had a real good time for a long time, then I had a bad patch at the end. And even that got me to where I am now. So what's to regret?'

Inevitably, the talk turns to Barack Obama, who, when this interview took place, had yet to be elected. Can he, I asked Jackson, carry the enormous weight of expectation? Is he experienced enough to steer America though the biggest crisis since the Great Depression? 'Hell yeah!' he shouts, leaning forward in his seat, palpably excited. 'Now we got that movie stuff out of the way, let's talk about some serious shit. The way I see it, this is a pivotal moment in America's development not just in terms of electing a black president but in terms of how we want to be perceived by the rest of the world. Do we want to keep on pretending we're a superpower when we're not? Or do we want to join the rest of the world in a new way of working shit out? Barack is the only person who can take us down the second route. He's the only guy who can rewrite the future.'

Before I can even agree with him, Jackson is off again on another monologue, excited now by his own rhetoric as much as anything. You can see how he steals the show when he's engaged with articulating something he believes in. 'Here's the deal,' he says, breaking into that Jules grin. 'Being president is like being a movie star. Young actors watch a movie star and say: "I want to be a movie star so bad." Young senators watch the president and say: "I want to be the president so bad." But when you get there, it's bigger than the dream. The responsibilities, the interviews, the choices you have to make, the free shit that comes with it. You got to be on your game 24/7. You learn stuff. Clinton didn't know the job, but he learned fast. The fool we got [now] never learned it. He still looks like he's reading his lines. Barack don't ever sound like he's just reading his lines.'

Jackson is just warming to his topic when the PR sticks her head around the door and announces our time is up. He looks crestfallen. 'Man, we were just getting down to it,' he says, shaking his head, 'but I got to go and do the Punch and Judy show.' I'm about to tell him it's the Richard and Judy Show but I figure, what the hell, he'll find out soon enough.

Before he departs, I ask him as diplomatically as I can if, now that he's hitting 60, he is considering redefining himself once more, maybe easing up on the action pics for roles more suited to his age. 'I hear what you're saying. I'm settling with that,' he says, nodding. 'I know what I was thinking when I was sitting watching Indiana Jones last year. I love Harrison, and I love watching him do what he does, but at a certain point it just doesn't look great.'

He grimaces, then breaks into that familiar mischievous grin. 'I like to think I have more respect for my audience than that.' He pauses, then adds: 'And for myself.'

• Lakeview Terrace is out now. Will Eisner's The Spirit is out on 2 January