Forest Whitaker is having himself a British moment, flashing back more than 30 years to his first visit to London. "The first time I ever went out of the country it was to London. I was with the choir from my college and we were touring around all these different churches. I loved it so much I tried to find a way to stay there. I tried to get a job but I had no work permit. I tried anything I could to stay. My feeling then was, this is where I was meant to be. I felt … freedom. I've been back many, many times since, made a lot of friends – and I've played a few Brits, too, in The Crying Game and other movies."

This is all prompted by the desire to make a visiting Briton feel welcome (and it works), but his memory-ramble through names such as Stephen Woolley of Palace Pictures, director Neil Jordan, Kevin MacDonald of The Last King of Scotland, and an array of other British and Irish friends, reminds me that the connection is indeed pretty long-lasting: by 1992, 12 years after that first visit, Whitaker was playing a cricket-loving gay squaddie in Jordan's Oscar-winning The Crying Game. That'll cement your loyalty to a place, I guess.

Here in his office, not far from the entrance to the Universal City theme park, I can't help noticing the huge movie poster on the wall behind him, for Jim Jarmusch's Ghost Dog: The Way of the Samurai. His braided hair, huge samurai sword raised aloft, and that calm fury in his sleepy, gentle eyes. Whitaker today, sitting right in front of the poster, is much leaner than he was then, and he's dressed down in a white windbreaker and jeans. But Ghost Dog reminds me just how Zen he is: this is the man who ended his speech at the 50th anniversary of the March on Washington in August with the words: "May we remain connected in love. We are one."

And so it proves as we talk. The man is an optimist whenever he can be one, and figures anything good that hasn't happened yet is just something that will surely happen soon, with a little patience and pressure. Even when he's talking of the worst in human nature – racism, the suffering of child soldiers, police brutality – his voice remains even and light. Anger is bad for you.

"Things do change," he agrees. "One of the things I noticed when we embarked on this presidency was a distinct sense that we were rejoining other nations as a citizen of the world again. That's a positive step. Martin Luther King spoke about the promissory note of life, liberty and the pursuit of happiness, the very tenets this country was built on. They haven't been fully achieved yet, but they haven't been achieved in the rest of the world either. We're on an accelerated schedule, though; we've gotten things done in this country that have never been achieved elsewhere in a thousand years. There are always small, earlier steps being taken throughout history. I learned doing The Butler that the very first black person to work in the White House – not as a servant – was in one of the social departments heading up their publicity section, but President Kennedy had their kids tutored together with his own in the White House. Nobody remembers that now, but it was another way of moving the needle."

And he doesn't see it a merely as party-political matter. "You can't just direct the arrow in one way, though, or from one place politically," he insists. "You have to ask if the country would have been ready for Barack Obama if we hadn't been prepared by Colin Powell and Condoleezza Rice. If they hadn't already been in positions of enormous power and influence – secretary of state, secretary of defence, you know what I mean? You can't get any higher, except to be their boss, the president. There's all these contradictions, I mean, a president might be a terrible warmonger, but redeem himself by doing great work for Africa at the same time. That happened – it is happening – with George W Bush, who's seen very differently in Africa than he is here, because with his work on Aids/HIV and other issues, he has really made a difference. And all policy considerations aside, Condoleezza changes the framework, the perceptions, just by being who she is, where she is. All of this moves the needle."

See that? Zen.

Forest Whitaker as Idi Amin in The Last King of Scotland. Photograph: Fox Searchlight/Freud Communications

But the things change/stay the same dichotomy plays out in the contrast between his latest two projects. As Cecil Gaines, the lead in Lee Daniels' The Butler, he plays a White House butler who serves seven presidents over four decades, from Eisenhower to Reagan, and lives long enough to witness, aged 92, the election of Obama. The other movie, Fruitvale Station, which Whitaker helped to produce, is the true story of the ugly police-shooting of Oscar Grant, a young black man in Oakland, on New Year's Day, 2009. Another Emmett Till moment, the day before yesterday, almost.

The Butler – very loosely derived from the true story of White House butler Eugene Allen – is a sort of political fantasy on postwar African-American history, following two generations of a well-to-do black family (it also contains a revelatory performance by Oprah Winfrey as Cecil's alcoholic wife). On parallel narrative tracks, we follow Cecil as he serves a succession of presidents, glad that his job, however servile, has offered him an escape from the Georgia cotton fields where he grew up in the 1920s, witnessing his mother's rape and his father being shot for protesting. His son Louis (David Oyelowo) meanwhile becomes a rising force in the civil rights movement, launching lunch-counter protests against segregation, joining the Freedom Riders, even being in the Lorraine Hotel when MLK is shot. The Butler is structured around generational conflict in one black family over how to change the world, and how fast.

"It's not often explored on film, that divide between generations in the black community in the 60s, it's a different kind of generation gap. I play a sharecropper's son who comes of age in a very different world in the mid-20s, with Jim Crow, and actual slavery not so far in the past. Early on I incite my father to take a stand that ends up getting him killed, which has to govern the way Cecil feels about resistance. His anxiety about how to protect his own son comes directly from that. A guy born in a shack ends up working at the White House. So even as all these terrible things are happening in the the nation, his life is improving. So he thinks of his own son: "How do I protect him?" He needs to get an education, maybe become a doctor, a lawyer – a traditional kind of advancement for his son. The son wants big changes right now, no matter the cost to himself. But Cecil becomes an activist in his own small way, too, asking for equal pay for the black White House staff, and he finally wins out on that one. Small victories, big victories, they're all victories."

Forest Whitaker in Jim Jarmusch's Ghost Dog: the Way of the Samurai Photograph: Artisan/Everett/Rex Features

Fruitvale Station, starring Michael B Jordan (first seen as Wallace in The Wire) in a career-making role, arose out of Whitaker's constant search, since his Oscar sealed his influence as a Hollywood player, for young first-time film-makers to mentor. Often these movies arise out of experiences he has had through his charitable foundation, which has seen him spending time in Africa and in America's inner cities over the past decade. He has backed documentaries about child soldiers in Africa (Kassim the Great), municipal politics in a historically black city (Brick City), or Rising From Ashes, about the first Rwandan Olympic cycling team. On Fruitvale, he met the young writer-director Ryan Coogler, after seeing some of his shorts. Coogler had three ideas and Whitaker plumped for Fruitvale Station. He was anxious that Oscar Grant be more than a corpse on a station platform or a forgotten name in yesterday's paper.

"The point was to bring a human face to Oscar Grant. You get to see his loves and his losses, his flaws and his pains and aspirations – which are pretty closed off. And then he's just dead, meaninglessly dead. He had to be made into a human being again."

The young Whitaker wasn't altogether unlike the young Oscar – it's easy to see why he was drawn to the subject. He was born in Longview, Texas in 1961, and his family moved to Compton, in South LA, when he was four, just in time for the Watts riots of 1965. That uprising, born of years of housing discrimination and a police department acting like an occupying army, tore the heart out of the city. Growing up in those years in Compton brought Whitaker close to certain hidden realities of the age. "My memory of the Panthers, for instance, is very different from most people's because as a kid I remember them on my street, and how nice they treated me – free breakfast programme, free schools, outreach and community work, gang members working hand in hand, Crips and Bloods. They invited me to go get breakfast every day. Then there's the other great divide going on again because my mom's like: 'Oh, you're not going there, nuh-uh.'"

In the background was the dull roar of "Benign neglect" and the catastrophic effect it had on the black community: factories closing, the return of poverty and mass unemployment, guns and hard drugs flooding the streets, as gangs moved from community work back into crime. Whitaker says he stupidly involved himself in one gang encounter as a wannabe teenage peacemaker (he chuckles at the memory), which suddenly made the streets much more dangerous for him. His interventionist parents enrolled him at Pacific Palisades High, which meant he was bussed two hours every day to a well-funded, largely white, school overlooking the ocean. It changed his life, he says.

"I'd spend every summer in Longview, on my grandfather's farm. It was a tiny little town divided by a river, which was the segregation line, that side white, this side black. And meanwhile I lived in Compton – basically another whole world sealed into 10 square blocks. It's interesting how insular an environment can be. There's a lot of safety and positivity in being enfolded by a community. But at the same time, my world really only ran a few blocks in each direction. And remember, I was a big kid who spent a lot of time being thrown over police-car hoods and put on the ground in handcuffs because of how I looked, not for anything I did. Palisades High changed that whole way of seeing the world, and it changed me."

He flowered, he says. He played trumpet and trombone, sang in the choir and a few musical shows. In college he moved from music to theatre. "The acting bug probably came to me through music, because I was studying classical voice. I guess in preparation for opera, certain things became apparent about acting. My voice teacher asked me to audition for Under Milk Wood, and I got the lead role. And I thought, I like this a lot better."

And not long after graduating he was earning his first TV parts on Cagney and Lacey and Hill Street Blues. Now that he has an Oscar on the shelf, I can joke with him a little about his early role in Fast Times at Ridgemont High. He was, he agrees, stuck with a one-dimensional borderline-racist stereotype, a growling quarterback, although he does add: "Bear in mind, I was a big scary football player in college!" An eye-catching cameo in The Color of Money led to his first lead role, for Clint Eastwood no less, playing Charlie Parker in Bird, a magnificent chance for a young actor to make his mark. He's never worked with Spike Lee – "we know each other, we like each other, we've just never worked together" – but his career has unfolded on a parallel track to the same final destination. He has clout in white Hollywood, a career on his own terms, speaking softly all the while. He never looked back.

Forest Whitaker and Oprah Winfrey in The Butler. Photograph: Weinstein Company/Everett/REX

The years since have seen consolidation and growth, indelible performances in The Crying Game, Smoke, Ghost Dog, Panic Room, an Oscar for his role as Idi Amin in The Last King Of Scotland, but no shortage of middling and outright bad movies alongside the triumphs, both pre- and post-Oscar. You have to assume he made Battlefield: Earth for the Church of Scientology only for the money to fund his busy schedule as a producer and charitable donor (he later publicly regretted his involvement). In the meantime, he has been married to the same woman for 20 years, has four kids, and no whiff of scandal or celeb-nonsense attaches to him, ever. He has sat on the President's Committee on Arts and Humanities, which is possibly why he won't be drawn too far on topics such as drones and foreign wars; and his hands-on charity work includes involvement with North House, a kind of therapeutic school and village in Uganda for traumatised ex-child soldiers and refugees. He has seen wonderful things and he has seen dreadful things, and gives the impression of having transcended a parochial, race-based worldview. I can't help noticing how he almost never uses the adjective "black".

But optimism is his lodestar: "Of course I'm optimistic," he says. "Sure, we have stop-and-frisk, racism, sexism, sexual-preference issues. But the thing is, these things aren't purely American phenomena. In Sweden they'll be rioting against immigrants, in France they're stopping women wearing burqas. You've had riots in Brixton in the past. I was in the Sudan not long ago with tribes that are warring with each other – and they look exactly the same as each other and they all come from the same tribal group, the same language-source. It's a worldwide, human issue, not necessarily just racial.

"But look around, at the same time you see how many more actors are working, how many people are running big companies like American Express, it's clear that things are always getting better in one way or another, somewhere – it's going from living in chains once upon a time to producing the leader of the free world. That's a happy ending – enjoy it!"

The Butler is out now

More on The Butler

• Whitaker and Winfrey talk to Hadley Freeman

• The Guardian Film Show reviews the film

• Peter Bradshaw's review; Mark Kermode's

• Reel history rates the historical accuracy

• This article was amended on 20 November 2013. An earlier version implied that the Irish director Neil Jordan is British.