Chris Rock arrives at the Guardian US office in New York looking more like an intern turning up for work experience than the man widely agreed to be one of the greatest standup comedians of all time. This is partly because, even though he has been famous since the early 90s, he carries none of the usual baggage of an American celebrity. He has come from his home in New Jersey on his own, and with no “Look at me, I’m famous!” attitude. But it is also partly because, despite turning 50, he has the slight build of a teenager.

“Still skinny. I work out now, because you have to when you’re single,” he says, making a gentle reference to his recent divorce from his wife of almost 20 years, Malaak Compton. “But I’m still skinny.”

While the photographer takes his picture, he takes on the performative role so natural to a standup, crowing about the journalists working around us: “Someone needs to tell these guys print is dead. Dead!” Once it’s just the two of us he becomes quiet with the softly spoken manners of a naturally private man, and he twists in his seat to look out of the window when he speaks. After 10 minutes or so, he feels comfortable enough to maintain eye contact and ask me questions about my life. A mutual acquaintance had warned me that Rock is “very serious”, but I don’t think that is quite the word. He is thoughtful, and if that is not a quality some might expect of the man who has too often played Adam Sandler’s sidekick in movies, well, Rock himself has recently realised that he is not quite the artist he always thought he was either.

For a man who obsessively crafts his comedy, he seemed to choose his films willy-nilly

There has long been a jarring divide in Rock’s work between his comedy and his film career. While his standup specials, from 1996’s Bring the Pain to 2008’s Kill the Messenger, are so dense with ideas, his movies – What To Expect When You’re Expecting, Grown Ups – have often felt downright disposable. For a man who obsessively crafts his comedy, he seemed to choose his films willy-nilly.

“I’m not sure if I chose them willy‑nilly – well, I guess I chose a couple of them willy-nilly,” he concedes.

Which ones?

“Um, Beverly Hills Ninja. Some were willy-nilly, some were [Adam] Sandler called you up and it’s like, ‘Yeah, OK.’”

His latest film, Top Five, which he wrote, directed and stars in, is a revelation to anyone who worried that Rock was disappearing down the same disheartening path as Martin Lawrence, Bernie Mac and too many other talented black comedians who ended up in schlock because they did not get the roles they deserved. Rock plays Andre Allen, a wildly famous comic actor who is desperate to be taken seriously but whose fans want him to make more funny franchises. Worse, he is becoming most famous for his upcoming wedding to a reality TV star, played by Gabrielle Union, and the film follows Andre over the course of a day spent with Chelsea Brown (Rosario Dawson), a journalist from the New York Times. By not aiming for mass laughs, Rock has made his most critically acclaimed film.

“You get to a point where you say, OK, I’m not going to be Iron Man. So maybe I should be hanging with [Richard] Linklater and [Alexander]Payne. Maybe this is my crew instead of Sandler and Apatow and Stiller. They’re all still my boys, but maybe, artistically, my crew is over here,” he says.

Three things, he says, brought him to this point. The first was his 2011 performance in the Broadway play, The Motherfucker in the Hat. I saw one of the earlier shows and already Rock seemed calm, confident and completely immersed in his role. The experience, he says, taught him how to respect acting in a new way.

Most comedies aren’t really movies – they’re just vehicles for the funny person that’s starring in them. Chris Rock

The next was being cast in Julie Delpy’s lovely 2012 film, 2 Days in New York, which reminded Rock of his movie-making hero Woody Allen’s approach to comedy: “Most comedies aren’t really movies – they’re just vehicles for the funny person that’s starring in them. No one cares where the story’s going, and if it doesn’t work, they’ll just throw in another set piece. But with 2 Days, the jokes come out of the drama. Woody doesn’t make comedies – he makes sad dramas with jokes.”

Given the positive reaction to the play and 2 Days, Rock thought: “I’m gonna do one for me now.” And so the third piece of the puzzle turned up in the form of Scott Rudin, the notoriously temperamental uber-producer, who was recently in the news when the hack into the Sony server exposed emails between him and the then chair of Sony Pictures, Amy Pascal. The two had joked that President Obama only liked to watch specifically “black” films such as 12 Years a Slave and The Butler.

“OK, Rudin’s basically fucking crazy,” Rock says, “but he had my back the whole time so I was able to go into a bubble and write the movie I wanted without dealing with anyone else. Scott Rudin’s not racist. Scott Rudin hates EVERYBODY.”

He cites Allen, Linklater, Payne, Noah Baumbach and Wes Anderson as among his favourite filmmakers, but in almost none of their films do you see black actors.

“Yeah, you kinda don’t. It’s a really weird thing in the ones that take place in New York. But I still like those movies – I’m cursed with taste,” he shrugs.

Not that things are any better in more mainstream films. As Rudin pointed out to him, in the Sandler and Stiller movies Rock is always “the friend of the main guy”, not the main guy. “And I’d rather be the guy,” Rock says. “I don’t know – those guys [Apatow and co] do what they do, the movies are funny and they are black, they just don’t have black people in them. Every joke has somebody saying some black slang or some reference to Wu Tang. But it is what it is.”

Last December Rock wrote in the Hollywood Reporter about what it is like to work in Hollywood – “a white industry, just as the NBA is a black industry” – where black actors are not even considered for general roles. “It was never like, ‘Is it going to be Ryan Gosling or Chiwetel Ejiofor for Fifty Shades of Grey?’” he wrote. If he’d tried to make Top Five at a Hollywood studio, “I would have had to have three two-hour meetings explaining that black people also read the New York Times.”

Has he ever suffered any kickback from his frankness about Hollywood?

“I don’t know. I mean, I don’t get cast that much. Most of the movies I’ve starred in I wrote,” he says. Along with Rudin, Jay Z and Kanye West co-produced Top Five. While the movie is kindhearted, it does poke fun at Andre’s reality-TV-star fiancée, Erica, who believes that “it’s not real if it doesn’t happen on camera”. Did West (whose own wife, Kim Kardashian, might be considered to have a similar mindset) raise any objections?

Watch a video review of Top Five Guardian

Rock makes a sly grin when he sees where the question is going.

“Aww man … You know, I haven’t talked to him for a while. I should text him, go for lunch …” he mutters, deflecting.

But you are making fun of the whole reality TV wedding scene, I say.

“Well …” he makes a “you caught me” laugh. “But it works because we’re not making fun of it. We treat her like a person, you know?” Which is not entirely true, given that Erica at one point screams at Andre that he has to go through with the wedding because she has always provided him with blow jobs on tap.

So much of Rock’s standup and films have been about how hard marriage can be. “Married and bored or single and lonely,” went one popular riff. Did his honesty ever cause him problems at home?

“Hey, I’m getting divorced now,” he says with an easy laugh.

When news broke in December about Rock’s divorce, it felt oddly dispiriting: sure, he warned us all how tough marriage is, but it was hard not to hope that someone so attuned to its pitfalls might navigate them. How’s he been doing?

“I’m doing OK. You know, some days are better than others, some days you’re sad outta your fucking mind. But my daughters are good and I’m only an hour away. Two houses close by. It’s good,” he says, a little quieter than before.

Will he still talk about personal relationships now that his own situation has changed?

“Oh, totally! You learn more from failure than success, right?”

I talk about women the way I talk about men. Everyone’s fair game and no one gets a pass Chris Rock

His fumbling steps back into the dating game could provide him with material. While he was being photographed he joked to the room, “I’ll be like, ‘What do you think of me?’ and she’ll be like, ‘I thought we were just going out for Mexican!’”

It is sometimes hard to know where you stand over a taco, I say.

“You do have to be careful when you make that turn [into making a declaration,]” he laughs. “It feels like being the girl.”

Top Five features the best female character Rock has ever created in the form of Chelsea, the journalist. But he has been criticised in the past for the way he characterises women in his standup – demanding, high–maintenance – and considering that he has always taken pride in avoiding cliches in his jokes, this has often struck me as oddly lazy.

“I talk about women the way I talk about men,” he says, and it is true he does stylise men in his standup as sex-obsessed dopes. “Everyone’s fair game and no one gets a pass. I talk about black people the way I talk about white people.”

That’s just my life. I’ve always navigated those things. I watched the fight the other night with Gab Union and [comedian] Hannibal Buress and tonight I’m having dinner with [entertainment executive] Tom Freston and [George W Bush’s daughter] Barbara Pierce Bush. I dance around these worlds,” he says.

Rock grew up in a working-class family in Brooklyn and was bullied badly at school. He later mined his childhood for material in the very sweet and funny 2005-2009 sitcom Everybody Hates Chris. After dropping out of high school at 16, he worked briefly flipping burgers before moving to standup. He soon caught the eye of still the biggest black comedian of all time, Eddie Murphy. Rock’s character in Top Five partly seems to be a homage to his former mentor, who, like Andre, decided in the 90s that he wanted to be taken seriously – with disastrous results, as anyone who has seen Beverly Hills Cop 3 knows.

Kanye West, pictured with Rock, is a co-producer of Top Five. Photograph: Jim Spellman/WireImage

Rock learned from Murphy’s mistakes. In Murphy’s standup specials – Delirious (1983) and Raw (1987) – you see the comedian backstage surrounded by yes men and entourages. In Rock’s specials, he is always notably alone. And where Murphy briefly decided to give up comedy, Rock – as is clear from Top Five – is on the fans’ side in wanting comedians to stay funny, “because I don’t see drama as more substantial than comedy. Look at Murphy – he was better in Bowfinger than he was in Dreamgirls.”

So did Murphy get tired of people’s expectations on him?

“You know, he was the funniest thing I ever saw, the superstar comedian,” Rock says, a little wistfully. “I don’t know. But he’s still relatively young …”

Are they still in touch?

“I see him when I go to LA, I go by his house, we watch the fights. But that thing happens: you get kids, you’re on different coasts, so I don’t really talk to him that much.”

Rock’s other comedy mentor in the early days was Bill Cosby.

“I haven’t talked to him in a long time. The whole thing is just sad. What can you say? I’m not gonna defend him and I’m not go Judd on him,” he says, referring to Apatow’s criticisms of Cosby. “You do still have to wait, he hasn’t been convicted. But it’s sad.”

These days, it is Rock who is the mentor. He cites Hannibal Buress and Kevin Hart (“My older daughter always tells me Kevin’s funnier than me”) as his favourite protegees. He is also about to direct a standup special for Amy Schumer, who he describes as “the real thing”. So, what did he think of The Daily Show hiring the relatively inexperienced South African comedian Trevor Noah?

Rock is uncharacteristically hesitant: “I … I … I think they hired the right guy, but it’s always nice to have done other TV stuff on your way in for an easier transition. But guys today are, like, just thrown into the fire. I don’t even know what the fucking joke was,” he says, referring to the criticism Noah got for several tweeted jokes.

There was a joke about knocking Jewish kids down in a German car, I tell him.

Recently, Rock has started documenting on social media each time he has been stopped by the police while driving.

Rock looks a little taken aback, but he soon rallies to the young comedian’s defence. “OK, um, BAD. Um, but you know, every Michael Jackson joke is about fucking kids. Where’s the outrage? It’s so selective. People let things go by all the time.”

Recently, Rock has started documenting on social media each time he has been stopped by the police while driving. “I’ve always been stopped by the cops. Cops stop black guys who drive nice cars,” he says simply.

American police brutality against young black men is in the news now more than ever. Has it become worse?

“It’s not that it’s gotten worse, it’s just that it’s part of the 24-hour news cycle. What’s weird is that it never happens to white kids. There’s no evidence that white youngsters are any less belligerent, you know? We can go to any Wall Street bar and they are way bigger assholes than in any other black bar. But will I see cops stop shooting black kids in my lifetime? Probably not,” he says.

Was he surprised that the six police officers in Baltimore have been charged with the manslaughter and murder of Freddie Gray?

“I am kinda surprised, and you know, unfortunately it may have something to do with the black mayor and the black police chief and all that stuff. But, hey, charged and convicted are different – so we’ll see.”

Does it make him glad he has two daughters, aged 10 and 12, instead of sons?

“It does, yeah. It really does. Girls have their own things to worry about, but [getting beaten up by the police] is just not a thing that generally happens to girls.”

How does he think growing up with a black family in the White House has affected his daughters’ outlook on the world?

“It’s not really my kids who are affected – it’s white people’s kids. It’s white people who have made progress. To call it black progress suggests we deserved everything that happened to us: the kids my kids grow up with won’t have a hard time picturing my daughters in an executive capacity – that’s progress, you know what I mean?”

And is he still an Obama supporter?

“Oh yeah – he’s been good. Great, even. He wasn’t going to solve America, but the country was off the rails and he was like Alec Baldwin in Glengarry Glen Ross, you know? He really sorted shit out.”

Rock feels good about the future. He is working on another film script and wants to make more documentaries like his excellent 2009 film about the black hair industry, Good Hair. He would love to do more standup, but now that he looks after his daughters on his own for part of the week, it is harder to get away to tour, “so I’m becoming like the Sade of comedy”, he says. Sure, he would like Woody to call, or Marvel. But he has been around long enough now to know that the only person who can – or will – give him the career he wants is himself.

“The only question now is can I keep it going and raise healthy daughters who don’t have daddy issues because I’m gone all the time? So who the hell knows what’s next?” And he laughs again, because the one person who knows is Chris Rock.

• Top Five is released in the UK on 8 May