Idris Elba opened up about feeling disappointed after people reacted negatively to him being cast as James Bond because of his race

The Fast and Furious Presents: Hobbs & Shaw actor appears on the latest cover of Vanity Fair where he got real about the disappointment he felt after people were opposed to him stepping into the famous secret agent’s shoes. The rumors first started in 2014 when an email leaked from Sony Pictures senior executive Amy Pascal revealed her feeling that “Idris should be the next Bond.”

Get push notifications with news, features and more.

“You just get disheartened when you get people from a generational point of view going, ‘It can’t be.’ And it really turns out to be the color of my skin,” Elba, 46, said in response to the backlash. “And then if I get it and it didn’t work, or it did work, would it be because of the color of my skin? That’s a difficult position to put myself into when I don’t need to.”

Still, Elba admitted that it’s an iconic role he’d want the chance to play if it was offered to him, but he wouldn’t go looking for it.

“James Bond is a hugely coveted, iconic, beloved character, that takes audiences on this massive escapism journey,” he said. “Of course, if someone said to me, ‘Do you want to play James Bond?,’ I’d be like, ‘Yeah!’ That’s fascinating to me. But it’s not something I’ve expressed, like, ‘Yeah, I wanna be the black James Bond.’ Because, by the way, we’re talking about a spy. If you really want to break it down, the more less-obvious it is, the better.”

Image zoom Idris Elba Stuart C. Wilson/Getty Images

PEOPLE’s reigning Sexiest Man Alive previously opened up in a new interview with the Daily Mail‘s Event about the consistent rumors that he’s taking over for Daniel Craig in the franchise. Craig is set to retire from the role after he wraps up the next installment.