Idris Elba has been in the James Bond conversation for years, but the actor says he’s bothered by some of the racial references that pop up when he’s mentioned as the next 007.

In fact, the Luther star finds it troubling that some people don’t think he could play the suave super spy because he’s black.

“You just get disheartened when you get people from a generational point of view going, ‘It can’t be.’ And it really turns out to be the color of my skin,” Elba says in the August issue of Vanity Fair. “And then if I get it and it didn’t work, or it did work, would it be because of the color of my skin? That’s a difficult position to put myself into when I don’t need to.”

Elba tells the magazine he’s never expressed interest in the role, although he admits if approached, he’d definitely say yes.

“James Bond is a hugely coveted, iconic, beloved character, that takes audiences on this massive escapism journey,” he adds. “Of course, if someone said to me ‘Do you want to play James Bond?,’ I’d be like, Yeah! That’s fascinating to me. But it’s not something I’ve expressed, like, yeah, I wanna be the black James Bond.”