This November’s SPECTRE will mark Daniel Craig’s fourth and possibly final outing as James Bond. (Though he didn’t totally close the door, he recently told Esquire, “I have a life and I’ve got to get on with it a bit.”) The potential need for a new 007 has kicked off the speculation mill about who might suit up next as the British super spy. Early (and contentious) online conversations have focused on Idris Elba and Tom Hardy, but as history (and the video above) shows, a whole lot has to go right for an actor to officially start ordering his Martinis shaken, not stirred.

Related: James Bond’s 12 Goofiest Gadgets, From Lethal Polaroids to Killer Umbrellas

For all the marquee names who have played Bond, there are many even bigger names that just missed out. Before Sean Connery launched the franchise on the big screen with 1962′s Dr. No, Richard Burton was considered for the first-ever 007. He wasn’t convinced that the character or movie would be anything special, according to family members, and wanted more money to take the part. Cary Grant was also asked to play the first Bond, but at 58, he felt he was too old, and would only commit to one film. Again, that was a no-go.

Other potential Bonds through the years included Michael Caine, who passed because didn’t want to be typecast after playing several other spies; Dick Van Dyke, who couldn’t nail the accent (see: Mary Poppins); Clint Eastwood, who felt Bond should be British; and Mel Gibson, who was rejected by the studio for not being British.

Apparently, being super-famous often makes it harder to be a secret agent.