LOS ANGELES—Despite the star power of Leonardo DiCaprio and director Martin Scorsese, the 2013 hit movie “The Wolf of Wall Street” took more than six years to get made because studios weren’t willing to invest in a risky R-rated project.

Help arrived from a virtually unknown production company called Red Granite Pictures. Though it had made just one movie, Red Granite came up with the more than $100 million needed to film the sex- and drug-fueled story of a penny-stock swindler.

...