TOKYO—Smartphone game makers working with Nintendo Co. are finding the home of Mario the plumber is putting up obstacles to scoring high revenue.

Since 2015, Nintendo has had revenue-sharing agreements for smartphone games that it creates with partners likeDeNA Co. The games are free to download but players can pay for enhancements to speed up game play or enter in-game lotteries to win special characters.

In...