LONDON—Tehran said Tuesday it was canceling a $2.5 billion deal with China National Petroleum Corp. for the development of a giant Iranian oil field, as Beijing's role as alternative to Western companies faces mounting challenges.

China had stepped in after Western oil firms pulled out of Iran following a European ban on investing in the Islamic Republic in 2010.

But...