Alibaba Group Holding Ltd. dominates the Chinese e-commerce market, but it increasingly is watching its back, thanks to a swarm of industry newcomers.

The clearest threat is from JD.com Inc., a longtime rival. Indeed, in China their rivalry has earned the nickname “the great cat and dog war”—referring to the respective mascots of Alibaba’s Tmall website and JD.com.

Alibaba’s...