HONG KONG—Alibaba Group Holding Ltd.'s planned listing in New York is a blow to Hong Kong's stock exchange, which failed in an effort to change its rules so as to accommodate what could be one of the world's biggest initial public offerings.

The exchange continues to capture the lion's share of listings by mainland Chinese companies—and has several big IPOs in its pipeline—but as it focuses on its commodities and derivatives business, it has missed out on a wave of Chinese tech companies. Instead, they've listed in the U.S....