BEIJING—North Korea’s fifth nuclear test brought calls for action from nations including Russia and Japan, but a deepening schism between the U.S. and China narrows the scope for cooperation on punishing Pyongyang.

Friday’s test shifted attention to what could be done about a nuclear program North Korea said is capable of producing nuclear-tipped missiles. South Korea’s president said she would seek stronger sanctions through the United Nations Security Council, and the top U.S. and Russian diplomats called for discussions about a course of action. Japanese Prime Minister Shinzo Abe conferred with U.S. President Barack Obama, urging that Pyongyang be made to “understand the costs” of its provocations.

Options the U.N. body could consider range from tightening sanctions imposed earlier this year to further cutting off North Korea from the international banking system, experts said. But whether the Security Council can act depends heavily on Beijing—North Korea’s ally and main trading partner—which has increasingly sparred with Washington over regional security issues, including tensions in the South China Sea and on the Korean Peninsula.

Central to Beijing’s growing distrust is a U.S.-built missile-defense system planned for deployment in South Korea, known as Terminal High-Altitude Area Defense, or Thaad. The U.S. says the system is designed to protect its ally from North Korean missiles, but Beijing says Thaad could also be used to track Chinese ones, undermining its nuclear deterrent.

For Beijing, Washington’s efforts against North Korea raise fears about U.S. encirclement. China, which fought alongside Pyongyang in the 1950-53 Korean War to push back the U.S., still sees North Korea as an important buffer. Beijing remains concerned that a North Korean collapse, politically or economically, would drive huge numbers of refugees across China’s borders and bring U.S. troops to its doorstep.