BEIJING—China took navy exercises to the farthest reaches of its claims in disputed waters, with four heavily armed ships coming within 50 miles of the coast of Malaysia, a country that has made relatively little noise about Beijing's recent assertiveness in the South China Sea.

China's official Xinhua news agency said the four ships that sailed to the James Shoal—a submerged reef in the South China Sea—on Tuesday included the Jinggangshan, China's largest amphibious landing ship. Experts say the ship is designed for launching...