The Pentagon rescinded an invitation to China to participate in an international military exercise in the Pacific Ocean next month, signaling disapproval to Beijing for what U.S. officials say is its refusal to stop militarizing South China Sea islands.

Defense Secretary Jim Mattis, after weeks of internal debate within the Pentagon, concluded that China shouldn’t be allowed to participate in the American-led biennial Rim of the Pacific exercise, slated to begin in June, according to U.S. officials. The invitation’s withdrawal...