SINGAPORE—U.S. Defense Secretary Jim Mattis on Saturday urged China to help the international community to isolate North Korea and persuade it to halt its ballistic missile and nuclear programs, but didn’t spare criticism of China for its own behavior, in the South China Sea.

North Korea’s ballistic missile and nuclear programs must be stopped and the era of “strategic patience,” a buzzword for the approach taken during the Obama years, is over, Mr. Mattis said during a speech at the Shangri-La Dialogue, a regional security...