BEIJING—China’s response to a U.S. warship’s patrol in the South China Sea shows the balance it must strike between satisfying nationalism at home and projecting firm diplomacy abroad without escalating tensions.

In the days after the U.S. sail-past this week, Chinese officials have vehemently protested what they call an illegal move that “threatened China’s sovereignty and security interests,” and warned Washington against further “provocative actions.”

China’s...