BERN, Switzerland—Hours after a U.S. strike killed Iranian Maj. Gen. Qassem Soleimani, the Trump administration sent an urgent back channel message to Tehran: Don’t escalate.

The encrypted fax was sent via the Swiss Embassy in Iran, one of the few means of direct, confidential communication between the two sides, U.S. officials said.

In the...