BEIRUT—Tighter new U.S. sanctions have proved more punishing than Iran’s leaders expected, driving Tehran to hit back militarily and breach limits it had agreed to put on its nuclear program.

This increasingly confrontational approach aims to raise the costs to the U.S. of its maximum-pressure campaign and to push Western European nations to offer economic relief, according to former Iranian officials and analysts.

Iran’s...