TEHRAN—Japanese Prime Minister Shinzo Abe had a rough debut as Middle East peacemaker, ending the first trip by a Japanese leader to Iran in 41 years with the U.S.-Iran conflict even more volatile than before.

Mr. Abe went to Tehran on Wednesday hoping to serve as a bridge between the U.S., Japan’s closest ally, and Iran, with which Tokyo has maintained warm ties.

By...