WASHINGTON — A senior American diplomat met with his Iranian counterpart in Vienna on Monday to explore whether the United States and Iran could work together to create a more stable Iraqi government and ease the threat from Sunni militants.

The initial meeting took place after Secretary of State John Kerry signaled that the Obama administration was open to cooperating with Iran on Iraq, raising the possibility of seeking help from a country that the United States has often described as a state sponsor of terrorism that must be prevented from obtaining a nuclear weapon.

The Obama administration’s strategy is to pressure Iraq’s prime minister, Nuri Kamal al-Maliki, and his Shiite-dominated government to form a multisectarian government with Sunnis and Kurds in an effort to heal the rifts being exploited by the insurgents. But that goal could be frustrated if Iran decided to back hard-line Shiite leaders or sent Quds Force fighters into Iraq, aggravating the already inflamed tensions.