BEIRUT—The threat of Sunni extremists eclipsing the power of its Shiite-dominated Arab ally presents Iran with the biggest security and strategic challenge it has faced since the U.S.-led invasion of Iraq in 2003.

With the Islamic State of Iraq and al-Sham, an offshoot of al Qaeda, rapidly gaining territory, Iran deployed Revolutionary Guards units to Iraq, according to Iranian security officials.

Iran...