As a general rule, Mr. Ross said, he viewed such proposals “not with a grain of salt, but a small ton of salt.” But he had gotten to know the Omanis through his work on Middle East peace issues in the 1990s, and he knew their ties to the Iranians were genuine. He said he decided to pass along Mr. Ismaily’s proposal, with a caveat-laden cover memo, to Mrs. Clinton. She told Mr. Ross to keep talking to him.

A few weeks later, the Iranian authorities cracked down brutally on antigovernment protesters, dashing Mr. Ismaily’s hopes to set up a channel and prompting the White House to shift from a strategy of engaging Tehran to one of pressuring it. Mrs. Clinton lobbied China and other countries in the United Nations Security Council to impose harsh new sanctions on Iran, a step widely seen as a crucial lever against the Iranians.

Around that time, Mr. Ismaily got another chance to demonstrate his skills as an intermediary. He negotiated the release of three young Americans who had been arrested by Iranian guards while hiking on the border between Iran and Iraqi Kurdistan. Oman paid bail for the three hikers, roughly $500,000 each.

In December 2010, several weeks after the release of the first American, Mr. Ross and a senior official on the National Security Council, Puneet Talwar, secretly traveled to Oman to hear from Sultan Qaboos himself how he thought a channel could work. They were impressed by what the sultan told them: He had visited the supreme leader of Iran, Ayatollah Ali Khamenei, and was confident of the country’s seriousness in seeking a nuclear deal.

The next January, Mrs. Clinton stopped in Muscat, the capital of Oman, for her own briefing. She expressed doubts that the Iranians could negotiate in good faith, but she agreed to put it to a test. Mr. Obama was more intrigued: He called Sultan Qaboos twice over the next few months to ask him about whether he could deliver Iranians who could speak with the authority of the supreme leader. The White House, intent on secrecy, did not disclose the calls.

Leaning Forward

Mr. Kerry had long nourished the idea of opening lines of communication to Iran, and he saw his chance when he got involved in trying to free the hikers. That put him in contact with the sultan and his emissary. (Mr. Ismaily confirmed this account, but declined to speak on the record about his role in the nuclear talks.)