Zalmay Khalilzad is a former U.S. ambassador to Afghanistan, Iraq and the United Nations. This piece is adapted from his book, The Envoy: From Kabul to the White House, My Journey Through a Turbulent World, just out from St. Martin’s Press.

As someone who has negotiated with Iran over the years perhaps more than any other U.S. diplomat, I disagree with those who argue that talks with Iran are akin to capitulation. I have seen little evidence that isolation has or will alter Tehran’s behavior in the right direction. Nor do I share the view that it is impossible to negotiate win-win deals with the Iranians.

Under the right conditions, which must include a hard-headed approach and tough actions to check Iran’s ambitions, Washington can benefit from bringing Iran into multilateral forums where the United States and its partners have the opportunity to narrow differences, create rules of the road and solve problems. Moreover, today we have little choice but to engage Iran on these broader issues, because no factor is shaping the order of the Middle East as much as the rivalry between Iran and its Sunni Arab neighbors, particularly Saudi Arabia and Turkey. This rivalry in turn is fanning sectarian conflict between Sunnis and Shiites across the Middle East, and increasingly, parts of Asia and Africa. And the wider chaos that has ensued continues to exact costs on our part of the world, as seen in terrorist attacks from San Bernardino to Brussels.


Gains by Iran’s Shiite proxies such as Hezbollah and the militias in Iraq and Yemen have led Saudi Arabia and Turkey to respond with expanded support for extremist Sunni groups. The heightened Shiite-Sunni conflict is undermining consensus and mutual acceptance among different communities in Iraq, Syria, Yemen and Bahrain.

The conflict between Iran and its neighbors is doubly worrisome because of what it portends for the state system of the Middle East. Underlying the motives of Shiite and Sunni sectarians alike is a desire to re-litigate the legitimacy of borders and lines of control in the region.

In multiethnic and multiconfessional states, sectarian groups are looking nostalgically to the legacies and traditions of pre-Westphalian empires. And as states fragment into ungoverned spaces, terrorist groups like the Islamic State are regrouping, waging attacks and creating massive humanitarian crises that are driving refugees around the world.

The United States has been slow and reactive in responding to these alarming trends in the region. Washington has sought to manage different crises in the Middle East without thinking more fundamentally about how U.S. diplomacy can catalyze a new, sustainable order for the region.

All of which brings us back to Iran. The Middle East cannot be stabilized without an agreement between Iran, Saudi Arabia and Turkey, not only on the region’s ongoing conflicts, but also on a sustainable architecture that can contain sectarian tensions. While its influence has declined in the region, the United States is the only power that can broker such an understanding. Just as the United States invested resources, energy and time in “normalizing” the geopolitics of Europe and East Asia during the Cold War, we must make a generational commitment to do so in the Middle East.

In large part, the passive U.S. approach on this front can be attributed to a lack of consensus in Washington on how to manage relations with Tehran, which would need to be a pillar in any new regional architecture. Given the chaos in the region today, I believe that we need to pursue a policy of hard-headed engagement—an expanded dialogue with Iran, in combination with measures to check Iran’s quest for hegemony, designed to bring Tehran onboard behind a new regional order that is consistent with U.S. interests.

But I’ve seen this kind of U.S. hesitancy predominate before, as ambassador to Iraq under George W. Bush. The Bush administration’s failure to engage Iran proved particularly problematic given that it never developed a broader strategy to bring Iran’s rivals behind a new architecture that could contain Iran’s hegemonic ambitions. As ambassador, I urged Saudi Arabia, Jordan, the United Arab Emirates and Turkey to support our Sunni Arab outreach efforts in Iraq and reassure Iraq’s Shiite leaders, which could have undercut Iran’s growing influence. My efforts yielded some success with Jordan and Turkey but largely failed to overcome the distrust between the Saudi royal family and Iraq’s Iran-backed Shiite parties. The crux of the issue was Riyadh’s broader distrust of Iran and its unrealistic demand that Washington, as King Abdullah told me, “cut the serpent’s head” and attack Iran—to weaken it and prevent Tehran from emerging as the rival leader of the Islamic world.

***

It took many decades to get to this place of bloody mistrust between the major Middle Eastern nations. When the British withdrew from the Middle East in the early 1970s, the region’s balance of power revolved around the competition between Iraq, which had strong military relations with the Soviets, and Saudi Arabia and Iran, which enjoyed supportive relations with the United States.

The Iranian Revolution removed a key pillar of regional stability. Iranian foreign policy suddenly focused on exporting its revolution to the Shiite populations of the Middle East—a strategy that resonated with long-oppressed Shiite populations in Sunni-dominated countries such as Saddam Hussein’s Iraq. Partly out of fear and partly out of opportunism, Iraq attacked Iran. Saudi Arabia, fearing that Khomeini could challenge its leadership in the Islamic world and stir up Shiite separatism in the eastern part of the country, supported Saddam.

Although the Iran-Iraq War did not end in a clear victory for either side, it did change the regional balance of power in Iraq’s favor. As a policy planner at the State Department, I made the then-controversial point that the U.S. priority should be to restore a regional balance of power. I assessed that without a balance, Iraq, now the dominant regional military power, would likely exploit its advantageous position to push for regional hegemony. I proposed that the United States should either tilt toward Iran or contain Iraq itself by building up its military power in the region. Secretary of State George Shultz rejected the proposal, and both the Reagan and George H.W. Bush administrations decided instead to try to manage the situation by engaging Iraq.

The United States missed the brief window between the end of the Iran-Iraq War and the Iraqi invasion of Kuwait to craft a sustainable balance between Iran, Iraq and other regional states. The enhanced U.S. military presence in the Persian Gulf after the Gulf War did help to deter direct interstate aggression by both Iraq and Iran but proved less effective in containing Iran’s growing reach. By creating and aiding militia-type forces such as Hezbollah, Iran was able to spread its revolution indirectly in Lebanon, Iraq and other Shia communities of the region.

U.S.-Iran relations entered a new period after 9/11. As a senior Bush administration official, I was deeply involved in the debate during this period on how to handle relations with Iran, particularly in the context of the wars in Afghanistan and Iraq. I supported the Bush administration’s decision to cooperate tactically with Iran in the immediate aftermath of 9/11 and believe, in hindsight, that it was the right move. I found that the United States had to deal with not one but multiple Iranian policies, some cooperative and some adversarial. The key to progress was to check Iran’s aggressive conduct while cooperating where possible.

After 9/11, the Iranians provided some support in the fight against Al Qaeda and facilitated contacts with a number of Northern Alliance commanders that they had supported throughout the 1990s. The most important gains from U.S.-Iran cooperation came at the 2001 Bonn Conference—the U.N. conference to form Afghanistan’s transitional government. The day before the start of the conference, I participated in talks with the Iranian delegation. Despite their long-standing ties to the Northern Alliance, the Iranians recognized that it would be best for the new government in Afghanistan, led by Hamid Karzai, to include other factions. Throughout the conference, my colleagues and I engaged in amicable and productive talks, which gave us important insights on what it would take to secure regional support for the post-Taliban government. Our engagement with the Iranians paid off when the Bonn Conference came to a standstill over the Northern Alliance’s refusal to make concessions on the composition of the new Afghan government.

Although sympathetic to the Northern Alliance’s position, the Iranians proactively coordinated with us to ensure that the conference reached a deal amenable to the United States and its allies. Iranian diplomatic help was critical in getting the Northern Alliance to accept the new government (despite simultaneous efforts by the Iranian Revolutionary Guards to undermine efforts at Bonn).

As ambassador to Afghanistan, I was authorized to meet with the Iranians in Kabul. Those meeting were helpful in supporting the Afghan government and diffuse crises such as the disputed 2004 presidential elections. However, I had to work with Afghan leaders repeatedly to once again check the destabilizing actions of the Iranian Revolutionary Guards.

Subsequent attempts to work with the Iranians vis-à-vis Iraq were less fruitful. Partly this was due to hostile U.S. statements—notably President Bush’s “axis of evil” speech—which weakened the Iranian’s willingness to cooperate. But it was mostly that the United States failed to pursue hard-headed engagement, which sometimes required assertive actions against Iran but also a willingness to talk.

In late 2002 and early 2003, Iranian policy was not altogether negative. As the U.S. special envoy to the Iraqi opposition to Saddam, I had to negotiate plans for post-Saddam Iraq with opposition groups based in Iran. At the December 2002 Iraqi opposition conference in London, I was able to secure a final deal through indirect contacts with Iran. Tehran ultimately put pressure on Abdul Aziz al-Hakim, the leader of the largest Shiite Islamist group, to ensure that he acquiesced to a deal with the other U.S.-backed opposition parties. Iran’s cooperation was understandable because we had common interests in working against Saddam.

Shortly before the Iraq War, I also engaged in direct talks with the Iranians on Iraq. Javad Zarif, currently Iran’s foreign minister, led the Iranian side. The meetings produced mixed results, as our interests diverged with respect to the post-Saddam political outcome. Talks on postwar Iraq produced few significant agreements, perhaps because the U.S. objective was unclear, with some senior officials seeking a rapid transfer of power to the Iraqis and other advocating for a period of U.S. occupation. We explored further deals on terrorism and other issues as the talks ended, though Iran refused to turn over to the United States, Afghanistan or the countries Al Qaeda figures who had taken refuge in Iran after the U.S. victory in overthrowing the Taliban in Afghanistan. However, Iran placed these figures under house arrest.

During my tenure as ambassador to Iraq from 2005 to 2007, I struggled to get the Bush administration to allow hard-headed engagement with Iran for most of that period. The administration had a contradictory policy. It spoke loudly against Iran, and it refused to talk with the Iranians. But it was not willing to take on Iranian activities in Iraq, which included arming and directing militias acting against the coalition. During this period, we had indirect contacts with the Iranians on Iraq issues. On one occasion, Iranian cooperation was helpful in persuading Prime Minister Ibrahim Jafari to step aside. Ultimately, Washington authorized limited engagement, and it was only toward the end of my tenure that the administration allowed punitive actions against Iran, including the seizure of senior Revolutionary Guards operatives who entered Iraq to coordinate with Shiite extremists.

The lesson of my experience dealing with Iran is that we need at once to take strong action against Iranian aggression and to engage continuously diplomatically. The Iranians will recognize reality if the United States takes hard-headed actions and will be willing to accept limits to their aspirations at the bargaining table.

***

In recent years, Saudi concerns about Iran have become prevalent in Turkey as well. In a break from Turkey’s post-World War II grand strategy, which focused more on Western and Eurasian geopolitics, President Recep Tayyip Erdogan and his Islamist AKP party have increasingly oriented Turkish foreign policy toward the Middle East. As Iranian power has grown, Turkey has sought to mitigate tensions with Saudi Arabia while strengthening ties with the Muslim Brotherhood in the region, and Iraq’s Kurdistan region.

U.S. policies have helped create conditions for intensified regional rivalry among Saudi Arabia, Iran and Turkey. Typical of the pendulum swings we have often seen in U.S. history, the Bush administration’s great exertion into the Middle East was followed by disengagement and retreat during the Obama years. In this case, the retreat came at a particularly inopportune moment. The success of the surge in 2007-08, which encouraged then-Iraqi Prime Minister Nouri al-Maliki to use force in a nonsectarian manner, had bolstered regional confidence in the United States more generally.

The U.S. decision to withdraw from Iraq because of the refusal of the Maliki government to sign a security agreement in 2010, however, undermined these gains and created a new vacuum. Iran gained more influence in Baghdad. Turkey bolstered ties with the Kurdish Democratic Party, dominant Kurdish party and Sunni Islamist parties close to the Muslim Brotherhood. And Saudi Arabia doubled down behind Sunni Arab parties and tribes. These developments in Iraq set the stage for the all-out regional proxy war that erupted in Syria in 2011.

The proxy war in Iraq and Syria has the potential to become more dangerous given that it is contributing to a broader deterioration in relations between Iran and Saudi Arabia. Saudi forces have already deployed near western Iraq in response to the movement of Iran-backed militias close to the Saudi border. The two countries are supporting the opposite sides in Yemen, in Bahrain and in Lebanon. Tensions in the Shia areas of eastern Saudi Arabia, where 10 million barrels of oil are produced every day, are one of several hot spots where sectarian war could disrupt global energy markets and instigate an oil price shock.

To negotiate a new regional arrangement, Washington will need to pursue a hard-headed engagement with Iran.

Claims in the United States that Iranian negotiators are too crafty for our diplomats are ironic in the sense that hard-liners in Iran say the very opposite. In a recent speech directed at rival factions of the Iranian regime, for example, Ayatollah Ali Khamenei cited the United States’ “wealth, large propaganda apparatuses, dangerous weapons and many resources” as a reason why negotiations with Washington are a losing proposition for Tehran.

***

Fear is often the motivation that leads states to take diplomatic steps that many thought impossible before. The horrific rubble of the Syrian civil war has made a powerful impression on every state in the region. Just as the religious wars of Europe eventually provided an impetus for a rules-based order through the Westphalian system, we should consider whether talks to end the Syrian stalemate might lay the groundwork for a broader forum for dialogue.

The United States should steer regional dialogue toward an internal political settlement in Syria. This already appears to be Secretary of State John Kerry’s intention, but it is doubtful that the United States has created the kind of balance of power on the ground that would facilitate such an outcome. The key is for regional players to think through power-sharing arrangements that can create a durable peace. This will require buy-in from all of the major communities in each country. It is likely that decentralized, federal or even confederal arrangements, in which the national government has limited functions and each community governs areas where its people predominate, would work best.

The fact that Iran is open to a federal arrangement suggests that an agreement remains possible. The conflict, however, has escalated to the point that a new multilateral organization may be necessary to resolve the serious disagreements among Iran, Turkey and Saudi Arabia that currently stand in the way of an internal political settlement. Iran and Saudi Arabia remain divided on Syrian leader Bashar Assad’s role and the composition of any successor government in Syria. Iran wants Assad to remain in power as long as possible, while Saudi Arabia is working toward his immediate ouster. In terms of a successor government, Iran wants the central government to remain dominated by an Alawite faction that allows Syria to remain a corridor for supporting Hezbollah. Saudi Arabia, in contrast, favors a Sunni-led unity government. The Kurdish region of Syria is another fault line. Turkey opposes the establishment of an autonomous Kurdish region and believes that Syrian Kurds are working with Iran.

The model for a new Middle Eastern regional organization might be the Association of Southeast Asian Nations. Just as ASEAN promoted regional integration through economic cooperation in Southeast Asia—a region long derided as the "Balkans of Asia"—a new forum for the Middle East could facilitate regionalization through engagement on Syria and other security issues. Countries like Oman, which maintain good relations with Saudi Arabia and Iran, could mediate the establishment of the organization with U.S. support.

Any attempt to lay the foundation for a new Middle Eastern order will also require an understanding between Saudi Arabia and Iran. Facilitating a dialogue between the two countries is a worthwhile and achievable goal for the Obama administration during its final year in office. While a meeting at the head-of-state level is unlikely as long as Khamenei is in power, talks between senior U.S. and Iranian officials, perhaps on the sidelines of a major international conference, could spur progress on this front.

A regional settlement would require a Saudi-Iranian deal that contains four main elements:

Mutual acceptance of religious differences. The leaders of the region must agree that Sunni and Shiite Islam are equally legitimate sects of Islam—a point neither side currently accepts. Wahhabis, Takfiris and other influential forces in Saudi Arabia do not consider Shiites to be legitimate Muslims entitled to the same rights as Sunnis. In Iran, meanwhile, there are no major Sunni mosques in the capital city.

Agreement on how minorities in Sunni and Shiite majority countries should be governed. The lack of minority rights is, in many ways, the root of the problem throughout the Middle East. Iran wants Shiite domination both in Iraq, where the Shiites are a majority, and in Syria, where they are a minority. A broad agreement on equitable power-sharing at the center and/or federalism and local autonomy for Syria, Iraq and Yemen is needed. Movement toward constitutional monarchies in the Gulf states, particularly Bahrain, would allow for popular political participation by their increasingly restless Shiite populations.

Non-interference in the internal affairs of neighbors. This principle stands in contrast to the concept President Barack Obama seemingly endorsed in his recent interview in the Atlantic: an understanding in which Saudi Arabia and Iran “share” the region through spheres of influence. While an agreement to curtail heavy Iranian and Saudi meddling in regional states faces enormous obstacles, it may be more tenable than a sphere-of-influence approach given that Iran, as a non-Arab state, would not, in all likelihood, be accepted by the Shia Arabs.

Guarantees by outside powers. An international conference in which the United States, Europe, China and Russia guarantee an accord between Iran and Saudi Arabia would give confidence to actors across the Middle East. Given how difficult verification and enforcement of such an agreement would be, a new regional cooperation organization along the lines of the Organization for Security and Cooperation in Europe would need to be formed to provide a mechanism for conflict resolution. The greatest challenge facing the organization would be to monitor clandestine support for militias or political groups—the kind of proxy competition that currently dominates the region.

This kind of progress requires more than diplomacy. Without a balance of power in the region that checks Iran’s bid for regional hegemony, Saudi Arabia and the Gulf states may not have the confidence to move toward engagement with Iran. The military capabilities of Saudi Arabia and the Gulf States will need to be strengthened, especially in the area of missile defense, to counter the buildup of Iranian missiles.

***

Statecraft on this scale will require initiatives by Congress and the president to address the United States’ increasingly outdated and ineffectual instruments. I am particularly concerned by the imbalance between our diplomatic and military institutions. Strengthening our diplomacy requires a number of steps, but the first is simply to increase funding for the State Department and other civilian agencies. To respond rapidly to changing circumstances, civilian agencies, like the military, need flexible operational funding and strong capabilities in developing and implementing regional strategies. One option to consider is for the State Department to develop the equivalent of the regional combatant commanders in the military: individuals who are not tied to the interagency process and have the necessary authority and resources to shape policies over an entire region. They could work and travel jointly with the regional combatant commander.

Longer-term changes are also necessary to sustain a new regional order over time. We need to reform the “watch and report” culture of the State Department. It is not, today, a mission-and-results-oriented institution with an operational and expeditionary ethos. On the bright side, I am convinced that a historic opportunity exists to catalyze such cultural change. An entire generation of diplomats—many of whom I got to know during my service in Afghanistan and Iraq—understands how to execute unconventional initiatives in the face of Iranian meddling. They know that diplomacy must be integrated with the use of other instruments, including force, to achieve U.S. objectives.

The above recommendations are far-reaching but well within the capacity of the United States. If we do not undertake this work, the problems of the region—extremism, terrorism and regional conflict—will continue to bleed over into our part of the world, particularly if the Westphalian state system disintegrates even further into sectarian morass. The immediate priorities to this end are to negotiate a settlement in Syria and put Saudi-Iranian relations on a more stable footing.

And to do that, we must reach out to Iran once again.