Through the fall of 2011, the main question facing the American military in Iraq was what our role would be now that combat operations were over. When President Obama announced the end of our combat mission in August 2010, he acknowledged that we would maintain troops for a while. Now that the deadline was upon us, however, it was clear to me—and many others—that withdrawing all our forces would endanger the fragile stability then barely holding Iraq together.

Privately, the various leadership factions in Iraq all confided that they wanted some U.S. forces to remain as a bulwark against sectarian violence. But none was willing to take that position publicly, and Prime Minister Nouri al-Maliki concluded that any Status of Forces Agreement, which would give legal protection to those forces, would have to be submitted to the Iraqi parliament for approval. That made reaching agreement very difficult given the internal politics of Iraq, but representatives of the Defense and State departments, with scrutiny from the White House, tried to reach a deal.

We had leverage. We could, for instance, have threatened to withdraw reconstruction aid to Iraq if al-Maliki would not support some sort of continued U.S. military presence. My fear, as I voiced to the President and others, was that if the country split apart or slid back into the violence that we’d seen in the years immediately following the U.S. invasion, it could become a new haven for terrorists to plot attacks against the U.S. Iraq’s stability was not only in Iraq’s interest but also in ours. I privately and publicly advocated for a residual force that could provide training and security for Iraq’s military.

Under Secretary of Defense Michèle Flournoy did her best to press that position, which reflected not just my views but also those of the military commanders in the region and the Joint Chiefs. But the President’s team at the White House pushed back, and the differences occasionally became heated. Flournoy argued our case, and those on our side viewed the White House as so eager to rid itself of Iraq that it was willing to withdraw rather than lock in arrangements that would preserve our influence and interests.

We debated with al-Maliki even as we debated among ourselves, with time running out. The clock wound down in December, and Deputy Secretary of Defense Ash Carter continued to argue our case, extending the deadline for the Iraqis to act, hoping that we might pull out a last-minute agreement and recognizing that once our forces left, it would be essentially impossible for them to turn around and return. To my frustration, the White House coordinated the negotiations but never really led them. Officials there seemed content to endorse an agreement if State and Defense could reach one, but without the President’s active advocacy, al-Maliki was allowed to slip away. The deal never materialized. To this day, I believe that a small U.S. troop presence in Iraq could have effectively advised the Iraqi military on how to deal with al-Qaeda’s resurgence and the sectarian violence that has engulfed the country.

Over the following two and a half years, the situation in Iraq slowly deteriorated. Al-Maliki was responsible, as he exacerbated the deep sectarian issues polarizing his country. Meanwhile, with the conflict in Syria raging, an al-Qaeda offshoot—ISIS, or the Islamic State of Iraq and Greater Syria—gained strength. Using Syria as its base, it began to move into Iraq in 2014, grabbing power in towns and villages across Iraq’s north, including Mosul and Tall ‘Afar. These were strategically important cities that U.S. forces had fought and died to secure.

The news from Iraq bothered me to no end. In my view, the ISIS offensive in 2014 greatly increases the risk that Iraq will become al-Qaeda’s next safe haven. That is exactly what it had in Afghanistan pre-9/11. After all we have done to decimate al-Qaeda’s senior leadership and its core, those efforts will be for naught if we allow it to rebuild a base of operations in the Middle East.

Courtesy Penguin

From Worthy Fights, by Leon Panetta and Jim Newton, to be published on October 7, 2014 by The Penguin Press, a member of Penguin Group (USA) LLC. Copyright © 2014 by Leon Panetta.

Leon Panetta served as the director of the Central Intelligence Agency from 2009 to 2011, and as secretary of defense from 2011 to 2013. He was a member of the U.S. House of Representatives from 1977 to 1993, the director of the Office of Management and Budget from 1993 to 1994, and President Clinton’s chief of staff from 1994 to 1997. He is the founder of the Panetta Institute for Public Policy, and has served as professor of public policy at his alma mater, Santa Clara University.

Jim Newton is an editor at large of the Los Angeles Times, where he has worked for twenty-five years as a reporter, an editor, a bureau chief, and a columnist. He is the author of two critically acclaimed biographies, Justice for All and Eisenhower.

Contact us at letters@time.com.