Aaron David Miller is a vice president at the Woodrow Wilson Center in Washington. He served in the State Department from 1978 to 2003.

Israel wanted no part in it. And neither the Israelis nor the Palestinians were scheduled to attend. Yet Secretary of State John Kerry remained optimistic ahead of Friday’s sure-to-go-nowhere Middle East peace conference in Paris. “What we are seeking to do,” he said, “is encourage the parties to be able to see a way forward so they understand peace is a possibility.”

I recognize that sentiment: wanting to remain upbeat, even while knowing that the odds are long. For much of my 24-year career as a State Department Middle East analyst, negotiator and adviser, I held out hope that a conflict-ending peace agreement was possible. I had faith in negotiations as a talking cure and thought the United States could arrange a comprehensive solution. I believed in the power of U.S. diplomacy.

But by the time I left government in 2003, I was a disillusioned diplomat and peace processor with serious doubts about what the United States could accomplish in the Middle East. I realize now that, like Kerry, I was tilting at windmills. U.S.-brokered peace in the Middle East is a quixotic quest. And the more we try and fail, the less credibility and leverage we have in the region.

Looking back now, the high point of my optimism was probably in 1991, the year we orchestrated another, more productive Middle East peace conference in Madrid. I remember that on one of nine trips that led to the conference, a large fly boarded the plane with us at Andrews Air Force Base and buzzed annoyingly around the staff compartment. I was vainly trying to swat it when Secretary of State James Baker walked by to brief the press in the rear of the aircraft. Hours later, while drafting talking points, I felt a presence over my shoulder and turned just as Baker’s large hand dropped the fly onto my yellow legal pad.

That kind of sums up how I thought about our diplomacy back then: With good timing and assertive American leadership (something short of fly-crushing brute force), we could solve festering problems once and for all. My memos at the time had a yes-we-can edge.

The moment seemed ripe for a Middle East breakthrough facilitated by the United States. Our influence in the region was at an all-time high. The U.S. military had just pushed Saddam Hussein out of Kuwait, and the Israelis and Arabs were off-balance — in the case of Jordan, Syria and the Palestinians, they were looking for ways to ingratiate themselves into America’s good graces. We were respected, admired and feared in the region to a degree we haven’t been since.

Baker, meanwhile, was probably the best U.S. negotiator to tackle the Middle East since Henry Kissinger brokered three disengagement agreements in the wake of the 1973 Arab-Israeli War. I watched Baker cajole, pressure and threaten to walk out on both Israel’s Yitzhak Shamir and Syria’s Hafez al-Assad, and I saw him huddle with Palestinians like a football coach to encourage them to attend the peace conference. It helped that he had the full backing of President George H.W. Bush — his close friend who cared about Mideast peace and was making good on a pledge to Saudi Arabia that he’d take on the ­Arab-Israeli issue after the Persian Gulf War.

The Madrid conference produced the first direct bilateral negotiations and peace process success between Israelis and Arabs — Syrians, Jordanians and Palestinians — since the Egyptian-Israeli agreement 12 years earlier. I reveled in our achievement and marveled at what U.S. diplomacy could accomplish when it was tough, tenacious and strategic.

My mistake was in believing that Madrid, which really produced only a procedural breakthrough, would necessarily create a foundation for progress on the substantive issues. I thought if we just kept the process going, if we were committed and creative, we would somehow find our way to agreement between the Israelis and the Palestinians on Jerusalem, borders and refugees, along with agreement between the Israelis and the Syrians on the Golan Heights. But we never got there. Process can’t substitute for substance.

I maintained my misplaced optimism into the Clinton administration. Sitting with my family on the South Lawn of the White House in September 1993, watching President Bill Clinton preside over the historic handshake between Israeli Prime Minister Yitzhak Rabin and Palestine Liberation Organization Chairman Yasser Arafat, I believed, in what had to be one of the most stunning misjudgments of my career, that the peace process had become irreversible.

The Israelis and the Palestinians, without U.S. involvement, had reached an agreement on mutual recognition and a declaration of principles that was supposed to get them toward talks on the big issues. I really thought they had taken ownership of their negotiations and would dedicate themselves to making the Oslo Accords stick.

Through the crises of the next seven years of the Oslo process — Palestinian terrorist attacks, Israeli settlement activity, the assassination of Rabin by an Israeli extremist opposed to Oslo — I kept the faith that the almighty peace process ultimately would succeed. I convinced myself that with added urgency from the United States, the confidence-building, interim measures laid out in the Oslo agreement could be made to work and pave the way for negotiations on the core issues. Early in 1997, literally down on my hands and knees in the West Bank city of Hebron measuring the width of a street that figured prominently in the negotiations, I felt both small and ennobled. This was important, and I’d do anything to keep the process alive.

My commitment, and the illusions that sustained it, would take me all the way to the ill-advised, ill-timed and ill-prepared July 2000 Camp David summit: a last-ditch effort to save the Oslo process. During a briefing a week before, Clinton went around the room asking everyone to gauge the prospects of the summit. And everyone, from the national security adviser to the secretary of state, said more or less the same thing: There was a chance; Ehud Barak and Arafat would make decisions only in the heat of a summit; the president owed it to the cause and to himself to pursue peace before the end of his term. The assessment we all should have given him was that there would be no conflict-ending accord or even a framework agreement, because neither Barak nor Arafat were ready to pay the price, and the president was unlikely to bridge the gaps. But I brushed aside my doubts and echoed the others. Part of me was concerned about pissing off everyone else in the room. The invitations to Arafat and Barak had already been issued, so the briefing really was a formality. But part of me still wanted to believe that we could make peace.

The president thought that if he could just get the Israelis and the Palestinians in the room, he could somehow get them to an agreement, building on what Barak was prepared to offer and using the famous Clinton powers of persuasion. But we had no strategy, we coordinated too closely with the Israelis, and we had no Arab buy-in on issues such as Jerusalem nor any sign that the Palestinians would move off their core demands. We didn’t run the summit; the summit ran us.

When I think back about that fateful period, I shudder. With the best of intentions, in eight months, we planned three presidential negotiations (two on the Syrian track and one on the Palestinian) and failed at all three.

What I should have realized all along was that strong U.S. mediation can’t make up for weak leadership of the parties to a negotiation. We can’t talk them into getting control over their political constituencies. And we can’t expect that our enthusiasm will persuade them to invest in solutions, take necessary risks or recognize that a negotiated settlement is in their interest (and not just ours).

In March 2002, during the height of the second intifada, President George W. Bush’s Middle East envoy, Anthony Zinni, and I were sent to negotiate a cease-fire between Arafat and Israeli Prime Minister Ariel Sharon. But that was either the Bush administration’s idea of a cruel joke or just a throwaway talking point before the final break with the PLO leader.

That week, a Palestinian suicide bomber had blown himself up at a Passover seder in Netanya, killing 30 Israelis and wounding 140. Israeli forces responded with Operation Defensive Shield, entering the West Bank and imposing closures on most major Palestinian cities and towns.

I’ll never forget the scene in Arafat’s compound. The place reeked of foul air, body odor and too few working toilets. The only light, in what had been in better days a reasonably well-lit conference room, came from candles and a bit of sun that managed to peek through windows that were almost completely blacked out for fear of Israeli snipers. And there in the gloom sat a self-satisfied Arafat, his black machine gun ominously displayed on the table, holding forth about how he’d be rather be martyred than surrender to Israel’s diktats.

There was no longer any way for me to rationalize the importance of process without direction, negotiations without substance or even the use of the word “peace.” Our overinflated optimism at Camp David had had real costs. After raising expectations we couldn’t deliver on, we blamed Arafat for the summit’s failure, and that made it easier for him, in the wake of Sharon’s provocative visit to the sacred Temple Mount, to acquiesce to and encourage the violence that would become the second intifada.

U.S. diplomacy can be effective when we have partners willing to make decisions, when all parties feel an urgency to make those decisions and when gaps separating the parties can actually be bridged. The Iran nuclear agreement, while greatly flawed, is a case in point. It succeeded because it was not a transformational but a transactional arrangement, a highly detailed arms-control accord of arguably limited duration and scope that both the United States and Iran wanted for their own reasons.

But when it comes to matters that cut to the core of people’s identities — such as Jerusalem or Palestinian refugees, or the social engineering required to end Syria’s civil war — or creating an outcome in Iraq or Libya that produces stability and good governance, the United States doesn’t have the horses to pull the wagon. The inconvenient reality is that we will never have a greater stake in this region, or more power to remedy its ills, than those who live there.

I haven’t given up hope for smart and well-timed U.S. diplomacy. But I’ve abandoned my illusions of just how much America is able and willing to do to repair a badly broken, cruel and unforgiving Middle East.

As the fix-it people, Americans have a hard time accepting that we can’t sort out conflicts when those directly involved aren’t willing or able to do so. But sometimes, it makes more sense for our diplomats and negotiators to stay home rather than look weak and ineffective while searching for solutions to problems they simply cannot resolve.

Twitter: @aarondmiller2

Read more from Outlook and follow our updates on Facebook and Twitter.