Scattered over the land between the Jordan river and the Mediterranean Sea lie the remnants of failed peace plans, international summits, secret negotiations, UN resolutions and state-building programmes, most of them designed to partition this long-contested territory into two independent states, Israel and Palestine. The collapse of these initiatives has been as predictable as the confidence with which US presidents have launched new ones, and the current administration is no exception.

In the quarter century since Israelis and Palestinians first started negotiating under US auspices in 1991, there has been no shortage of explanations for why each particular round of talks failed. The rationalisations appear and reappear in the speeches of presidents, the reports of thinktanks and the memoirs of former officials and negotiators: bad timing; artificial deadlines; insufficient preparation; scant attention from the US president; want of support from regional states; inadequate confidence-building measures; coalition politics; or leaders devoid of courage.

Among the most common refrains are that extremists were allowed to set the agenda and there was a neglect of bottom-up economic development and state-building. And then there are those who point at negative messaging, insurmountable scepticism or the absence of personal chemistry (a particularly fanciful explanation for anyone who has witnessed the warm familiarity of Palestinian and Israeli negotiators as they reunite in luxury hotels and reminisce about old jokes and ex-comrades over breakfast buffets and post-meeting toasts). If none of the above works, there is always the worst cliche of them all – lack of trust.

Postmortem accounts vary in their apportioning of blame. But nearly all of them share a deep-seated belief that both societies desire a two-state agreement, and therefore need only the right conditions – together with a bit of nudging, trust-building and perhaps a few more positive inducements – to take the final step.

In this view, the Oslo accords of the mid-1990s would have led to peace had it not been for the tragic assassination of the Israeli prime minister Yitzhak Rabin in 1995. The 1998 Wye River Memorandum and its commitment to further Israeli withdrawals from the West Bank would have been implemented if only the Israeli Labor party had joined Benjamin Netanyahu’s coalition to back the agreement. The Camp David summit in July 2000 would have succeeded if the US had been less sensitive to Israeli domestic concerns, insisted on a written Israeli proposal, consulted the Arab states at an earlier phase, and taken the more firm and balanced position adopted half a year later, in December 2000, when President Clinton outlined parameters for an agreement. Both parties could have accepted the Clinton parameters with only minimal reservations had the proposal not been presented so fleetingly, as a one-time offer that would disappear when Clinton stepped down less than a month later. The negotiations in Taba, Egypt, in January 2001 were on the brink of agreement but failed because time ran out, with Clinton just out of office, and Ehud Barak facing almost certain electoral defeat to Ariel Sharon. The two major peace plans of 2003 – the US-sponsored road map to peace in the Middle East and the unofficial Geneva accord – could have been embraced had it not been for a bloody intifada and a hawkish Likud prime minister in power.

And on it goes: direct negotiations between the Palestinian president Mahmoud Abbas and Netanyahu in 2010 could have lasted more than 13 days if only Israel had agreed to temporarily halt construction of some illegal settlements in exchange for an extra $3bn package from the United States. Several years of secret back-channel negotiations between the envoys of Netanyahu and Abbas could have made history if only they hadn’t been forced to conclude prematurely in late 2013, because of an artificial deadline imposed by separate talks led by secretary of state John Kerry. And, finally, the Kerry negotiations of 2013–2014 could have led to a framework agreement if the secretary of state had spent even a sixth as much time negotiating the text with the Palestinians as he did with the Israelis, and if he hadn’t made inconsistent promises to the two sides regarding the guidelines for the talks, the release of Palestinian prisoners, curtailing Israeli settlement construction, and the presence of US mediators in the negotiating room.

Each of these rounds of diplomacy began with vows to succeed where predecessors had failed. Each included affirmations of the urgency of peace or warnings of the closing window, perhaps even the last chance, for a two-state solution. Each ended with a list of tactical mistakes and unforeseen developments that resulted in failure. And, just as surely, each neglected to offer the most logical and parsimonious explanation for failure: no agreement was reached because at least one of the parties preferred to maintain the impasse.

The Palestinians chose no agreement over one that did not meet the bare minimum supported by international law and most nations of the world. For years this consensus view supported the establishment of a Palestinian state on the pre-1967 lines with minor, equivalent land swaps that would allow Israel to annex some settlements. The Palestinian capital would be in East Jerusalem, with sovereignty over the holy site known to Jews as the Temple Mount and to Muslims as the Noble Sanctuary or al-Aqsa mosque compound, and overland contiguity with the rest of the Palestinian state. Israel would withdraw its forces from the West Bank and release Palestinian prisoners. And Palestinian refugees would be offered compensation, a right to return not to their homes but to their homeland in the State of Palestine, acknowledgment of Israel’s partial responsibility for the refugee problem, and, on a scale that would not perceptibly change Israel’s demography, a return of some refugees to their pre-1948 lands and homes.

Yitzhak Rabin, Bill Clinton and Yasser Arafat watch as Shimon Peres signs the Oslo peace accords at the White House in September 1993. Photograph: J David Ake/AFP/Getty Images

Although years of violence and repression have led Palestinians to make some small concessions that chipped away at this compromise, they have not fundamentally abandoned it. They continue to hope that the support of the majority of the world’s states for a plan along these lines will eventually result in an agreement. In the meantime, the status quo has been made more bearable thanks to the architects of the peace process, who have spent billions to prop up the Palestinian government, create conditions of prosperity for decision-makers in Ramallah, and dissuade the population from confronting the occupying force.

Israel, for its part, has consistently opted for stalemate rather than the sort of agreement outlined above. The reason is obvious: the deal’s cost is much higher than the cost of making no deal. The damages Israel would risk incurring through such an accord are massive. They include perhaps the greatest political upheaval in the country’s history; enormous demonstrations against – if not majority rejection of – Palestinian sovereignty in Jerusalem and over the Temple Mount/Noble Sanctuary; and violent rebellion by some Jewish settlers and their supporters.

There could also be bloodshed during forcible evacuations of West Bank settlements and rifts within the body implementing the evictions, the Israeli army, whose share of religious infantry officers now surpasses one third. Israel would lose military control over the West Bank, resulting in less intelligence-gathering, less room for manoeuvre in future wars, and less time to react to a surprise attack. It would face increased security risks from a Gaza-West Bank corridor, which would allow militants, ideology and weapons-production techniques to spread from Gaza training camps to the West Bank hills overlooking Israel’s airport. Israeli intelligence services would no longer control which Palestinians enter and exit the occupied territories. The country would cease extraction of the West Bank’s natural resources, including water, lose profits from managing Palestinian customs and trade, and pay the large economic and social price of relocating tens of thousands of settlers.

Only a fraction of these costs could be offset by a peace agreement’s benefits. But chief among them would be the blow dealt to efforts to delegitimise Israel and the normalisation of relations with other nations of the region. Israeli businesses would be able to operate more openly in Arab states, and government cooperation with such countries as Saudi Arabia and the United Arab Emirates would go from covert to overt. Through a treaty with the Palestinians, Israel could attain the relocation of every Tel Aviv embassy to Jerusalem, and receive additional financial and security benefits from the US and Europe. But all of these combined do not come close to outweighing the deficits.

Nor have the moral costs of occupation for Israeli society been high enough to change the calculus. Ending international opprobrium is indeed important to the country’s elites, and as they find themselves increasingly shunned, the incentive to withdraw from the occupied territories will likely increase. But so far Israel has proven quite capable of living with the decades-old label of “pariah”, the stain of occupation and the associated impact on the country’s internal harmony and relations with diaspora Jews. For all the recent fretting about decreasing American Jewish support for Israel, the conversation today is not so different than it was at the time of the first Likud-led governments decades ago. Similarly enduring – and endurable – are the worries that occupation delegitimises Zionism and causes discord within Israel. More than 30 years ago, former deputy mayor of Jerusalem Meron Benvenisti wrote of growing numbers of Israelis who had doubts about Zionism, “expressed in the forms of alienation, emigration of young Israelis, the emergence of racist Jews, violence in society, the widening gap between Israel and the diaspora, and a general feeling of inadequacy”. Israelis have grown adept at tuning such criticisms out.

It was, is, and will remain irrational for Israel to absorb the costs of an agreement when the price of the alternative is so comparatively low. The consequences of choosing impasse are hardly threatening: mutual recriminations over the cause of stalemate, new rounds of talks, and retaining control of all of the West Bank from within and much of Gaza from without. Meanwhile, Israel continues to receive more US military aid per year than goes to all the world’s other nations combined, and presides over a growing economy, rising standards of living and a population that reports one of the world’s highest levels of subjective wellbeing. Israel will go on absorbing the annoying but so-far tolerable costs of complaints about settlement policies. And it will likely witness several more countries bestowing the State of Palestine with symbolic recognition, a few more negative votes in impotent university student councils, limited calls for boycotts of settlement goods, and occasional bursts of violence that the greatly overpowered Palestinians are too weak to sustain. There is no contest.

The real explanation for the past decades of failed peace negotiations is not mistaken tactics or imperfect circumstances, but that no strategy can succeed if it is premised on Israel behaving irrationally. Most arguments put to Israel for agreeing to a partition are that it is preferable to an imagined, frightening future in which the country ceases to be either a Jewish state or a democracy, or both. Israel is constantly warned that if it does not soon decide to grant Palestinians citizenship or sovereignty, it will become, at some never-defined future date, an apartheid state. But these assertions contain the implicit acknowledgment that it makes no sense for Israel to strike a deal today rather than wait to see if such imagined threats actually materialise. If and when they do come to be, Israel can then make a deal. Perhaps in the interim, the hardship of Palestinian life will cause enough emigration that Israel may annex the West Bank without giving up the state’s Jewish majority. Or, perhaps, the West Bank will be absorbed by Jordan, and Gaza by Egypt, a better outcome than Palestinian statehood, in the view of many Israeli officials.

It is hard to argue that forestalling an agreement in the present makes a worse deal more likely in the future: the international community and the PLO have already established the ceiling of their demands – 22% of the land now under Israeli control – while providing far less clarity about the floor, which Israel can try to lower. Israel has continued to reject the same Palestinian claims made since the 1980s, albeit with a few added Palestinian concessions. In fact, history suggests that a strategy of waiting would serve the country well: from the British government’s 1937 Peel Commission partition plan and the UN partition plan of 1947 to UN Security Council Resolution 242 and the Oslo accords, every formative initiative endorsed by the great powers has given more to the Jewish community in Palestine than the previous one. Even if an Israeli prime minister knew that one day the world’s nations would impose sanctions on Israel if it did not accept a two-state agreement, it would still be irrational to strike such a deal now. Israel could instead wait until that day comes, and thereby enjoy many more years of West Bank control and the security advantages that go with it – particularly valuable at a time of cataclysm in the region.

Donald Trump and Mahmoud Abbas at the White House earlier this month. Photograph: APA/Rex/Shutterstock

Israel is frequently admonished to make peace in order to avoid becoming a single, Palestinian-majority state ruling all the territory from the Jordan river to the Mediterranean Sea. But that threat does not have much credibility when it is Israel that holds all the power, and will therefore decide whether or not it annexes territory and offers citizenship to all its inhabitants. A single state will not materialise until a majority of Israelis want it, and so far they overwhelmingly do not. The reason Israel has not annexed the West Bank and Gaza is not for fear of international slaps on the wrist, but because the strong preference of most of the country’s citizens is to have a Jewish-majority homeland, the raison d’être of Zionism. If and when Israel is confronted with the threat of a single state, it can enact a unilateral withdrawal and count on the support of the great powers in doing so. But that threat is still quite distant.

In fact, Israelis and Palestinians are now farther from a single state than they have been at any time since the occupation began in 1967. Walls and fences separate Israel from Gaza and more than 90% of the West Bank. Palestinians have a quasi-state in the occupied territories, with its own parliament, courts, intelligence services and foreign ministry. Israelis no longer shop in Nablus and Gaza the way they did before the Oslo accords. Palestinians no longer travel freely to Tel Aviv. And the supposed reason that partition is often claimed to be impossible – the difficulty of a probable relocation of more than 150,000 settlers – is grossly overstated: in the 1990s, Israel absorbed several times as many Russian immigrants, many of them far more difficult to integrate than settlers, who already have Israeli jobs, fully formed networks of family support and a command of Hebrew.

As long as the Palestinian government and the Oslo system are in place, the world’s nations will not demand that Israel grant citizenship to Palestinians. Indeed, Israel has had a non-Jewish majority in the territory it controls for several years. Yet even in their sternest warnings, western governments invariably refer to an undemocratic Israel as a mere hypothetical possibility. Most of the world’s nations will refuse to call Israel’s control of the West Bank a form of apartheid – defined by the International Criminal Court as a regime of systematic oppression and domination of a racial group with the intention of maintaining that regime – so long as there is a chance, however slim, that Oslo remains a transitional phase to an independent Palestinian state.

Contrary to what nearly every US mediator has asserted, it is not that Israel greatly desires a peace agreement but has a pretty good fallback option. It is that Israel greatly prefers the fallback option to a peace agreement. No tactical brilliance in negotiations, no amount of expert preparation, no perfect alignment of the stars can overcome that obstacle. Only two things can: a more attractive agreement, or a less attractive fallback. The first of these options has been tried extensively, from offering Israel full normalisation with most Arab and Islamic states to promising upgraded relations with Europe, US security guarantees, and increased financial and military assistance. But for Israel these inducements pale in comparison to the perceived costs.

The second option is to make the fallback worse. This is what President Eisenhower did following the 1956 Suez crisis when he threatened economic sanctions to get Israel to withdraw from Sinai and Gaza. This is what President Ford did in 1975 when he reassessed US relations with Israel, refusing to provide it with new arms deals until it agreed to a second Sinai withdrawal. This is what President Carter did when he raised the spectre of terminating US military assistance if Israel did not immediately evacuate Lebanon in September 1977. And this is what Carter did when he made clear to both sides at Camp David that the United States would withhold aid and downgrade relations if they did not sign an agreement. This, likewise, is what the US secretary of state James Baker did in 1991, when he forced a reluctant Prime Minister Yitzhak Shamir to attend negotiations in Madrid by withholding a $10bn loan guarantee that Israel needed to absorb the immigration of Soviet Jews. That was the last time the United States applied pressure of this sort.

The Palestinians, too, have endeavoured to make Israel’s fallback option less attractive through two uprisings and other periodic bouts of violence. But the extraordinary price they paid proved unsustainable, and on the whole they have been too weak to worsen Israel’s fallback for very long. As a result, Palestinians have been unable to induce more from Israel than tactical concessions, steps meant to reduce friction between the populations in order not to end occupation but to mitigate it and restore its low cost.

Forcing Israel to make larger, conflict-ending concessions would require making its fallback option so unappealing that it would view a peace agreement as an escape from something worse. That demands more leverage than the Palestinians have so far possessed, while those who do have sufficient power have not been eager to use it. Since Oslo, in fact, the US has done quite the reverse, working to maintain the low cost of Israel’s fallback option. Successive US administrations have financed the Palestinian government, trained its resistance-crushing security forces, pressured the PLO not to confront Israel in international institutions, vetoed UN Security Council resolutions that were not to Israel’s liking, shielded Israel’s arsenal from calls for a nuclear-free Middle East, ensured Israel’s military superiority over all of its neighbours, provided the country with more than $3bn in military aid each year, and exercised its influence to defend Israel from criticism.

The wall separating Israel and the West Bank at Bethlehem. Photograph: Thomas Coex/AFP/Getty

No less importantly, the United States has consistently sheltered Israel from accountability for its policies in the West Bank by putting up a facade of opposition to settlements that in practice is a bulwark against more significant pressure to dismantle them. The US and most of Europe draw a sharp distinction between Israel and the occupied territories, refusing to recognise Israeli sovereignty beyond the pre-1967 lines. When the limousine of the US president travels from West to East Jerusalem, the Israeli flag comes down from the driver-side front corner. US officials must obtain special permission to meet Israelis at the IDF’s central command headquarters in the Jerusalem settlement of Neve Yaakov or at the Justice Ministry in the heart of downtown East Jerusalem. And US regulations, not consistently enforced, stipulate that products from the settlements should not bear a made-in-Israel label.

Israel vehemently protests against this policy of so-called differentiation between Israel and the occupied territories, believing that it delegitimises the settlements and the state, and could lead to boycotts and sanctions of the country. But the policy does precisely the opposite: it acts not as a complement to punitive measures against Israel, but as an alternative to them.

Differentiation creates an illusion of US castigation, but in reality it insulates Israel from answering for its actions in the occupied territories, by assuring that only settlements and not the government that creates them will suffer consequences for repeated violations of international law. Opponents of settlements and occupation, who would otherwise call for costs to be imposed on Israel, instead channel their energies into a distraction that creates headlines but has no chance of changing Israeli behaviour. It is in this sense that the policy of differentiation, of which Europeans and US liberals are quite proud, does not so much constitute pressure on Israel as serve as a substitute for it, thereby helping to prolong an occupation it is ostensibly meant to bring to an end.

Support for the policy of differentiation is widespread, from governments to numerous self-identified liberal Zionists, US advocacy groups such as J Street that identify with centrist and centre-left parties in Israel, and the editorial board of the New York Times. Differentiation allows them to thread the needle of being both pro-Israel and anti-occupation, the accepted view in polite society. There are of course variations among these opponents of the settlements, but all agree that Israeli products that are created in the West Bank should be treated differently, whether through labelling or even some sort of boycott.

What supporters of differentiation commonly reject, however, is no less important. Not one of these groups or governments calls for penalising the Israeli financial institutions, real estate businesses, construction companies, communications firms, and, above all, government ministries that profit from operations in the occupied territories but are not headquartered in them. Sanctions on those institutions could change Israeli policy overnight. But the possibility of imposing them has been delayed if not thwarted by the fact that critics of occupation have instead advocated for a reasonable-sounding yet ineffective alternative.

Supporters of differentiation hold the view that while it may be justifiable to do more than label the products of West Bank settlements, it is inconceivable that sanctions might be imposed on the democratically elected government that established the settlements, legalised the outposts, confiscated Palestinian land, provided its citizens with financial incentives to move to the occupied territories, connected the illegally built houses to roads, water, electricity and sanitation, and provided settlers with heavy army protection. They have accepted the argument that to resolve the conflict more force is needed, but they cannot bring themselves to apply it to the state actually maintaining the regime of settlement, occupation and land expropriation that they oppose.

Since the end of the cold war, the United States has not so much as considered using the sort of pressure it once did, and its achievements during the past quarter-century have been accordingly meagre. US policymakers debate how to influence Israel, but without using almost any of the power at their disposal, including placing aid under conditions of changes in Israeli behaviour, a standard tool of diplomacy that officials deem unthinkable in this case.

Listening to them discuss how to devise an end to occupation is like listening to the operator of a bulldozer ask how to demolish a building with a hammer. The former Israeli defence minister Moshe Dayan once said: “Our American friends offer us money, arms and advice. We take the money, we take the arms, and we decline the advice.” Those words have become only more resonant in the decades since they were uttered.

Until the US and Europe formulate a strategy to make Israel’s circumstances less desirable than the concessions it would make in a peace agreement, they will shoulder responsibility for the oppressive military regime they continue to preserve and fund. When peaceful opposition to Israel’s policies is squelched and those with the capacity to dismantle the occupation don’t raise a finger against it, violence invariably becomes more attractive to those who have few other means of upsetting the status quo.

Through pressure on the parties, a peaceful partition of Palestine is achievable. But too many insist on sparing Israelis and Palestinians the pain of outside force, so that they may instead continue to be generous with one another in the suffering they inflict.

This is an adapted extract from The Only Language They Understand: Forcing Compromise in Israel and Palestine, published by Metropolitan Books. Main photograph: Jim Young/Reuters

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.