Living under the rule of Da’esh (the self-described Islamic State) would undoubtedly be horrific. Even if you accept that some of those apparent enthusiasts for the Rule of the Saints whom Da’esh interviews in Mosul, Raqqa or Deir az-Zour for its ­sequence of video postings do mean it – and aren’t just worrying about the guys in the background with beards, AK-47s and hunting knives – you have to assume that most people do not want to live in a world where every cigarette, every tune that springs to the lips, every morning shave, every slip of the veil, any expression of disgust at the dropping of helpless men from tall buildings, the enslavement of minorities or a flicker of aberrant sexual desire could lead to instant execution.

Just reading about all of this is bad enough. I do a lot of it for professional reasons. And I sometimes want to scream. It’s not because the quality of the commentary is low. Quite the contrary. There is an impressive debate out there – in a dozen languages – about how Islamic the group is, possible links to the Ba’ath Party and the former Iraqi army, the significance of its particular form of Islamism for the regional and international order, about its relationship to other Islamist groups, what we are doing wrong in countering its model of violent extremism, how it is or isn’t a state, how many fighters it has (30,000? 100,000?), whether it is more or less of a threat than al-Qaeda, why we shouldn’t say it is new and why we should (sometimes in the same piece), about whether religion is a helpful category at all, how we should explain apparently significant levels of popular support and whether political science has all the answers.

Most recently, I’ve read that Da’esh represents a Dostoevskyan revolt against the unenchantment of the world, the bad faith of anomic individualism, global capitalism and that pantomime villain, neoliberalism, with hundreds of millions of wannabe Raskolnikovs pausing on the bridge of ­history before taking their revenge. All of this to ghostly cheers from Carlyle, Schopenhauer and other prophets of the alienation apocalypse.

Yet none of this flood of commentary, it seems to me, has had any impact on real-world policy responses – either regionally, from China, Russia or India, or from the west. Is this a dialogue of the deaf, the usual disconnect between practitioners and the commentariat? Is it the inevitable consequence of some scholars’ distaste for “peddling their wares in the service of the state” (perish the thought). Is it the self-importance of theorists? Is it the sheer pressure of events on policymakers and military planners – or perhaps their ignorance? Or is something more profound going on?

Three months ago, I was in Naypyidaw talking to Daw Aung San Suu Kyi about precisely these issues (and yes, we also discussed the Rohingya). We’d had occasional conversations about the Middle East and sacralised extremism when I was British ambassador to Burma between 1999 and 2002, but the topic has a new resonance now. At the International Institute for Strategic Studies Shangri-La Dialogue in Singapore a few days earlier, the focus had been not just on the expansion of the Sinosphere, as you might expect, but also on the rise of various forms of violent or exclusionary Islamism.

This worries the Muslim-majority states of south-east Asia; it worries Burma and Thailand, which face a visceral reaction from majority Buddhists to the apparent threat; and it worries China, which has problems in Xinjiang and keeps a wary eye on Afghanistan, and whose New Silk Road is designed to drive through the Asian heartlands of Islam to the Gulf and Europe, both of which also face the same apparently intractable threat.

This is no longer some exotic subject, confined to area specialists and policy wonks, and representing a specific material challenge within the context of one or two regional state systems. It is global, surfing world networks of communication, exchange and trade. It is transnational. And above all it is ideational: that is to say, the claims made by certain forms of political Islamism about the nature of the state, the meaning of Islam, the character of the authentically Muslim, the relationship between the sacred and the profane, the ungodliness of human political dispensations, and the religious duty to replace them by force if necessary, represent a more serious and sustained challenge to the stability and security of the Middle East than bombs, missiles, guns and knives. Most of these claims are not new. Many of them figure in disputes that Islamic historiography tells us go back to the earliest days of the faith. But they have resonated down the centuries, shape the expression of 21st-century grievances and are associated with modern forms of communication and political mobilisation that amplify their impact.

In late 2009, a few months before the most important general elections in Iraq since 2003, the so-called De-Ba’athification Committee, the Robespierrean mechanism established under the Coalition Provisional Authority in 2003 to purge Iraq’s body politic of Ba’athism, was suddenly reactivated. I was the British ambassador at the time. I was warned by Sunni leaders – and by secular Shia politicians such as Iyad Allawi – that this was simply designed to emasculate any opposition to Prime Minister Nouri al-Maliki and his Iranian backers: it represented a resectarianisation of Iraqi politics and would lead to disaster. Some important diplomatic colleagues disagreed. But Allawi and his allies were right. With a few others, I tried to fight against the tide. We failed.

In early 2011, as Maliki strengthened his grip on power with no noticeable impact on the quality of government or provision of services or welfare across the country as a whole, I thought the protests in Baghdad and elsewhere against the government’s failure to govern in the common interest might lead to such pressure that the gov­ernment would be forced to reconstitute itself as a more efficient, less partisan actor. I was wrong. The lure of power was too strong. And the protests, which culminated with the violent clearing of the mass Sunni protest site in Hawija, near Kirkuk, in April 2013, went rogue.

***

In Abu Dhabi a few weeks ago, I sat late at night with my old friend Dr Rafi ­al-Issawi, the Sunni former finance minister of Iraq. He was one of the very few senior figures – Sunni or Shia – in Iraq during my time there who was prepared to reach out across sectarian boundaries and think creatively about political solutions to essentially political problems. He still does. Perhaps that is how a medical director who kept his hospital open during the savage battles for Fallujah in 2004, one of the most effective of all the Sunni politicians in Iraq and a man of integrity, is bound to think. His reward? Attempted assassination and then exile on trumped-up charges of supporting terrorism.

We talked of ways to bring the Sunnis back into a political process – as they had been brought back, often by themselves, between 2005 and 2008. I had the same conversation a few days later in Amman, and have done so again in the past week in Erbil and Sulaymaniyah, with other Iraqi friends and politicians. Others have identical aspirations. But the Sunni leadership in Iraq, never cohesive, is now even more fragmented than it was in 2011. And the divisions along the lines of identity in Iraq, a country that the distinguished late Iraqi sociologist Ali al-Wardi once characterised as defined by multiple parallel identities, are deeper than ever before.

This represents the triumph of those who have sought to instrumentalise sectarian identities in the pursuit of power: to demonise Sunnis as closet Ba’athists who want revenge, to terrify the Shias with the prospect of renewed oppression, and to claim that security in a state such as Iraq (and by implication the wider region) is assured only through the communal protection afforded by the political expression of essentialist identities. And Da’esh is the flipside. Its appeal in Iraq and elsewhere is powerful. It claims to see through the broken promises and conspiracies of the past decade – ­indeed, the past century, and sometimes the past millennium. It offers a new, sacred, transnational and at the same time territorial model of righteousness. More importantly, it offers a framework within which to construe the pain of dispossession, the loss of power, the rise of others less devout, less deserving, less human. Not only does it claim to answer the question of why sinners’ ways prosper, but by framing the entire issue as one of godliness, it provides a divine dispensation for anything and everything that advances the practical goal of recapturing power and wreaking revenge on an infidel and hostile world. It combines an appeal to faith, tribalism (more precisely to asabiyya, the group solidarity that the great Arab historian Ibn Khaldun identified 600 years ago as the driver of cohesion but also of cyclical conflict in the societies he knew), resentment and the promise of glory.

The mobilisation of identity in the service of combatant politics is hardly exclusive to Da’esh. We see it elsewhere: nationalist identity in China, nationalist-Buddhist in the case of Burma, ethno-religious in Malaysia, sectarian and religious in the case of Islamist movements and some states across the Muslim world. We saw it in the Balkans in the 1990s. We also see a wide use of modern communications techniques to create and then exploit allegedly ancient passions and fears; the mobilisation of grievances; the deliberate destruction of individual and social solidarity; and the construction of radically simplified, harshly communal but powerful forms of connectivity.

The use of purposeful and often sacralised narratives to shape the approach to secular political goals may not be new. They were as much a feature of early-17th-century Europe as of Baghdad in 2010 or Cairo in 2013: think Anabaptist Münster or the English Fifth Monarchists.

***

But, it seems to me, what is new in the case of Da’esh is the striking combination of a number of features: the clarity of its transnationalism; the speed, professionalism and discipline with which its global religio-identity narratives are produced and flexed; the way these are backed by an encompassing Islamist jurisprudence (look, say, at Turki Binali’s Muqarrar fi al-Tawhid – Da’esh’s basic coursebook); the skill with which they are harnessed to its political goals. There is also the subtlety with which it tracks and shapes opinion (the first message was “we stop genocide in Syria”; then “we champion Sunnis in Iraq”; then “we build a service state”; then “we are a righteous caliphate”); and the effectiveness (so far) of its hybrid military operations in the Syria/Iraq theatre and its stripped-back crowdsourcing model. All these features are linked by a complex thread – the studied marketing of a constructed identity, or performativity, as the theorists say: this is designed to reinforce Da’esh’s self-image as a rightly guided eschatological and millenarian movement, to convey what one analyst calls “a sense of apocalyptic time”, but also to proclaim the temporal success of such an enterprise and communicate purpose while the world watches.

When in July 2014 Abu Bakr al-Baghdadi stood up in his black robes and turban at the great mosque of Nur ad-Din Zengi in Mosul to proclaim the caliphate, he was implicitly claiming to be the heir of the Abbasids (scourges of Ummayad decadence) and of Nur ad-Din’s celebrated military commander, Salah ad-Din (scourge of the Crusaders) – a righteous religious and secular leader and a conqueror. That resonates in a region where history happened yesterday. This is a movement that wants to seem not just blessed and rooted in a mythical past but also profanely effective in a present that holds a mirror to the past.

It is irrelevant whether any of this reflects reality (whatever that may mean). The Da’esh narrative shows a striking disdain for what most of us understand to be the intractable complexity of the world as it is and the unknowability of the future.

This seems to me profoundly significant. It is not that I think we are helpless in the face of Da’esh and its analogues. We are not – though sometimes it can seem so as news breaks of yet another atrocity or another wilayat declaring its loyalty.

We almost certainly exaggerate its power and its appeal. Da’esh controls strips of the most impoverished and sparsely populated parts of Iraq and Syria. We talk of those Muslims who go to fight: but we do not attempt to explain why most Muslims do not (and may even exaggerate the numbers that do). Those who glimpse the long-awaited revolt of humanity against the Weltgeist fail to explain why so many of those Iraqi Sunnis who fought against the coalition between 2003 and 2007 later fought with the coalition for the right to be part of what they hoped would be a new, pluralist state reintegrated into a global economy. There may be significant sympathy in the region for Da’esh as an expression of Sunni revanchism, and confusion about exactly how Islamic Da’esh is. But all credible opinion polling shows a consistently low level of support for its political project: claims to the contrary are unsourced. Indeed, alongside that sympathy, the widespread belief in the region even among the educated elite, that Da’esh is a US-led conspiracy against Arabs, perhaps reflects both familiar refusal of personal agency and a deep unease that such a project might in fact succeed.

What we saw consistently throughout the so-called Arab spring was that ordinary people throughout the Middle East wanted respect, security for themselves and their families, institutional stability, education for their children, better services and an economy that provided decent jobs. They wanted better practical outcomes, not demented visionaries.

We also underestimate the resilience of the state system in the Middle East. I do not think it is about to disintegrate. Some of it certainly has: but that is more about the failings and the crimes of leaders such as Bashar al-Assad, Muammar Gaddafi and Saddam Hussein – and because of often ill-judged and uncoordinated interventions by the US, the UK, France, Russia, Iran, Turkey or other regional actors. There is a lot of nonsense talked about Sykes-Picot. All the evidence suggests that people in the Middle East rather like their national identities and generally feel Iraqi, Syrian, Egyptian, Palestinian, Saudi, Bahraini, Emirati and so forth as well as or before feeling Arab or Amazigh or Muslim or Christian. The late Yasser Arafat used to tell me that Palestinian Christians were an essential part of Palestinian society. They were the yeast – al-khamirah – that leavened identity and blended it with the land. Otherwise, the lure of de-territorialised extremism would be too strong. That applies to all unrooted monocultures, which is precisely why Da’esh seeks to create them. But we in the west, policy practitioners and academic analysts alike, can still seem paralysed in the face of this challenge. It is as if we are simply dazzled by the account that Da’esh gives of itself. If we are not dazzled, we are ignorant.

An article in the Guardian last month by Federica Mogherini, the EU’s high representative for foreign affairs and security policy, hailed the Iran nuclear agreement as a “disaster for Isis”. “There is nothing more worrisome to Isis than co-operation between ‘the west’ and the Muslim world,” she wrote, “for it defies the narrative of a clash of civilisations the group is trying to revive.” But Da’esh does not understand the Muslim world in this sense. It is specific: authentic Muslims believe in tawhid (the indivisibility of God and His order) and hakimiyyah (the absolute sovereignty of God, which invalidates human constructs such as non-sharia legal systems and electoral democracy), reject tawaghit (a Quranic word meaning roughly “idolatry” or “an illegitimate object of worship”, used to damn all current Arab regimes) and have nothing to do with kuffar (infidels).

So the sort of co-operation that Mogherini proposes – between those whom Da’esh despises as Rawafid (Shias, therefore infidels or apostates) and Nasarah (Christians, therefore polytheists and infidels) – precisely proves their point. If we are to do better, and we must, then how do we understand the nature of the policy challenge better and what do we and others do about it?

When, after Rangoon, I was posted as consul general to Jerusalem in May 2003, within a couple of months or so there were three major terrorist attacks in the city, two involving suicides and one a car bomb. As ambassador in Baghdad from 2009 to 2011, I became inured to the crump of a car bomb at entrances to the Green Zone or the sudden rush of a mortar overhead. Military colleagues were still being targeted with IEDs, EFPs and RDX. Violence against civilians remained too high. But it had plateaued. Al-Qaeda’s successor, the Islamic State in Iraq, was apparently bankrupt and isolated. Even in 2011 in Libya, when there were at least two attempts by Abdullah Senussi, Gaddafi’s sinister henchman, to blow us up, I thought this was a passing phase, the terror of a dying regime in a single country.

***

I was wrong yet again. Nothing comes from nothing. And you have to go back to the beginning to understand properly what we are seeing. That means at least as far as Ismailiya in March 1928: Hassan al-Banna later claimed that it was here he recruited, from the Suez Canal Company, the first six members of his new Muslim Brotherhood, the first mass political movement driven by the idea of religious rather than national identity in a modern Arab state. The Brotherhood did not emerge from a vacuum. It had roots in the Muslim reformist movements of the late 19th and early 20th centuries, as well as in the collapse of the Ottoman empire, the disappointments of the Paris Peace Conference and the compromises of Sèvres. You could trace in their thinking a mix of modernist, Salafi, Sufi, Egyptian nationalist and, indeed, Syrian influences. But they were new in the sense that al-Banna brought all this together for the first time in the Arab world in order, like certain radical European movements of the time, including the Italian Partito Nazionale Fascista and later the Spanish Falange, to pose a bold challenge to a modern state through techniques of mass mobilisation, linked to a set of ideas about tradition that was often invented and informed by a simplified and directly personal way of encountering foundational texts.

The Brotherhood has never been al-Qaeda or Da’esh, though it has often had, in its various manifestations, an ambiguous and situational attitude to sectarianism and use of violence (ask the Alawites who fled Aleppo between 1978 and 1982, Egyptian Christians and Shias, or many Libyans today).

But they and counterparts such as Abul-A’la al-Maududi and Abul Hasan Ali Nadwi in Pakistan and India introduced a lasting set of ideas into the mainstream of political debate in the Muslim world – a restored caliphate; sharia as an unchallengeable and eternal body of God-given law regulating individual, social and political behaviours; the vague aspiration for a more authentically Islamic state; Sayyid Qutb’s thinking about jahiliyya (the pre-Islamic state of religious ignorance, reinterpreted to anathematise modern Muslim states); combined with quasi-Gramscian notions of political struggle and national/transnational reculturation. These ideas have provided the template for all subsequent movements, whatever their actual preference for manoeuvre against or frontal assault on the institutions of the modern Arab state.

After the suppression of the Egyptian Brotherhood by Nasser in 1954, its members scattered – across the Middle East, later into Europe and then the United States. In the Gulf in particular, sponsored often by governments for their own purposes, Brotherhood mobilisation and political activism became a model for independent-minded Salafi groups that rejected the accommodationist establishment. In turn, elements of the Brotherhood adopted certain Salafi approaches: a more intense and literalist focus on the foundational texts, an emphasis on the political application of tawhid (which brought politics into the sphere of the divine) and a rigorous conservatism in exegesis, personal behaviour and religious practice. There were parallels in Shia Islam, with Ali Shari‘ati acting as midwife to a heterodox imamism, previously foreign to Twelver Shiism, which hijacked a revolution. Shari‘ati died in Southampton before he could see the full impact of his ideas. Ali Khamenei translated Sayyid Qutb into Persian.

It was out of this intellectual ferment that other streams of activist and often far more violent Islamism arose: al-Jihad, al-Gama’at al-Islamiyya (GaI) and Takfir wal-Hijrah in Egypt, Jamaah Salafiyyah al-Muhtasibah in the Gulf (including Juhaiman al-Otaybi and his group, which sought to seize the Grand Mosque in Mecca in 1979), various strands of the so-called Islamic Sahwa (“awakening”) in Saudi Arabia in the 1990s and eventually al-Qaeda and its offspring. This is the stage on which Da’esh now struts.

***

Yet some respected commentators tell us that perhaps Da’esh isn’t that bad: look at the appalling brutality at the root of all modern European states, with their ritualised and theatrical acts of foundational violence. That didn’t turn out so badly, did it? But cutting people up in the 13th century, or indeed in 1661, was normative. Everybody did it. The Buddhist kings of imperial Burma, like the Turks and their enemies in eastern Europe, impaled their opponents. The Golden Horde made mountains of skulls. In 1571 the defeated Venetian governor of Cyprus, Marcantonio Bragadin, was flayed alive by Lala Mustafa Pasha after a promise of safe conduct. If you were from an English noble family in 1470 you had probably seen half your male relatives beheaded or hacked to death.

But we don’t do this today, and to suppose that what Da’esh does follows in a long tradition of state foundation is wrong. Its use of mediatised terror is deliberately and spectacularly transgressive: that is why we pay attention. Da’esh claims to wish to destroy the state system not just in the region but the entire world. And any practices or institutions of state that it produces in its place are not obviously designed to appeal to a shared past that might, in the eyes of some, become a common future. They seem designed instead to overturn all the assumptions on which we and the states of the region base our notions of civic, social and political behaviour. This is an Islamist Stunde Null (zero hour).

At the same time, nothing Da’esh does is individually new. We’ve seen theatrical brutality before. We’ve seen claims to resurrect the caliphate before: by one count, 19 jihadi proto-states (mostly short-lived) between 1989 and 2015. We’ve seen transnationalism and, at the same time, attempts to expand specific territorial control before. We’ve seen takfirism and religiously inspired eschatological nihilism before. We’ve seen the popular use of social media to spread violent jihad and disaffection from society before. But we haven’t seen these features combined and directed with the clarity, speed and tactical adaptiveness of Da’esh. It has internalised and acted on all the lessons radical jihadi groups have learned, certainly since 1979 and probably since Nasser’s suppression of the Muslim Brotherhood in the 1950s. The Da’esh forces are market leaders in a form of savage ideological puritanism, designed to appeal not locally or regionally, but to an archipelagic network of supporters and sympathisers scattered throughout the world. We – western and regional governments – talk about lessons learned but mostly only ever succeed in identifying them. We lack nimbleness, we sometimes lack knowledge and we seem to lack determination. Violent extremists have to adapt rapidly or die.

So is it inevitable that western powers, when faced with apparently intransigent movements that sustain the capacity to act, will simply fail? If so, why? If not, what should we do, and how can experts and practitioners help each other?

In the end, Da’esh can be defeated militarily. That has to be our immediate shared objective. Defeat deprives Da’esh of the narrative of success foretold – a common theme among Islamist movements. The consequences of not doing so are terrible, for the peoples of Syria, Iraq and the wider region. I happen to think they are also terrible for the world. Armed conflict between states globally may be shrinking but it is increasing at a destructive, substate level. So are the piles of victims, most of them Muslim and in the Middle East, with over 230,000 in Syria alone. And newly emerging forms of conflict, some cyber-related, create an important non-material dimension.

Da’esh comprehends both the material and the ideological. It may not represent the same threat at the moment to the security of western states as al-Qaeda. But doing nothing to stop it in its heartlands of Iraq and Syria corrodes the very notion of international order, an indispensable aspiration, whatever you may think of the reality.

Such a defeat can be procured only on the back of a new, widely shared understanding of what security means in the region. This cannot be imposed by lectures or arise out of university seminars or pious hopes. It requires patience, partnerships with states in the Middle East sharing the same goals, better command-and-control systems, more effective intelligence collection, the sort of sustained, informed and frank analysis that we have recently lacked – and adequate resourcing. Aid and military force are the handmaidens of politics, not autonomous instruments in their own right; and area experts have a public responsibility at times like these.

It also requires political will and a high level of ambition. This is in short supply after the experiences of Iraq and Libya. The west and the key Sunni Arab states still

often talk past each other. It also requires the US, post the Iran deal, to act as convener. I understand why this might not seem attractive to many Americans or, perhaps, to some in the region. But if you think the Arab and Islamic worlds are important, I don’t see an alternative. Nudges will not work: they usually produce tactical responses and alliances of convenience. And half-hearted containment is an admission of impotence. What is required is full-scale re-engagement.

In a region where there is no natural balance of power, little prospect of one emerging (in my view) and a lack of capacity to deal with a multitude of simultaneous challenges, only the US will do.

***

I would love Britain and the European Union to be part of this effort. That will require a more convincing public account of the global common interest than we collectively have at present, when all too often a form of political as well as economic mercantilism reigns on all sides and gestures take the place of thought. And we can do without the narcissism of much foreign-policy posturing.

The basis for such a new coalition can only be found in a new seriousness of purpose that looks not for eye-catching initiatives, but for a sober and sincerely joint appreciation of the threat, together with a commitment by all concerned to see the task through together, buttressed by a recommitment to the international rule of law. That would probably require a reshaped Nato and UN as well. One can dream.

Even then the wider challenge will remain. This is not because Da’esh is materially powerful and will endure. It may collapse under the weight of its own brutal contradictions, even though it seems to have been able so far to replenish combat losses successfully and has developed significant extractive financing mechanisms. It is rather because it represents the latest manifestation of a metastasising idea that has shown an extraordinary capacity to survive and sustain in different shapes a challenge to the legitimacy of the post-colonial state order, not only in the Middle East but in the Islamic world more generally. Whatever you may think of this order, the events of the past four years have shown beyond doubt that a remedy for whatever discontents exist can be found not through radical rupture, revolution or accommodation, but only through collaborative effort, through a willingness to use collective force where force is justified and where force can be effective, and through the sort of socio-political evolution that revolutionary violence makes impossible.

This is fundamentally a struggle within Sunni Islam for ideational hegemony: the power to determine norms. That is precisely why the Taliban in Pakistan have argued that Abu Bakr al-Baghdadi’s claim to a caliphate is non-Islamic, why Abu Musab al-Zarqawi, the leader of the original Islamic State in Iraq, split with Ayman al-Zawahiri, Osama Bin Laden’s deputy in al-Qaeda, nearly a decade ago, why al-Qaeda in the Arab Peninsula is competing with Da’esh in Yemen and why Da’esh is fighting with Jabhat al-Nusra in Syria. It is why there is a shift among certain states to co-opt rather than denounce certain Muslim Brotherhood-related groups and why the competition between states and substate groups for bay’ah (pledges of loyalty) is so fierce.

And this struggle goes, as always, to the heart of politics and power. Only other Muslims can win it, but they cannot do so by appealing to ancient institutions or simply by directing people to believe the right things. For a century, states have sought and failed to co-opt Islam. Da’esh may claim to draw on the great Islamic traditions of scholarly exegesis and jurisprudence. But it does so wilfully, selectively and self-servingly, scorning what it sees as the higher-purpose-and greater-good-based rulings of the orthodox, the scholastic timidity of the Azhar schoolmen, the subordination and Hanbali clericalism of the Saudi ulema, and the radical doubt of intellectuals. It thinks all this serves worldly interests, not the purposes of God. Like Sayyid Qutb, Da’esh insists that God speaks directly and clearly to the individual believer through the sacred texts, which the individual has the right to encounter without the mediation of conventional scholars. It reframes the doctrine of jihad as a collective and individual obligation to take up arms offensively. It claims the right to correct the historical errors of the world Muslim community by violence if necessary. And it makes windows into men’s souls through which to pass judgements about life and death.

Da’esh ideologues are postmodern and millenarian fuqaha’ al mafahim: magicians of meaning. They learned this bricolage from the Muslim Brotherhood, from al-Jihad, from al-Gama’at al-Islamiyya, from al-Banna, Maududi, Qutb, Azzam, al-Maqdisi. And they display it in their lived paradoxes: a state that is not a state, the godly who murder and steal at will, self-proclaimed champions of women who keep women as sexual chattels, instruments of the divine who practise a theology of rape, paladins of the oppressed who oppress in plain view, advocates of God’s freedom who enslave for the sake of true liberation.

This is why, in a world of epistemological instability, at least where religion is concerned you need new and credible forms of canonical authority to prevent others commanding the hermeneutical heights. This matters in a region where faith informs and shapes the politics and sociology of power and legitimacy. If there is such a thing as the classic Islamic state, it recognises a functional difference between the religious and the secular – between ibadah (worship) and mu‘amalat (transactional relationships). That is the often unspoken basis for the entire modern state system in the Arab world. And it is the only basis for the future benign exercise of legitimate power. To put it more brutally, if you think you can hear the voice of God, see a doctor.

Economic success helps. And Da’esh and its analogues undoubtedly emerge from a particular socio-political moment. But materialism is not enough on its own. Any solution will be built on the persuasive articulation of a new relationship between God and the world, religion and political praxis, the state and the individual, the individual and the group, the group and society. Da’esh poses this challenge in the most acute and demanding way that we have so far seen. The only way it can be met successfully is through the existing state system in the region. Anything else is an illusion.

There is another difficulty. The choice that is increasingly posed, and that we see driving wedges between the Sunni states of the Middle East and more widely, is about the relative seriousness of the threat from the Islamic Republic of Iran (whose rulers are heirs to the only successful Islamist state seizure in the Middle East) or from sacralised Sunni extremism, and the sequencing of a response. It is true that we cannot combat the threat of the latter without doing something about the former, quite aside from how ever the nuclear deal turns out. But this is not a binary choice. We need to address both issues comprehensively together.

***

That is precisely what we have not done since at least 2009, when coalition forces in Iraq last intensively targeted both Sunni and Shia bad guys. For me, the great failure of Iraq was our failure to sustain attention. We in the UK wanted our troops back home almost as soon as we had sent them in. The US-led surge of 2007 to 2008 was a classic counterinsurgency strategy, designed to buy time for politics to kick in.

At the same time the coalition, through the vision and determination of Ryan Crocker and David Petraeus and the efforts of Generals McChrystal, McMaster and Lamb, had begun to reel Sunni militants back into the realm of politics through the so-called Iraqi Sahwa. But then we switched off. We let Iraqi politics become resectarianised and we abandoned those Sunnis who had trusted in us to reconstitute an inclusive Iraqi state. We pretended that order would emerge from disorder without any intervention. We live now with the consequences of that folly. Da’esh simply learned the lessons of the conflict much better than we did. And for us now to turn away and pretend it is nothing to do with us is shameful.

Perhaps the tide is turning. More attention is being paid to strengthening the defence of the Gulf Co-operation Council states, particularly against missile attack, and indeed the Turkish border with Syria. There is more support going to the Iraqi security forces from the US and the UK. A more coherent anti-Da’esh coalition may be emerging. The Turkish government has announced it will let the US use Incirlik Air Base for operations against Da’esh and other extremist groups in Syria. It has also launched strikes against what it says are Da’esh targets inside Syria – and more against the Kurdish PKK. It has cracked down on alleged Da’esh supporters and businesses inside Turkey. And there is talk (which may be just that) of carving out safe territory for the non-Islamist opposition to hold around Aleppo, the great symbol of what Syria was and could be again.

Yet even if miracles happen and all this comes to pass, we need far more. First, we need to remember who our real enemies are. Second, we need a strategy that combines much more co-ordinated intelligence, cyber, interdiction and military action against Da’esh and Jabhat al-Nusra, with similar action against Assad’s forces, who have killed far more people. Whatever the truth about the supplies of military kit to individual groups, inserting a handful of fighters into hostile territory without adequate protection doesn’t really cut it.

This needs to be associated with a new, sustained effort to identify the right political exit: with Assad in recent weeks publicly acknowledging the strain of holding territory, we should think even harder about what a future Syria should look like. We need to stand up an equitable central state in Iraq free from external interference. Prime Minister Haider al-Abadi is trying to do the right thing but many still don’t have electricity, the displaced people can’t return home, protests continue and Shia militias call the shots. We need more effective humanitarian support for the three million Syrian refugees, whose presence threatens in the long run to destabilise Lebanon and Jordan and cause complications for Turkey, as well as the nearly ten million people internally displaced across Iraq and Syria. We need to recognise that Libya is not a sideshow: in many ways it is a fallback location for Da’esh if it loses ground in the Levant. That the problem has not been addressed is a reproach to a divided Europe, which refuses to equip itself to act forcefully even in its own near abroad. We need to do more to peel away non-Islamists in Misurata (for example) from Islamists and support the Tobruk government more decisively. Above all, we need to push back against Iranian proxies in Lebanon, Iraq, Syria and Yemen.

We will never win a battle for Sunni hearts and minds with Shia militias – however effective they claim to be in combat. Da’esh has lost ground recently. But between 2005 and 2009 it was Sunnis who rolled back a Sunni insurgency. Their absence now merely guarantees a prolonging of the agony. And this will make it difficult to enlist the leading Sunni states of the region as allies in this fight (as we are seeing already). We also need to guard against thinking that my enemy’s Sunni enemy is also my friend. Any policy that supposes that Jabhat al-Nusra in Syria, Ansar al-Sharia anywhere, loose affiliates of al-Qaeda in Yemen or the Shura Council of Mujahedin in Derna are appropriate allies in this fight is insane. In the end, these groups represent differing forms of the same ideological deformation as Da’esh. That challenge, in the end, can only be met by stable states, lawful governments and open societies, but these will not emerge overnight, any more than they did in Europe. And in the Middle East beyond the Arab-Israeli conflict – which, as the region turns on its axis and previously unspoken alliances take visible shape, I still think is in urgent need of renewed creative approaches – outsiders have a very limited role in helping them do so.

What we can help provide is a greater sense of collective endeavour and security. Without skin in the game, we have no role. Without security there is no stability. Without stability there is no shared political space. That space may be in short supply everywhere. But it won’t emerge out of civil war, sectarian confrontation, chaos or a sense of abandonment.

John Jenkins is a former British ambassador to Saudi Arabia, Libya, Iraq, Syria and Burma. He also served as consul general in Jerusalem, as director for the Middle East and North Africa at the Foreign Office in London and with British diplomatic missions in Abu Dhabi, Kuwait and Malaysia. He is now executive director (Middle East) for the International Institute for Strategic Studies, and is based in Bahrain.