It seems such a long time ago: the brazen defiance of Abu Bakr al-Baghdadi on the pulpit of the Mosul Grand Mosque on 4 July 2014, declaring a caliphate; military parades through the streets of Raqqa; the slick videos; the flood of foreign fighters and Muslim sympathisers into Syria and Iraq, the beheading of British and American hostages. Across the two major cities that Islamic State (IS) once controlled – Mosul in Iraq and Raqqa, its de facto capital, in Syria – all that remains today is mostly rubble and ruin.

For a period, it seemed inconceivable – even impossible – to imagine how IS would be overcome. At its height, the terror group was believed to have almost half as many soldiers as the British army, controlled a territory larger than the size of the United Kingdom, and had access to sophisticated weaponry that had been seized from Iraqi forces after they were routed in the campaign to take Mosul.

As IS asserted itself across large parts of the Levant, neither the Syrian Arab Army nor the Iraqi Armed Forces could contain its expansion. The group’s fighters proved adept at gaining and holding territory. Had they been content merely with fighting in the region, it is likely that their proto-state would still exist. That much was clear from the actions of Barack Obama, whose administration was adamant that the US should not become embroiled in another Middle East war.

Yet none of this satisfied Islamic State. It continued to provoke and enrage by beheading more Western hostages, launching terrorist attacks inside France and generally causing such revulsion that demands to take action against it eventually grew overwhelming, even for a reluctant American administration.

To outsiders, the strategic calculations of IS made little sense. Why couldn’t it limit itself to the territory seized and held in Syria and Iraq, consolidating its state-building project?

The answer is: IS was more than a putative state. It was a movement that had to be examined with reference to its own eschatology. When understood, this revealed the actions of rational ideologues. From the moment the group’s fighters erupted into Syria and then smashed through the border with Iraq, they believed they were the agents of a divine plan. The plan was contradictory because the ultimate triumph of Islamic State would, they believed, result in the end of the world. Islamic State was territorially expansionist: its ambition was for “the word of Allah” to reign supreme. But this temporality existed only to serve spiritual ends. IS believed that the revival of the caliphate had been prophesied and that once it returned it would, unlike its predecessor, succeed in global conquest. This would, in turn, hasten the end of days or the so-called day of judgement. In this respect, IS was buoyed by scripture: the Prophet Muhammad predicted a future in which a revived caliphate would conquer the world, shortly before the end of time.

For IS the revival of the caliphate marked the end of history; a historical ground zero. And it was intent on destroying much of Syria and Iraq’s cultural heritage. The vandalism in Palmyra has been well documented, but other historic sites such as Hatra in Iraq were also destroyed, along with scores of churches and monasteries from antiquity. Even mosques and Sufi shrines that did not conform to IS’s narrow doctrinal interpretation of Islam were bulldozed.

****

Islamic State leaders – such as the firebrand cleric Turki al-Bin’ali – believed that they were living through the last days of time. Prophecy had predicted they would win, and continue to win. It was an intoxicating period for IS fighters, who seemed to be securing one improbable victory after another: God had to be on their side. “[Allah] was honest with his promise, supported his soldiers, and defeated the [enemy] parties alone,” Baghdadi told the congregants of the Mosul Grand Mosque in July 2014, when he declared the establishment of the caliphate.

The extent to which this belief permeated through the group was evident when IS began its morbid ritual of beheading one Western hostage every fortnight. The beheadings were carried out in remote, undisclosed locations. The exception was that of a 26-year-old American called Abdul-Rahman Kassig (also known as Peter Kassig), whose execution video was staged in Dabiq, in Syria, in November 2014. Kassig had previously served as a US army ranger in Iraq before entering Syria as a civilian aid worker.

Before the IS assassins turned to the American, they featured a line-up of several Syrian army soldiers who were also killed. Mohammed Emwazi, the British executioner known as “Jihadi John”, started by addressing some prepared remarks to President Obama. “To Obama, the dog of Rome, today we are slaughtering the soldiers of Bashar and tomorrow we will be slaughtering your soldiers,” he said. “And with Allah’s permission we will break this last and final crusade.”

The “crusade” Emwazi was referring to was the American-led coalition of forces against Islamic State. The greater significance, however, was that this sequence of beheadings was conducted in Dabiq, a town in northern Syria close to the Turkish border. A prophecy found in the hadith – the recorded traditions and sayings of Muhammad, constituting a body of law alongside the Quran in normative Islam – suggests that a decisive end-of-days confrontation between Muslims and their enemies will occur in Dabiq, resulting in the final triumph of Islam before the caliphate marches on to global dominance.

That conviction informs the next sequence of the video in which Emwazi hovers over Kassig’s severed head while still addressing the US president. “We also remind you of the haunting words that our sheikh Abu Musab al-Zarqawi [the leader of al-Qaeda in Iraq, 2003-2006] told you: ‘The fire has been lit here in Iraq and its heat will continue to intensify by Allah’s permission until it burns the crusader army in Dabiq.’” Emwazi continued: “And here we are, burying the first American crusader in Dabiq, eagerly waiting for the rest of your armies to arrive.”

In the event, when the time came, Islamic State retreated from Dabiq without much resistance. It was widely ridiculed for seemingly “postponing” the apocalypse it had craved.

IS leaders desperately needed the West to enter the conflict in Syria, believing this would provide the stepping stone towards its ultimate victory. Like Homer’s myth of the Sirens, the allure of what might come next proved so irresistible to Islamic State that it created the conditions that destroyed the caliphate. With the exception of a long, violent last stand in Mosul, IS forces melted away with relative ease in many of the areas they had controlled.

Recovered documents revealed both an awareness of and resignation to their fate. There was no way IS could challenge the coalition of forces that dominated the skies above it and which created pathways for local forces on the ground to recapture territory. Indeed, these recovered documents revealed sophisticated battlefield understanding. Fighters were told not to fire unnecessarily on advancing forces because this would reveal which buildings they were using as shelters.

Once their position had been discovered, the other side would call in an air strike and wait. Instead, fighters were told to pick off those they could with certainty but, when the end looked inevitable, to retreat and live for another day.

That was a lesson IS learned early in the war. Whenever the group came under pressure, as it did in February 2014 in the northern Syrian town of Azaz, it generally demonstrated a willingness to withdraw and retreat from lost causes.

The only notable exception to this was Kobane, a Kurdish town in northern Syria under YPG (the majority-Kurdish People’s Protection Units) control since 2012. There IS launched a futile and costly assault in September 2014. American military officials estimate that the group lost as many as 2,000 soldiers in the campaign – among them, several prominent British fighters, including 19-year-old Mehdi Hassan from Portsmouth.

It is impossible to say how many members of IS are alive today, although most credible estimates suggest that its strength can still be counted in the thousands. Having retreated from the major cities, IS has now melted into the expanses of the Syrian desert and is believed to be regrouping in towns along the Syrian-Iraqi border. That is an alarming assessment, because when al-Qaeda in Iraq was declared defeated by US officials in 2007, its membership was believed to be in the hundreds. But it was that groupuscule that spawned what later became Islamic State.

****

As the caliphate collapsed, scores of its members were arrested and detained by the Syrian Democratic Forces (SDF) – a primarily Kurdish-led group. Among the detainees are two former Britons from London who have since been stripped of their citizenship, Alexanda Kotey and El Shafee Elsheikh. Both were among the so-called Beatles, a group of four British IS members (also including Emwazi) who are said to have been responsible for the detention of Western hostages. Although the American and British captives were killed, others held by the men were later freed after their governments negotiated their release. Those detainees have revealed a regime of sadistic torture by the Beatles, including random beatings, mock executions and waterboarding.

American authorities want Kotey and Elsheikh to be repatriated to the UK to stand trial for their crimes. When Theresa May was home secretary, however, she expanded powers given to the government originally in the British Nationality Act 1981, whereby dual nationals can be stripped of their citizenship. Now, it was held, this could apply even to those who could reasonably be considered eligible for another nationality; it still cannot be used against those who would end up stateless. These powers had been exercised sparingly by previous home secretaries, but May used them as one of the government’s main weapons for mitigating the threat from Britons who had joined IS.

There is a significant impasse with the Americans. Kotey and Elsheikh are in effect stateless, with nowhere to go – even if technically they are entitled to citizenship elsewhere. The longer they remain in Syria, the greater chance there is that these men might escape or be released back on to the battlefield.

American authorities are deeply concerned about this but are also, understandably, reluctant to transfer them formally into US custody. Should they do so, they believe, it would open the floodgates for other countries to refuse responsibility for their nationals.

The saga has angered Donald Trump’s White House: an administration that has decided to put “America First”, in part by redressing the lopsidedness of the US’s financial commitments around the world. Attorney general Jeff Sessions told a Congressional panel: “I have been disappointed, frankly, that the British… are not willing to try these cases but pretend to tell us how to try them.”

There are numerous cases of Westerners who fled the crumbling caliphate and now find themselves stuck in a legal quagmire. After the excesses of Guantanamo Bay and the scandal of extraordinary rendition, there is caution in Western capitals, as the various parties involved work out how to deal with their citizens held by the SDF. Few would sympathise with IS members, but there are also many children being detained alongside their parents. It is unclear how many foreign fighters escaped from Raqqa when IS made a tactical retreat from the city in October 2017, although there is evidence that many got out; a prominent German member of the terror group, Denis Cuspert, was killed by SDF forces in January in Syria’s Deir ez-Zor province to the south.

****

The firefight that killed Cuspert revealed something about the next incarnation of Islamic State. Beaten in the major urban centres that it once held, it has reverted to its insurgent roots, retreating to the deserts along the Syrian and Iraqi border. This is where al-Qaeda regrouped after being “defeated” by coalition forces in the years after the 2003 invasion of Iraq. Then, as now, fighters simply waited for more favourable circumstances.

This points to a much more desperate situation on the ground, where all of the local conditions that first gave rise to IS not only remain but have been exacerbated by the actions of the Western coalition. The indiscriminate nature of the bombing campaign on Mosul and Raqqa was the equivalent of a modern-day blitzkrieg, in which jets pummelled targets from the air, creating pathways for troops to move in below. Eyewitness accounts of what transpired in the two cities attest to febrile assaults. There is no sentimentality for jihadi fighters caught up in this campaign, but some broader considerations are necessary.

It is an inconvenient truth that the civilian populations of both Mosul and Raqqa grew under IS rule. This is not to suggest that people were sympathetic to the nascent state – most were not – but they nonetheless made pragmatic decisions based on local realities.

Many had been forced previously to leave their homes because of the instability and insecurity caused by years of war. Life under IS may have been brutal and barbaric, but, for a short while, a semblance of order was restored to the territories it governed. For the vulnerable Sunni Arab poor, it was an opportunity to regain control of their lives. In the campaign to recover these territories many civilians died. The independent monitoring group Airwars estimates that “a minimum of 6,259 to 9,604 civilians are likely to have died in coalition actions”.

During its final weeks in Raqqa, Islamic State publicised these casualties, releasing graphic videos of injured children after coalition bombing raids.

An Australian foreign fighter who served as a doctor in Raqqa, Tareq Kamleh, stood over a badly burned girl in one video, asking: “Is this one of the terrorists that America’s wanting to hit?” The video ends with reports that the child died from her injuries. Kamleh is another of the foreign fighters who is believed to have escaped from Raqqa.

Although much of the IS senior leadership is dead, Abu Bakr al-Baghdadi is alive and remains in control. Shortly after the group lost Raqqa, he released an audio message in which he spoke about people’s insecurity and civilian casualties. It is among the communities who have had their homes destroyed and friends and family killed that IS will seek to recruit members and rebuild support for its shattered caliphate.

Beyond the Middle East, Islamic State affiliates claimed responsibility for attacks last month in Canada, Tajikistan and the Philippines; the last of which, a suicide bombing on 31 July, killed ten people. More significantly, this attack was conducted by a Moroccan, suggesting that IS affiliates are also able to attract foreign fighters. The Filipino authorities arrested a Spanish man in January who was on his way to join IS in Basilan, located in the troubled Mindanao province of the southern Philippines. Groups aligned to Islamic State in and around the Sahel in Africa are also gaining strength.

The threat to European towns and cities remains acute. In recent months, authorities in Germany and France have thwarted separate terrorist plots involving the use of ricin – a highly toxic poison. These discoveries reflect an ongoing fear that IS will eventually succeed in launching a chemical or biological attack in a European city. After the ricin plot in Cologne was uncovered, a source close to the investigation told the Bild newspaper that it represented “the biggest potential threat ever found in Europe”.

Attacks abroad allow Islamic State to project power and remind the world of its potential; however, what the group craves above all is a return to territorial prowess. And the underlying issues that led to its dramatic rise – political instability, weak civic institutions, sectarianism, injustice and grievance – have only been exacerbated by the privations of the crises in Syria and Iraq.

For the average citizen of these countries who simply wants to live an ordinary life, prospects for the future are bleak. The heat from the fires of Islamic State may have been cooled, but favourable winds are still fanning the flames of fanaticism.

Shiraz Maher is a New Statesman contributing writer and director of the International Centre for the Study of Radicalisation at King’s College London