For Qassem Suleimani, the 62-year-old head of Iran’s Quds Force, and one of the most powerful men in the Middle East, the short flight from Damascus to Baghdad late on Thursday was routine. Travelling by private charter, the architect of Tehran’s strategic efforts from Lebanon through Syria and Iraq to Yemen would have avoided all formalities at one of the world’s most securely guarded airports.

As his plane touched down just after midnight on Friday morning after reportedly disappearing from commercial flight trackers shortly before landing, two cars were waiting to greet the burly general at the aircraft steps.

On the tarmac was a familiar face, Suleimani’s long-time associate Abu Mahdi al-Muhandis, the Iraqi leader of the Iranian-backed Shia militia Kata’ib Hezbollah, whose supporters had laid siege to the US embassy in Baghdad for two days last week in retaliation for US airstrikes that killed 25 militia fighters on 29 December.

Muhandis and Suleimani had urgent matters to discuss – not least their joint desire to pile pressure on the US. Despite Muhandis ordering his militia away from the American embassy in Baghdad’s fortified Green Zone, they had been targeted by the Americans in response to a series of attacks on bases where US troops were present.

Suleimani also had pressing issues on his mind. According to US officials, he had flown to Baghdad concerned about escalating anti-government protests in Iraq, some of them fuelled by anti-Iranian sentiment, and anxious to canvass further assistance from Muhandis in countering them. Growing numbers of Iraqis have become disillusioned by a failing economy and ballooning unemployment, and angered by the power wielded in their country by Tehran.

Climbing into the first car with Muhandis, with their guards following in the second, Suleimani and his small convoy sped towards an access road by the far end of the runway.

What the cars’ occupants did not know was that details of Suleimani’s visit had been leaked days before. They were being observed by an American MQ-9 Reaper drone transmitting their movements to the Pentagon and CIA.

As the SUVs turned to skirt the airport’s low perimeter wall topped with barbed wire, at a safe distance from other travellers and Iraqi officials, the drone fired two missiles, slamming into the first car carrying Suleimani and Muhandis, buckling the metal and setting it on fire.

A third missile followed quickly, striking the second car carrying the bodyguards. The impacts were as devastating as they were lethal. While Muhandis appears to have been vaporised on contact, Suleimani’s hand, covered in grey ash, was pictured dangling through a window, identifiable by the ruby ring he always wore.

Play Video 0:35 Qassem Suleimani: moment Iranian general killed by US strike reportedly caught on CCTV – video

The assassination of Suleimani, ordered by President Donald Trump from his Florida golf resort, took only an instant, but it is a moment that now threatens radically to transform the volatile politics of the Middle East and challenge the basis of America’s own engagement there.

And as so often with Trump, it was an act that was deeply contradictory, flying in the face of everything the president had said before, not least his repeatedly expressed desire to exit what he has called “the stupid endless Mideast wars”. Far from promising a quick way out of the “stupid wars”, many experts agree, Trump’s decision to kill Suleimani seems to presage only a deeper and more violent entanglement.

Confronted with the questions “why?” and “why now?”, many observers have struggled to move beyond both the enormity of the event and fear of what many believe will be an inevitable blowback.

“It is hard to overstate the significance,” remarked David Petraeus, former head of the CIA and a former commander in Iraq, adding grimly: “There will be responses in Iraq and likely Syria and the region.”

A second former acting head of the CIA from the Obama era, Michael Morell, was more brutally forthright. “There will be dead Americans, dead civilian Americans, as a result of this,” he said, adding for good measure: “I think we’ve now ended any hope of keeping Iraq out of Iran’s arms.”

What few have any doubt about, however, is that Suleimani’s assassination, an act that the George W Bush and Obama administrations both eschewed, has escalated the conflict between Tehran and Washington that has intensified since Trump withdrew from the Obama-era deal to curb Tehran’s alleged nuclear weapons programme, and instead ratcheted up sanctions in the policy known as “maximum pressure.”

By one reading, the US drone strike was the culmination of a deadly cycle of events that began, largely unnoticed at first, on 27 December more than 250km north of Baghdad on the sprawling K1 air base outside the oil-rich city of Kirkuk, a centre for the fight against the remnants of Isis still operating in the western desert.

At 7.30 in the evening, a barrage of 31 107mm Katyusha rockets, fired from a lorry, slammed into a building housing Iraqi federal police and the counterterrorism service. The Iraqis, however, were not the target. Instead, the missiles were aimed at an area housing around 100 US and other foreign trainers, killing one American civilian interpreter and injuring several more.

In the immediate aftermath it was at first suspected that remnants of Isis had launched the attack. But within hours US officials, including Mike Pompeo, the secretary of state, were pointing to another suspect – the Kata’ib Hezbollah militia, set up by Muhandis.

The officials pointed to a recent pattern of rocket attacks they attributed to Iranian-backed militias in Iraq. Pompeo warned that any further attacks “will be answered with a decisive US response”.

But a response to the K1 attack, as it would turn out, was already well in train. The Pentagon authorised airstrikes on several Kata’ib Hezbollah bases identified as housing the perpetrators of the attack, killing 25 and wounding dozens more. Suddenly, a conflict which had been simmering largely under the radar was thrust into view.

On New Year’s Eve it was the turn of the militia to respond. Summoning supporters to the vast US embassy in Baghdad, they stormed the compound’s reception, sacking it and setting it on fire.

Play Video 0:42 Iraqi militia supporters storm US embassy in Baghdad – video report

Three days later Suleimani flew into Baghdad, and to his death, with the US alleging he had travelled there to orchestrate new attacks on its forces with his right-hand man in Iraq, Muhandis.

The manner of Suleimani’s death at the hands of a US foe he had for so long opposed, often violently, has added a new layer of complexity to the legacy of a figure whose life story was full of bloody contradictions.

A key figure in the grotesque slaughter of the Syrian war, where he helped prop up the murderous regime of Bashar al-Assad, Suleimani was also instrumental in the fight against Isis in Iraq, earning him the admiration of many Shia there. And while some saw him as the Middle East’s malign puppet-master, he saw himself as a loyal servant of Iran’s supreme leader Ayatollah Ali Khamenei. Either way, he had become an almost mythical figure in the region, a far cry from his humble roots.

Born in impoverished circumstances, and with little formal education, his school was the Islamic Revolutionary Guards Corps where he became commander of the Quds Force in 1998, tasked with expanding Iran’s influence through covert operations and through ties with the network of proxies he developed from Lebanon to Yemen.

His skills as a tactician earned him a grudging respect even in senior US military circles.

One telling assessment was supplied last year by the American general who once contemplated killing him, Stanley McChrystal, writing in Foreign Policy. In almost admiring terms, the former commander in Iraq painted Suleimani as a “confident, proven leader”, “a calculating and practical strategist” and a “shrewd pragmatist”.

For Vali Nasr, however, the Iranian-American academic and author who served as dean of the Johns Hopkins School of Advanced International Studies, Suleimani’s reputation was as much a creation of Washington as it was of Iran.

“The US built him since 2008 so he was the only military commander who was known in US foreign policy circles,” he told the Observer, adding that there were other figures active in prosecuting Iran’s strategy while Suleimani occupied a role similar to the role of US Central Command.

Facebook Twitter Pinterest Donald Trump at an Evangelicals for Trump event in Miami on 3 January. Photograph: Tom Brenner/Reuters

“Although he was very capable in many ways, he was seen as this one-man army responsible for everything happening in Iraq and Syria, which wasn’t accurate. Now he’s dead, the assumptions of the Trump administration appear completely naive in terms of what to expect, not least the comments in Pompeo’s talking points saying that Iraqis are happy that they’ve been liberated from him.

“People are more complicated than that. Iraqis can be simultaneously unhappy over Iran’s meddling but still have a warm view of Qassem Suleimani who they credit with defending the south against Isis and who was the one who showed up in Erbil with enormous amounts of weaponry to provide defence for the Kurds.” And far from strengthening US influence in Iraq, argues Nasr, the assassination may have “pulverised” US relations with Iraq’s government. “The momentum is now with the hardliners to push the US out.”

Ali Vaez, an Iran expert with Crisis Group, is also sceptical of the assumptions in the Trump administration over the hoped-for outcomes from Suleimani’s killing.

“The biggest problem here,” he told the Observer, “is that the Americans believe their own rhetoric. The fact that the US designated the Islamic Revolutionary Guards Corps [of which Suleimani’s Quds Force was a key part] as a terrorist organisation doesn’t mean that it functions like a terrorist organisation.

“It doesn’t mean if you decapitate it, it will be paralysed. It’s a state institution of the armed forces of Iran. I think he will be difficult to replace but not impossible. The Quds Force has lots of people with similar profiles with a similar degree of experience and expertise who would be able to push forward Iran’s regional policy.

Vaez cautions too: “It’s often the case that the man standing behind steps in, even if he lacks the vision and tactical skills, and risks being more of a hardliner, which will only exacerbate the stresses.”

If the events of the last week appear at first surprising, all they have really done is highlight a competition for influence in Iraq that has festered since the moment US-led forces deposed Saddam in 2003, a conflict in which Suleimani has long been a key player.

Play Video 2:31 'Death to America': grief and fury as thousands mourn death of Qassem Suleimani – video report

While the Bush administration had glibly assumed the victors’ spoils would fall to the US in what it saw hubristically as an opportunity to reshape the Middle East, Tehran also saw an opportunity to influence its old neighbour and foe with the replacement of the Sunni-dominated Baath party by its Shia co-religionists.

Among exiles returning after the war were those who had sought refuge in Iran in the Saddam era, including veterans of the Badr Corps – Iraqis who had sided with Iran in the conflict’s brutal trench warfare that, with its use of poison gas and massed attacks, seemed like a rerun of the battles on the Western Front.

One of those key figures was Muhandis, born in Basra to an Iraqi father and Iranian mother, who had fled Iraq in the Saddam era and had risen through the ranks in Iran to become a senior commander.

Even as the US forces were battling an insurgency by Sunnis drawn from the old Baath party ranks and a new iteration of al-Qaida, Shia militias, many taking guidance from Iran and Suleimani, were pursuing their own agenda to undermine the US-led vision for Iraq, a policy that would see the lives of some 700 US servicemen laid at Suleimani’s door.

Facebook Twitter Pinterest Iranians take to the streets as they mourn the death of Qassem Soleimani. Photograph: UPI/Barcroft Media

If that tension was put on the back burner during the devastating rise of Isis in Iraq, when Suleimani and the US would be fighting a common enemy, all that changed with the election of Trump and his abrupt withdrawal from the Iran nuclear deal. As Trump formed an alliance with Saudi Arabia against Tehran, the scene was set for the escalation of a series of proxy wars across the region, one that in recent months has seen the Kata’ib Hezbollah militia targeting US interests in Iraq.

According to a Reuters report on Saturday, that involved a meeting in October in a Baghdad villa where Suleimani told militia commanders under the direction of Muhandis to form a new group of low-profile paramilitaries – unknown to the United States – who could carry out rocket attacks on Americans housed at Iraqi military bases.

And despite the celebration of Trump and his allies, for some experts the killing of Suleimani smacked not of strength, but of weakness.

“President Trump’s decision to escalate hostilities by killing Qassem Suleimani epitomises the failure of his strategy for dealing with Iran,” said David Schanzer, an expert on terrorism at Duke University.

“Trump abandoned diplomacy as a means to constrain Iran by backing out of the nuclear accord in 2018, opting instead to use punitive economic sanctions to pressure Iran to end its aggressive behaviour. The result? Iran is now advancing its nuclear programme, acting with greater aggression in the region, and fomenting greater violence inside neighbouring Iraq.”

All of which raises the question of why Trump took what may be the biggest gamble of his presidency?

At Mar-a-Lago, Trump’s pink-walled Florida resort where the president had retreated from Washington to celebrate the holidays, there was no hint early last week of what was being contemplated.

Even on Tuesday evening in the aftermath of the violent attack on the US embassy in Baghdad, Trump’s messaging on Iran, and its allied militias in Iraq, was typically muddled as he mused on the prospect of war with Iran. “I don’t think that would be a good idea for Iran. It wouldn’t last very long,” Trump said as he warned of the consequences of further attacks. “Do I want to [go to war]? No. I want to have peace. I like peace.”

Reporting in the US media suggests, however, that behind the scenes Trump and his advisers were already heavily engaged in drawing up a response to the K1 attack, including the retaliatory airstrikes, when the president was alerted that US intelligence agencies had learned that Suleimani was flying to Baghdad from Damascus.

And on the agenda since at least last Sunday, even before the violent demonstrations at the US embassy, was the proposed assassination of Suleimani, a notion, some suggest, that may have first been mooted as long ago as early summer.

According to the Washington Post, Trump was prodded by officials who reminded him that he had failed to respond to previous attacks blamed on Iran, including the mining of oil tankers in the Gulf, the shooting down of a US surveillance drone and the rocket attack on a Saudi oil facility.

“The argument is, if you don’t ever respond to them, they think they can get by with anything,” one White House official told the paper.

When it came to the crunch, the moment of decision was oddly banal. On Thursday, just hours before the assassination, Trump played a round of golf, then later huddled with his campaign team to review plans for this year’s election, leaving the meeting for a few minutes at around 5pm, the moment, some believe, when he delivered the “kill order”.

And after the fact there has been a slew of new rationalisations.

In a conference call with US media on Friday, the national security adviser, Robert C O’Brien, described the strike as “aimed at disrupting ongoing attacks that were being planned by Suleimani and deterring future Iranian attacks through their proxies or through the … Quds Force directly against Americans,” while Trump himself suggested the killing was intended to prevent a war, not provoke one.

That may be a moot point.

By Saturday, as thousands of mourners for Suleimani marched through Baghdad chanting anti-American slogans, what was certain, even as Washington deployed 3,000 fresh troops to the Middle East, is that the next move will be decided in Tehran.

Every major newspaper and state-controlled TV broadcast has focused on Suleimani’s death, with even reformist newspapers such as Aftab-e Yazd warning of revenge, and new billboards carrying the same message.

“His work and path will not cease, and severe revenge awaits those criminals who have tainted their filthy hands with his blood and the blood of the other martyrs,” Ayatollah Ali Khamenei said, while the defence minister, Amir Hatami, promised a “crushing” response.

Facebook Twitter Pinterest Abu Mahdi al-Muhandis at a funeral for his fighters killed in US airstrikes retaliating for attacks by his militia. Photograph: Thaier Al-Sudani/Reuters

The only question is the timing, with many anticipating that there will be careful deliberations among Iran’s leaders before any response.

“Look, this is a pragmatic regime not a suicidal one,” said Vaez, “and the pragmatism emanates from the very top. That pragmatism is not going to change but Iran will have to respond to this otherwise it will be seen as weakness inviting more US aggression. But retaliation is inevitable even if the risk of a miscalculation is very high.”

The key calculation in Tehran, argued Vaez, is predicting Washington’s response and whether a “massive retaliation” might force a withdrawal similar to the one after the 1983 bombing of a US marines barracks in Beirut that claimed 241 lives, or whether a more proportionate response – perhaps targeting a senior US official for assassination – is a safer course.

And back in the US not everyone is buying the official explanations for the killing. “[Suleimani] has been killing Americans in Iraq since 2003,” Jon Soltz, chairman of VoteVets.org and an Iraq war veteran told the New York Times. “I was in one of his attacks in Taji in 2011. They were dropping 240mm rockets on us. So this is not a surprise that he’s involved in killing Americans.

“But the question is what was different last night? The onus is on Trump to prove something was different, or this is no different than another weapons of mass destruction play.”