One evening at the end of March, a Syrian rebel leader returned from a meeting across the border in Turkey and called an urgent gathering of his commanders. The five men turned up at their boss’s house in Idlib province expecting to receive the same pleas for patience that they had always heard and more grim news about cash and weapons being hard to find. This time, though, they were in for a shock.

“He arrived looking eager,” said one of the commanders. “That caught our attention straight away. But when he started to speak, we were all stunned.”

The leader, who asked that his unit not be identified, said he told his men that the grinding war of attrition they had fought against the Syrian government since early 2012 was about to turn in their favour.

“And the reason for that was that I could now get nearly all the weapons I wanted,” he told the Observer. “For the first time they were not holding anything from us – except anti-aircraft missiles. The Turks and their friends wanted this over with.”

The leader says he explained that they and every other opposition group in the north, with the exception of Islamic State (Isis), were about to be beneficiaries of a detente between regional powers who had agreed to put their own rivalries aside and focus on a common enemy – the Syrian regime.

The agreement had been secured by Saudi Arabia, which had resolved to do all it could to end the Bashar al-Assad regime and, more important, to quash the ambitions of Assad’s main backer, Iran, to control the course of the war. It signified a new phase in an age-old tussle between regional rivals for power and influence that was to have profound ramifications for the way the war in Syria, and proxy standoffs elsewhere in the Middle East, were to be fought.

In early March, senior regional figures had been summoned to Riyadh by the newly crowned King Salman to hear his plans for the region. The Turkish president, Recep Tayyip Erdogan, was one of the first to arrive. Qatari officials and Gulf Co-operation Council leaders soon followed.

His message was threefold: first, there was to be no more division along regional lines, which had seen the Muslim Brotherhood-aligned governments of Turkey and Qatar pour support into allied Syrian groups, while Saudi focused on more mainstream outfits. Second, Riyadh would agree to send gamechanging weaponry to northern Syria in return for guarantees of coordination and discipline. And, finally, the US would not stand in the way. “Quite frankly,” a Saudi official told the Observer, “it would not have bothered us if they had tried to.”

Within weeks, the new push had paid clear dividends. Armed with dozens of guided TOW anti-tank missiles, which could take out regime armour from several miles away, opposition groups – among them al-Qaida’s affiliate in Syria, Jabhat al-Nusra, which is proscribed by the US as a terror group and has long been viewed warily by Riyadh – started advancing into towns and cities that they had not dared to attack until then.

The results were shocking. The regional capital of Idlib fell within days. Several weeks later, the nearby town of Jisr al-Shughour also fell to an amalgam of jihadists and moderates who had kept their end of the bargain.

Now, the agricultural plains that stretch towards Syria’s third and fourth cities, Homs and Hama, appear more vulnerable than at any time since mid-2012. So does the Mediterranean coast, and the mountains to the north, which are the heartland of the Alawite sect, dominant in Syria’s political and security establishment. And to the east, Aleppo, which had been under serious threat of being encircled by Assad’s forces six months ago, now looks much more likely to fall to the rebels.

In Ankara and Riyadh and even Baghdad and Beirut – both nominally Assad allies – there is now a strong sense that the war is going poorly for the regime. Every battle that its military has fought since March has ended in a rout, including the brief fight for two gas fields north of Palmyra last week, which fell to the jihadist group Isis, along with Palmyra itself – home to some of the world’s most important archeological sites.

Part of the ancient city of Palmyra. Photograph: CHRISTOPHE CHARON/AFP/Getty Images

Damascus sent some of its elite troops to defend the gas fields, which are an essential part of Syria’s energy supply, but they were quickly defeated by Isis, which also ran rampant through neighbouring Iraq’s Anbar province last weekend.

“This is not just the ebb and flow of battle,” said one senior diplomat from the Arab world. “This has been clear and repeated evidence that the regime army cannot defend itself, or the country, even with the ever-heavier backing of its sponsor, Iran.”

Across the border, Iraqi officials are expressing the same fears about their own national army. Better-armed and trained than the Syrians, Iraq’s best forces capitulated in Ramadi – the most important city in Anbar province – in about 72 hours, giving Isis its biggest triumph since it stormed into Mosul in July. The terrorist group’s stated intention then was to dismantle the post-Ottoman boundaries that have defined the modern Middle East, especially the Anglo-French Sykes-Picot agreement that created modern Syria and Lebanon. One year later, Isis could hardly be criticised for sounding overly ambitious.

On Friday, the Syrian Observatory for Human Rights said Isis was now in control of all six border crossings between Iraq and Syria – even with the constant presence of US air force jets. Nearly 1,000 miles of frontier is now out of the control of either country, with Isis enjoying complete freedom of movement, except when the warplanes are around.

Though 20 of its 43 top-tier officials have been picked off by US drones and jets, Isis leaders can still travel freely across the large tract of land, roughly the size of Jordan, that they now call an Islamic caliphate. And the key to their growth does not always rest with the brutal inflexible ideology that some leaders use to bring to heel the communities they conquer.

“Before the Islamic State came along, we were the animals of the Shias,” said a surgeon speaking from the Isis-controlled Iraqi city of Fallujah. “No matter what we said or believed, we were treated as Isis anyway,” he said of the Shia-led government. “Well, we may as well be with them, because the government will never come to help. They have more power and authority than Baghdad has had since Saddam.”

The surgeon’s views were echoed by residents of Deir Azzour in eastern Syria, who were contacted by phone. “I can see the appeal of Isis,” said one man who called himself Abu Ayman. “As much as I don’t like them, I can see that they are leading some Sunni communities towards a dignity that no government will give them.”

The spectacular successes of the jihadists last week has perhaps done more to demonstrate the weakness of the state actors they are fighting than to showcase their own strength.

“This is a big problem,” said Hisham al-Hashimi, an Iraqi strategist and writer on Isis issues. “It is something that no state wants to talk about. They cannot acknowledge that they can’t find solutions by themselves. But it is very much an issue that the region is concerned about.”

As Baghdad was menaced by Isis last July, Iran was quicker than any other ally to send advisers to shore up its defences. They did not have to travel far. Nor did they need to start from scratch; Iranian generals had been doing precisely the same thing in Syria for at least two years.

One year on, Iran is more committed than ever before to securing both teetering states and safeguarding its own strategic goals in the Arab world.

The survival of the regimes in Baghdad and Damascus – or at least the same systems and structures in place now – are essential to the Iranian regime’s core goal in the Middle East: maintaining an arc of influence that runs from Tehran, through Qom, Baghdad, Najaf and Damascus and into southern Lebanon, where its proxy militia remains a potent threat to Israel’s northern border. Damascus has allowed weapons, money and fighters to flow to Hezbollah and, since 2003, Baghdad had also become an essential conduit for Iran’s ambitions, with Shia Iraqi groups who fought the occupying US army acting as paid-up proxies.

While Riyadh has watched uneasily as Iran has steadily gained a hand in Iraq since the US army fully withdrew in 2011, it has become far more alarmed as Syria has disintegrated. “That showed us how far they would go to defend their interests,” said the Saudi official. “They were not just defending what they had, but trying to build on it. And all the while under American cover.”

Throughout the Syrian war, and the Arab revolts more generally, the Saudi relationship with its most important global ally has withered significantly. King Salman, who was formerly defence minister, is known to have railed against Barack Obama’s equivocation on Syria, especially his decision not to bomb Damascus in August 2013, after a sarin gas attack blamed on the Syrian regime. “That was the moment when we realised that our most powerful friend was no longer reliable,” the official said. “We had to step out from behind the curtain.”

Within weeks of Salman being named king, a more assertive regional policy was evident. So, too, was a willingness to openly defy the US, a stance borne out of deep resentment at a deal brokered by Obama to compel Iran to surrender its nuclear programme in return for sanctions relief and, according to officials in Riyadh and the GCC, the global legitimisation of Iran 36 years after the Islamic revolution.

Gulf officials contacted in recent weeks say fears that Iran will end up playing a lead role in resolving the region’s most intractable disputes will never sit well with them, or with Riyadh. All insisted that the March takeover of Yemen by Houthis from the Shia-linked Zaydi sect, who are allied to Iran, was evidence of Tehran asserting itself ahead of the talks. And in a place where it most matters to Riyadh – Saudi Arabia’s eastern border.

“They will spend their whole treasury to protect what they have,” said the official. “They are very open about it. The Revolutionary Guards say they will never compromise on Damascus and Hezbollah and they are the most powerful institution in the country. And now they want Yemen too. But this is a ploy. They will try to trade that with us in return for Damascus.”

Out of all the recent chaos – the looming destruction of a cradle of civilisation, resurgent anti-Assad fighters and the unchecked brutality of the jihadists – a rare clarity has emerged. The region’s main powers are now openly locked in a struggle for the destiny of the region – and of Syria in particular – that can only be stopped with a globally brokered solution.

“Iran said the key to the Arab world was Syria,” said the Saudi official. “Well, we agree with them.” And Riyadh is no longer worried about showing it.