The Iranian endorsement of a plan by its arch regional rival, Saudi Arabia, to stabilise global oil prices could be seen as a diplomatic coup for Riyadh.

However, Tehran’s support for a production freeze has not been driven by a new desire for political rapprochement as much as acceptance of a greater enemy: collapsed commodity prices. The markets are flooded in crude at a time when demand is faltering due to the slowdown in expected growth from key importers such as China. Desperate times require desperate measures.

Iran and Saudi Arabia are rival oil producers but also bitter adversaries in regional politics. Short of a shooting war, tensions could hardly get any worse at this particular moment.

The Saudis severed diplomatic relations with Iran in January following a mob attack on their embassy in Tehran – a protest at the execution of a leading Shia cleric, Nimr al-Nimr, in Saudi Arabia’s eastern province.

The two countries stand on opposite sides in the wars in Syria and Yemen – their strategic competition is interlaced with vicious sectarian hostility – adding a proxy element to an already toxic mix.

Despite this unpromising geopolitical backdrop, last week there were positive moves towards an oil production deal. The price of Brent blend crude soared 7.5% on Wednesday after Bijan Zanganeh, Iran’s oil minister, came out of a two-hour meeting with some of his Opec counterparts to approve a deal hatched by Saudi with non-Opec member Russia the day before.

“Iran backs any measures which help stabilise the market and improve the price of crude oil,” said Zanganeh, although, crucially, he gave no indication as to whether Iran would curtail its own production.

Iran, which has the fourth-biggest oil reserves in the world, had previously trumpeted its desire to vastly expand its output following the lifting in January of western sanctions imposed over its nuclear programme.

It still remains unclear exactly what will happen now: the Saudis and Russians said they would hold their output at January levels, but only if other key players – including Iran – joined in.

Deals of this kind were brokered by Saudi Arabia in the 1980s only to see Russia and others renege on their commitments, forcing Riyadh to cut its output from 10 million barrels per day (bpd) to 2.5 million with little impact on prices.

Last week the price of Brent blend continued to rise towards $36 per barrel following Iran’s statement, although it fell back on Friday in the face of new US data showing stockpiles of oil were bigger than ever. This is still well ahead of the sub-$28 price seen last month but a long way off the $115 highs back in June 2014.

Oil traders and investors see last week’s developments as an increasingly hopeful sign that producers have reached their pain threshold and will do what it takes to try to force prices upwards again.

However, the wider political and religious rivalry between Saudi Arabia and Iran is unlikely to go away any time soon. Saudi control of Mecca and the hajj pilgrimage gives it legitimacy in the Sunni Muslim world, while Iran is a beacon for Shias everywhere. Arabs and Persians have long memories of prejudice, though in modern times their animosity began with the 1979 revolution and grew after the Iraq war in 2003.

More recently, Iran has deployed its own Revolutionary Guards and Lebanon’s Hezbollah, as well as Shia militiamen from Iraq, Afghanistan and Pakistan, to help keep Bashar al-Assad in power in Damascus, while the Saudis see Tehran’s covert hand behind the Houthi rebels it is bombing in impoverished Yemen. Tehran routinely lambasts Riyadh as a US stooge and a sponsor or incubator of fanatical terror.

But it was last year’s Iranian nuclear deal that really set nerves jangling in Riyadh, fuelling fears that the decades-long US-Saudi bargain of oil for security was ending as Barack Obama leant towards Tehran.

Meanwhile, the Saudi strategy over the past 18 months to pump in excess of 10 million bpd and discount its crude prices on key markets has backfired.

The kingdom, Opec’s biggest oil exporter, is said to need $100-a-barrel oil to cover its enormous public spending commitments. The International Monetary Fund warned recently that Riyadh would burn through all its financial reserves within five years under a $50 oil price unless it took steps to raise taxes or cut spending, which it has since started to do.

In recent months the Saudis have introduced tax increases, removed fuel subsidies and even suggested they are willing to privatise at least part of its state oil company, Saudi Aramco, in clear signals of their need for cash.

So far the Opec powerbroker has refused to return to its traditional role of market balancer by organising a cartel cutback in production, but the proposed deal with Russia agreed on Tuesday shows it is keen to try to break the impasse.

A rig at the Eagle Ford shale formation in Texas. Photograph: Shuli Hallak/Corbis

Iran wants better prices to help power its post-sanctions economy out of recession. As recently as 2011 it was producing 3.6 million barrels a day but is now down to 2.8 million, while exports are down to 1.1 million, half the level seen before sanctions. But it has also promised more rather than less output. Following the historic sanctions-lifting deal with the US on 16 January, Tehran said it would to pump an extra half-million barrels as soon as possible.

There is no doubting the scale of the oil price problem for producers, which started 18 months ago when traders woke up to the dual realisation that US shale production was going to continue rising and that Opec was no longer willing to reduce its output.

The stand-off was interpreted as an unspoken determination by Saudi and other oil cartel members to keep on pumping in an attempt to drive higher-cost US drillers out of business.

The strategy is certainly working – but much too slowly to avoid enormous problems building up inside the Saudi economy and in other countries almost completely dependent on oil revenues.

American shale production is now almost half a million barrels a day lower than in April 2015, according to the US Department of Energy and is forecast to fall another half million this year.

American production had been growing at 15% per annum when crude prices were racing along at over $100 per barrel but that growth has now stalled, despite attempts by US producers to keep going by cutting costs.

Latest figures from industry specialist Baker Hughes show there are now around 540 US rigs operating – 800 fewer than a year ago – while bankruptcies have risen to above numbers seen at the height of the 2008-09 banking crisis.

The shale drillers need oil prices of $70 a barrel to really thrive, according to Fadel Gheit, an oil analyst with Oppenheimer & Co, compared with around $50 for the UK North Sea and less than $6 for the likes of Saudi Arabia.

The big question now is will a production freeze take hold – and, if it does, will it not just provide a fillip for the shale drillers? Traders are nervous but hopeful and Tom Nelson, head of commodities at Investec Asset Management, believes oil prices may reach $60 a barrel by the end of 2016.

But opinion is polarised. Professor Paul Stevens, an energy expert at Chatham House, insists of the latest Saudi deal that “nothing will come of it” and predicts Brent could be anywhere between $25 and $50 by the end of 2016.

He adds: “The huge ‘fracklog’ of US shale wells that have already drilled but not completed and which can quickly be brought back into operation as prices rise mean there is a ceiling of around $55 on prices for some time to come.”