Everybody suddenly seems to have an opinion about Sheikh Nimr al-Nimr, until his execution on Saturday an obscure Saudi Shia cleric. Once his death was announced, the airwaves exploded. As a former British ambassador to Riyadh still active in the region, I was asked several times to comment. On Monday morning Nick Robinson on the BBC’s Today programme suggested I might like to stop explaining what was in Saudi minds and simply condemn the act.

I understand the point of the question. But I have been wondering since then what exactly it is that I and others are being invited to condemn. The fact of an execution, its nature, the Shia identity of the victim, his status as a cleric, that the Saudis still practise capital punishment, the nature of their judicial system, the timing of the act, the suspicion that it might undermine the peace process in Syria or infuriate Iran – or perhaps all of this and more?

Yet condemnation without understanding is futile. It is not enough to say that this is simply the result of the ascendancy of a new set of inexperienced senior princes. The reasons for Saudi – and Iranian – actions are structural.

Consider the context. Saudi Arabia feels with good reason more threatened than at any time in its modern history, at least since the subversive Kulturkampf of the 1950s and 1960s from Nasser’s Egypt. This stems from five sources: first, the challenge of Sunni and largely Salafi jihadism; second, the sustained ideological and material challenge of the Islamic Republic of Iran; third, the collapse of large parts of the Middle East state system following the Arab spring; fourth, a sharp fall in global energy prices; and fifth, a sense that historical alliances – notably but not only with the United States – are fraying.

These threats are real. A decade or so ago, the heirs to Juhaiman al-Otaybi’s 1979 Grand Mosque attackers, al-Qaeda in the Arabian Peninsula, launched a terror campaign within the kingdom with the aim of inspiring a general Sunni insurgency. The Saudis were slow to realise what they were facing. Once they did, they mobilised and ruthlessly crushed the terrorists. But they did not go away. The remnants regrouped in Yemen and from there plotted, recruited others (including the American imam Anwar al-Awlaki) and directed further attacks, against Western and Saudi targets. More recently there has been a wave of attacks, claimed by Islamic State, mostly on Shia targets – but also on security forces and a Sunni mosque at a military base near the Yemeni border.

The threat from Iran is of longer standing but has waxed and waned over time. Ayatollah Khomeini proclaimed that the revolution of 1979 was not just Iranian or Shia but Islamic and global, come to rescue Frantz Fanon’s damnés de la terre – suffering subaltern humanity – from their oppressors. There was widespread unrest fomented by Tehran in Kuwait, Bahrain and Saudi Arabia’s Eastern Province. A variety of revolutionary Shia groups emerged across the Middle East, including the Islamic Action Organisation, the Front for the Islamic Revolution in the Eastern Arabian Peninsula, the Islamic Front for the Liberation of Bahrain, Hezbollah in the Hejaz and, most notoriously, Hezbollah in Lebanon. An attempt was made on the life of the Kuwaiti emir, Kuwaiti airliners were hijacked, the French and US embassies in Kuwait, and in Beirut the US embassy and US and French barracks were attacked by suicide bombers.

This eventually calmed down under the pressures of real-world politics. The Saudis liked it when pragmatists such as Akbar Hashemi Rafsanjani and Mohammad Khatami became president in Iran. But experience confirmed that security policy remained in the hands of hardliners. And the apocalyptic populism of Mahmoud Ahmadinejad brought back the threat of an exported revolution.

The disturbances in Bahrain and, to a lesser extent, the Eastern Province in 2011 confirmed for the Saudis that whether they started it or not (probably not), the Iranians would always be willing accomplices to social and political unrest in the Sunni states of the Gulf. Tehran’s rather clumsy attempts at a rapprochement with Mohammed Morsi’s government in Cairo in 2012/13 alarmed the Saudis further (needlessly, as it turned out) because it raised the possibility of a renewed alliance between Shia and Sunni Islamists in the region’s two most populous countries. They saw further proof of the threat in Iranian support for the Houthi insurrection in Yemen and the capture of large parts of the Iraqi state by Iranian agents of influence. The cap was the direct Iranian/Hezbollah intervention on the side of the murderous Assad regime in Syria.

This has all contributed to the sectarianisation of political conflict in the Middle East. This is not a product of recent Saudi state policy, as some have suggested. As we see from its rapid and visible reaction to attacks inside the kingdom on Shia targets, the government sees domestic sectarian division as a national security concern. Nor is it the inevitable result of age-old enmities (an idea that is generally tripe). There is undoubtedly deep popular anti-Shia prejudice. And Shias have indeed been disadvantaged in many Sunni states and communities. Iran has claimed to be their protector since the 17th century. Arguments about precisely what role Shiaism and Shia mujtahids (jurisprudents) should play in the regulation of the righteous state go back even further. But this usually produced communal quietism and the defence of religious, not political rights. The critical inflection point was the dramatic political mobilisation of the Iranian Revolution, which gave life to a thousand forms of adversarial, transnational and often violent Shia activism, including those that seem to have inspired Sheikh Nimr.

Now the Saudis face a period of sustained low energy prices at a time when the costs of a newly interventionist and expeditionary foreign policy are rising dramatically and when the need to restructure the economy to create perhaps an extra four million new jobs by 2020 has become urgent. At the same time they know that a small but significant section of the Sunni population of the kingdom is vulnerable to the dark seductions of Islamic State, because they regard it as more legitimately Islamic, or as the only organised Sunni group pushing back against Iran, the Shia, or both. There is no clear link between economic deprivation and radicalisation. But the former doesn’t help if it leads to idle hands and claims of social injustice.

To cap it all, the Iranian nuclear deal angered the Saudis not because it was a nuclear deal but because it was simply a nuclear deal, failing in their view to address malign and subversive non-nuclear Iranian activities in Bahrain, Iraq, Syria, Lebanon and Yemen, and rewarding Iran prematurely. They have felt very abandoned by the US and other Western states. And they believe the apparent pragmatism of the Rowhani government is a façade, offering privileged access in return for the suspension of any critical faculty. That makes the issue of the Vienna peace talks on Syria secondary. There will certainly be an impact. Yet it is not as if the Saudis had disguised their deep scepticism. They had been pressured to sit with the Iranians, but they had also insisted on continuing to support opposition forces in the field and have not wavered in their insistence that Assad needs to go.

You might think this is all special pleading. But before you say that the matter is a straightforward one of a benighted justice system administering medieval punishments to dissidents, reflect on this. Sheikh Nimr advocated the destruction of the rulers of Saudi Arabia and Bahrain and the secession of the Eastern Province. His version of a righteous Islamic state is not a thousand miles from that of Abu Bakr al-Baghdadi (and a long way from the non-takfiri, non-caliphal, neo-Westphalian pragmatism of the Saudi state). He called for wilayat al-faqih, the heterodox Guardianship of the Jurisprudent espoused by Khomeini.

The vengeful early years of the Islamic Republic, when clerics who previously would not have hurt a fly enthusiastically participated in the judicial murder of thousands in the name of righteousness, show some of the consequences. So does the arrest and humiliating mistreatment in 1982 of the venerable Ayatollah Shariatmadari, who stood up to Khomeini and dared to object to the implementation of any Islamic hudud punishments in the absence of the Hidden Imam. So does the continued rate of executions in Iran (nearly 700 by July last year, according to Amnesty International) and the Islamic Republic’s own treatment of dissidents – and, indeed, of the ordinary protesters of 1999, 2009 and 2011.

The signals the Saudi state sought to send by executing 43 Saudi Sunnis convicted of terrorism at the same time as Sheikh Nimr and his three fellow Shias reflected all of this. To their own citizens the message was: we shall enforce the judgment of the courts on all those who seek to undermine the stability of the kingdom and the legitimacy of its government, irrespective of sect, and on your behalf we shall resist Iranian expansionism and Islamic State predation with equal vigour. To Iran it was: Saudi citizens owe loyalty in tribal fashion to their king, not to foreign religious leaders or to some ideal of transnational Islamism, and we shall not tolerate interference. To the rest of the world it was: we shall not bend in the face of the storms raging round the region, if necessary alone.

Even allowing for all that, you might still think the execution of Sheikh Nimr unconscionable and that it should be condemned as a symbolic act of state brutality, visiting on one man and his three companions the fears of a community which might be better allayed by acts of patient policy. That would be a perfectly ethical position to hold. But it is not a policy. And understanding must still come first.

If we think that a large part of the reason for states lashing out is the fear in which they exist, then doing something to address that fear is a large part of the answer. In this case, that principally demands showing that we mean to enforce the Iranian nuclear deal rigorously – not hold off on additional measures against provocative missile testing (for instance) because the Iranians won’t like it (that’s the point); supporting Prime Minister Haider al-Abadi in Iraq as he seeks to capitalise on success in Ramadi, slowly bring alienated Sunnis back into the fold and reduce the pernicious influence of the Shia militias; and push back firmly against Iranian meddling in Bahrain and Yemen, which may be exaggerated but exists.

Middle Eastern security won’t fix itself and stability needs sponsors. It won’t be easy. Iran is an important state and its reintegration into the regional state system would be a huge achievement. But this reintegration depends on its own actions. In the meantime we need to recognise where our real interests lie. We have huge interests at stake in the Sunni states of the Gulf. And the state system in the region will be the basis of any conceivable future political stability. So, we need to engage with the likes of Saudi Arabia as these states search for ­solutions to the challenges they face. We can’t simply stand back, wring our hands, say it’s all too messy and in any case it’s their own fault – and then profess to be appalled by the consequences.

John Jenkins is a former British ambassador to Saudi Arabia, Libya, Iraq, Syria and Burma. He is now executive director (Middle East) of the International Institute for Strategic Studies, and is based in Bahrain