It is now 12-and-a-half years since the September 11 attacks that put al-Qa’ida firmly on the map of global terrorism. The US has spent billions of dollars on its ‘war on terror’ to counter the threat and succeeded in killing Osama bin Laden three years ago. And yet al-Qa’ida-type groups are arguably stronger than ever now, especially in Syria and Iraq where they control an area the size of Britain, but also in Libya, Lebanon, Egypt and beyond.

In a groundbreaking five-part series, The Independent’s award-winning foreign correspondent, Patrick Cockburn, investigates the resurgence of the terrorist organisation. Today, he asks: why did the ‘war on terror’ go wrong?

Al-Qa’ida-type organisations, with beliefs and methods of operating similar to those who carried out the 9/11 attacks, have become a lethally powerful force from the Tigris to the Mediterranean in the past three years. Since the start of 2014, they have held Fallujah, 40 miles west of Baghdad, much of the upper Euphrates valley, and exert increasing control over the Sunni heartlands of northern Iraq. In Syria, their fighters occupy villages and towns from the outskirts of Damascus to the border with Turkey, including the oilfields in the north-east of the country. Overall, they are now the most powerful military force in an area the size of Britain.

Download the new Independent Premium app Sharing the full story, not just the headlines

The spectacular resurgence of al-Qa’ida and its offshoots has happened despite the huge expansion of American and British intelligence services and their budgets after 9/11. Since then, the US, closely followed by Britain, has fought wars in Afghanistan and Iraq, and adopted procedures formerly associated with police states, such as imprisonment without trial, rendition, torture and domestic espionage. Governments justify this as necessary to wage the “war on terror”, claiming that the rights of individual citizens must be sacrificed to secure the safety of all.

Despite these controversial security measures, the movements against which they are aimed have not only not been defeated but have grown stronger. At the time of 9/11, al-Qa’ida was a very small organisation, but in 2014 al-Qa’ida-type groups are numerous and powerful. In other words, the “war on terror”, the waging of which determined the politics of so much of the world since 2001, has demonstrably failed.

How this failure happened is perhaps the most extraordinary development of the 21st century. Politicians were happy to use the threat of al-Qa’ida to persuade people that their civil liberties should be restricted and state power expanded, but they spent surprisingly little time calculating the most effective practical means to combat the movement. They have been able to get away with this by giving a misleading definition of al-Qa’ida, which varied according to what was politically convenient at the time.

Jihadi groups ideologically identical to al-Qa’ida are relabelled as moderate if their actions are deemed supportive of US policy aims. In Syria, the US is backing a plan by Saudi Arabia to build up a “Southern Front” based in Jordan against the Assad government in Damascus, but also hostile to al-Qa’ida-type rebels in the north and east. The powerful but supposedly “moderate” Yarmouk Brigade, which is reportedly to receive anti-aircraft missiles from Saudi Arabia, will be the leading element in this new formation. But numerous videos show that the Yarmouk Brigade has frequently fought in collaboration with Jabhat al-Nusra (JAN), the official al-Qa’ida affiliate. Since it is likely that, in the midst of battle, these two groups will share their munitions, Washington will be permitting advanced weaponry to be handed over to its deadliest enemy.

This episode helps explain why al-Qa’ida and its offshoots have been able to survive and flourish. The “war on terror” has failed because it did not target the jihadi movement as a whole and, above all, was not aimed at Saudi Arabia and Pakistan, the two countries that had fostered jihadism as a creed and a movement. The US did not do so because they were important American allies whom it did not want to offend. Saudi Arabia is an enormous market for American arms, and the Saudis have cultivated and, on occasion bought up, influential members of the American political establishment.

A measure of the seriousness of the present situation is that, in recent weeks, Saudi Arabia has for the first time been urgently seeking to stop jihadi fighters, whom it previously allowed to join the war in Syria, from returning home and turning their weapons against the rulers of the Saudi kingdom. This is an abrupt reversal of previous Saudi policy, which tolerated or privately encouraged Saudi citizens going to Syria to take part in a holy war to overthrow President Bashar al-Assad and combat Shia Muslims on behalf of Sunni Islam.

In recent weeks, Saudi Arabia has called on all foreign fighters to leave Syria, and King Abdullah has decreed it a crime for Saudis to fight in foreign conflicts. The Saudi intelligence chief, Prince Bandar bin Sultan, who had been in charge of organising, funding and supplying jihadi groups fighting in Syria, has been unexpectedly removed from overseeing Saudi policy towards Syria, and replaced by a prince who has led a security clampdown against al-Qa’ida inside Saudi Arabia.

The US is increasingly fearful that support for the Syrian rebels by the West and the Sunni monarchies of the Gulf has created a similar situation to that in Afghanistan in the 1980s, when indiscriminate backing for insurgents ultimately produced al-Qa’ida, the Taliban and jihadi warlords. The US Under-Secretary for Terrorism and Financial Intelligence, David Cohen, warned this month that “terrorist” movements, such as JAN and the Islamic State of Iraq and the Levant (Isis), were not only destabilising Syria but “these well-funded and well-equipped groups may soon turn their attention to attacks outside of Syria, particularly as scores of newly radicalised and freshly trained foreign recruits return from Syria to their home countries”. The number of foreign fighters that Mr Cohen gives is a significant underestimate, since the head of US intelligence, James Clapper, estimates foreign fighters in Syria to number about 7,000, mostly from the Arab world, but also from countries such as Chechnya, France and Britain.

Al-Qa’ida has always been a convenient enemy. In Iraq, in 2003 and 2004, as armed Iraqi opposition to the American and British-led occupation mounted, US spokesmen attributed most attacks to al-Qa’ida, though many were carried out by nationalist and Baathist groups. According to a poll by the Pew Group, this persuaded 57 per cent of US voters before the Iraq invasion to believe that there was a connection between Saddam Hussein and those responsible for 9/11, despite a complete absence of evidence for this. In Iraq itself, indeed the whole Muslim world, these accusations benefited al-Qa’ida by exaggerating its role in the resistance to the US and British occupation.

Precisely the opposite PR tactics were employed by Western governments in 2011 in Libya, where they played down any similarity between al-Qa’ida and the Nato-backed rebels fighting to overthrow the Libyan leader, Muammar Gaddafi. This was done by describing as dangerous only those jihadis who had a direct operational link to the al-Qa’ida “core” of Osama bin Laden. The falsity of the pretence that the anti-Gaddafi jihadis in Libya were less threatening than those in contact with al-Qa’ida was forcefully, if tragically, exposed when US ambassador Chris Stevens was killed by jihadi fighters in Benghazi in September 2012. These were the same fighters lauded by governments and media for their role in the anti-Gaddafi uprising.

Al-Qa’ida is an idea rather than an organisation, and this has long been so. For a five-year period after 1996, it did have cadres, resources and camps in Afghanistan, but these were eliminated after the overthrow of the Taliban in 2001. Subsequently, al-Qa’ida’s name was a rallying cry, a set of Islamic beliefs such as the creation of an Islamic state, the imposition of sharia, a return to Islamic customs, the subjugation of women and waging holy war against other Muslims, notably the Shia, as heretics worthy of death. At the centre of this doctrine for making war is an emphasis on self-sacrifice and martyrdom as a symbol of religious faith and commitment. This has turned out to be a way of using untrained but fanatical believers to devastating effect as suicide bombers.

It has always been in the interests of the US and other governments that al-Qa’ida should be viewed as having a command-and-control structure like a mini-Pentagon, or the Mafia in America as shown in the Godfather films. This is a comforting image for the public because organised groups, however demonic, can be tracked down and eliminated through imprisonment or death. More alarming is the reality of a movement whose adherents are self-recruited and may spring up anywhere.

Osama bin Laden’s gathering of militants, which he did not call al-Qa’ida until after 9/11, was just one of many jihadi groups 12 years ago. But today its ideas and methods are predominant among jihadis because of the prestige and publicity it gained through the destruction of the twin towers, the war in Iraq and its demonisation by Washington as the source of all anti-American evil. These days, there is a decreasing difference in the beliefs of jihadis, regardless of whether or not they are formally linked to al-Qa’ida central, now headed by Ayman al-Zawahiri. An observer in southern Turkey discussing 9/11 with a range of Syrian jihadi rebels earlier this year found that “without exception they all expressed enthusiasm for the 9/11 attacks and hoped the same thing would happen in Europe as well as the US”.

"Osama bin Laden’s death had no impact on al-Qa’ida-type jihadi groups, whose greatest expansion has been since 2011"

Unsurprisingly, governments prefer the fantasy picture of al-Qa’ida because it enables them to claim a se ries of victories by killing its better-known members and allies. Often, those eliminated are given quasi-military ranks, such as “head of operations”, to enhance the significance of their demise. The culmination of this most publicised but largely irrelevant aspect of the “war on terror” was the killing of Bin Laden in Abbottabad in Pakistan in 2011. This enabled President Obama to grandstand before the American public as the man who had presided over the hunting down of al-Qa’ida’s leader. In practice, his death had no impact on al-Qa’ida-type jihadi groups, whose greatest expansion has been since 2011.

The resurgence of these jihadis is most striking on the ground in Iraq and Syria, but is evident in Afghanistan, Libya, Somalia and, in recent months, Lebanon and Egypt. In Iraq, it was a final humiliation for the US, after losing 4,500 soldiers, that al-Qa’ida’s black flag should once again fly in Fallujah, captured with much self-congratulatory rhetoric by US Marines in 2004. Aside from Fallujah, Isis, the premier jihadi movement in the country, has rapidly expanded its influence in all parts of Sunni Iraq in the past three years. It levies local taxes and protection money in Mosul, Iraq’s third largest city, estimated to bring in $8m (£4.8m) a month.

It has been able to capitalise on two factors: the Sunni revolt in Syria and the alienation of the Iraqi Sunni by a Shia-led government. Peaceful protests by Sunni started in December 2012, but a lack of concessions by Prime Minister Nouri al-Maliki and a massacre at a peace camp at Hawijah last April is transmuting peaceful protest into armed resistance.

Last summer, Isis freed hundreds of its leaders and experienced militants in a spectacular raid on Abu Ghraib prison. Its stepped-up bombing campaign killed 9,500 people, mostly Shia civilians, in the course of last year, the heaviest casualties since 2008. But there is a crucial difference between then and now. Even at the previous peak of its influence in 2004-06, al-Qa’ida in Iraq did not enjoy as strong a position in the Sunni armed opposition as it does today.

Jessica D Lewis, of the Institute for the Study of War, commented in a study of the movement at the end of 2013 that al-Qa’ida in Iraq “is an extremely vigorous, resilient and capable organisation that can operate from Basra to coastal Syria”.

In Syria, Isis was the original founder in early 2012 of JAN, sending it money, arms and experienced fighters. A year later, it tried to reassert its authority over JAN by folding it into a broader organisation covering both Syria and Iraq. The two are now involved in a complicated intra-jihadi civil war that began at the start of the year, pitting Isis, notorious for its cruelty and determination to monopolise power, against the other jihadi groups. The more secular Free Syrian Army (FSA), once designated along with its political wing by the West as the next rulers of Syria, has collapsed and been marginalised.

The armed opposition is now dominated by jihadis who wish to establish an Islamic state, accept foreign fighters, and have a vicious record of massacring Syria’s minorities, notably the Alawites and the Christians. The Islamic Front, for instance, a newly established and powerful alliance of opposition brigades backed by Turkey and Qatar, is fighting Isis. But that does not mean that it is not complicit in sectarian killings, and it insists on strict imposition of sharia, including the public flogging of those who do not attend Friday prayers. The Syrian jihadis rule most of north-east Syria aside from that part of it held by the Kurds. The government clings to a few outposts in this vast area, but does not have the forces to recapture it.

The decisions that enabled al-Qa’ida to avoid elimination, and later to expand, were made in the hours immediately after 9/11. Almost every significant element in the project to crash planes into the twin towers and other iconic American buildings led back to Saudi Arabia. Bin laden was a member of the Saudi elite, whose father had been a close associate of the Saudi monarch. Of the 19 hijackers on 9/11, 15 were Saudi nationals. Citing a CIA report of 2002, the official 9/11 report says that al-Qa’ida relied for its financing on “a variety of donors and fundraisers, primarily in the Gulf countries and particularly in Saudi Arabia”. The report’s investigators repeatedly found their access limited or denied when seeking information in Saudi Arabia. Yet President George W Bush never considered holding the Saudis in any way responsible for what had happened. The exit of senior Saudis, including Bin Laden relatives, from the US was facilitated by the government in the days after 9/11. Most significantly, 28 pages of the 9/11 Commission Report about the relationship between the attackers and Saudi Arabia was cut and never published – despite a promise by President Obama to do so – on the grounds of national security.

Nothing much changed in Saudi Arabia until recent months. In 2009, eight years after 9/11, a cable from the US Secretary of State, Hillary Clinton, revealed by WikiLeaks, complains that “donors in Saudi Arabia constitute the most significant source of funding to Sunni terrorist groups worldwide”.

Moreover, the US and the west Europeans showed themselves indifferent to Saudi preachers, their message spread to millions by satellite TV, YouTube and Twitter, calling for the killing of Shia as heretics. These calls came as al-Qa’ida bombs were slaughtering people in Shia neighbourhoods in Iraq. A sub-headline in another State Department cable in the same year reads: “Saudi Arabia: Anti-Shi’ism As Foreign Policy?” Five years later, Saudi-supported groups have a record of extreme sectarianism against non-Sunni Muslims.

Pakistan, or rather Pakistani military intelligence in the shape of the Inter-Services Intelligence (ISI), was the other parent of al-Qa’ida, the Taliban and jihadi movements in general. When the Taliban was disintegrating under the weight of US bombing in 2001, its forces in northern Afghanistan were trapped by anti-Taliban forces. Before they surrendered, hundreds of ISI members, military trainers and advisers were hastily evacuated by air. Despite the clearest evidence of ISI’s sponsorship of the Taliban and jihadis in general, Washington refused to confront Pakistan, and thereby opened the way for the resurgence of the Taliban after 2003, which neither the US nor Nato has been able to reverse.

Al-Qa’ida, the Taliban and other jihadi groups are the offspring of America’s strange alliance with Saudi Arabia, a theocratic absolute monarchy, and Pakistani military intelligence. If this alliance had not existed, then 9/11 would not have happened. And because the US, with Britain never far behind, refused to break with these two Sunni powers, jihadism survived and prospered after 9/11.

Following a brief retreat, it took advantage of the turmoil created by wars in Iraq and Afghanistan and, later, by the Arab uprisings of 2011, to expand explosively. Twelve years after the “war on terror” was launched it has visibly failed and al-Qa’ida-type jihadis, once confined to a few camps in Afghanistan, today rule whole provinces in the heart of the Middle East.