As they fled Afghanistan after 9/11 some of bin Laden’s followers wondered whether the attacks on the US had been a mistake. Among them was one of al-Qaida’s most acerbic writers, Abu Musab al-Suri. In public he backed bin Laden: privately he described him as an obstinate egotist. And he was scathing about the consequences of 9/11: ‘The outcome, as I see it, was to put a catastrophic end to the jihadi current which started in the early 1960s.’ Al-Suri believed that the Afghan Taliban regime, the most religiously correct Islamic emirate in centuries, had been destroyed for the sake of a provocative attack on a country al-Qaida could not defeat. Before 9/11, the organisation’s training camps had processed a steady stream of highly motivated recruits. After the attacks it was on the run. Another senior al-Qaida figure, Abu al Walid al-Masri, put it even more bluntly. Bin Laden, he said, had led his followers to ‘the abyss’.

A decade later those concerns seemed to have been vindicated. By 2011 al-Qaida had been reduced to a few bands of men hiding in the mountains along the Pakistan-Afghanistan border. Distracted by the need to evade death or capture they were capable of mounting only puny attacks. Their allies in the Afghan Taliban were a shadow of their former selves: they may have been fighting US forces with increasing vigour, but they were nowhere near conquering Kabul for a second time.

It was much the same story in North Africa, where the local al-Qaida branch, al Shabab, was thrown out of Mogadishu by African Union forces fighting in support of the Transitional Federal Government of Somalia. In Saudi Arabia, al-Qaida had suffered an even more devastating reverse. In 2003 the royal family had to be persuaded that al-Qaida was a genuine threat, but once that was done the state was running a concerted and well-resourced security and propaganda campaign. Senior clerics were seen on TV denouncing the organisation.

At the same time, many of al-Qaida’s most capable leaders, including Khalid Sheikh Mohammed, the man who put together the 9/11 attack, were languishing in Guantánamo. Those who were still in Pakistan faced drone strikes of such accuracy that they had to be continually on the move and could not risk meeting up for more than a few minutes. The West’s assault on al-Qaida’s finances had left bin Laden, the son of one of the wealthiest families on earth, worrying about money for the first time in his life. Then in May 2011 the US located and killed him.

All of which makes plain how remarkable al-Qaida’s resurgence over the last three years has been. Northern Syria may well come to play the role Afghanistan once played as a place for jihadis to train before they are deployed back in their home countries. MI5 believes that the number of British jihadis in Syria is in the low hundreds. The figure may be exaggerated, but it is still likely enough that young men from the UK who fail to find martyrdom in Syria will return to fight at home. Syria is now host to two al-Qaida-related organisations: the al-Nusra Front and alongside it (and sometimes in competition with it) the Islamic State of Iraq and al-Sham (ISIS), an organisation that fights for Islamic rule not only in Syria but in Lebanon, Israel and the Occupied Territories. ISIS is now responsible for as many as 68 car bombings a month in Iraq. In Yemen, al-Qaida in the Arabian Peninsula threatens the survival of the government.

As the Westgate mall attack in Nairobi highlighted, there have been similar developments in Africa. In early 2013 pro-al-Qaida elements swept through northern Mali and drew in the French army. In February, six weeks after he had deployed troops to northern Mali, François Hollande said that the French forces were ‘in the final phase of the operation’. Six months later the French were still there and the chief of the armed forces, Admiral Edouard Guillaud, admitted that the conflict had expanded to neighbouring countries. ‘I think we should hunt them down everywhere,’ he said. ‘That’s why we are working with our neighbours in Niger, Burkina Faso and Chad, and also co-operating with Algeria so that there is no sanctuary for them.’ We should not be surprised if in another six months he is saying something must be done in Nigeria and Libya too. Ayman al-Zawahiri surely views his period as leader of al-Qaida with satisfaction.

In Decoding al-Qaida’s Strategy Michael Ryan draws a useful distinction between the drawn-out or ‘deep’ battle of ideas and the ‘close’ battle of combat. The recent al-Qaida advances can be seen as close-battle victories of relatively little importance, but al-Qaida and the West are also engaged in a broad ideological struggle and it is here, significantly, that the West’s inability to put up a decent counterargument to al-Qaida is more worrying. In the last three years bin Laden and now al-Zawahiri have convinced millions of people that, for all their excesses, they have a point. Their key messages relate to Western double standards now so widely discussed as to require only the briefest rehearsal: the West’s creation and subsequent abandonment of a mujahedin fighting force to confront the Soviets; the neglect of the hallowed principle of habeas corpus implicit in extraordinary rendition; detentions without trial in Guantánamo; the humiliation and torture of prisoners in Iraq; the CIA’s use of drones in Pakistan, Somalia and Yemen; and, most recently, the NSA’s disregard for privacy. This litany of human rights abuses, al-Qaida argues, is explained by the West’s hatred of Islam. The actions of a few fringe figures such as Pastor Terry Jones who do indeed seem to hate Islam are then cited as supporting evidence.

One aspect of the US’s use of torture, incidentally, has received too little attention. Khalid Sheikh Mohammed was waterboarded no fewer than 183 times. At some point he worked out his interrogator’s protocols, including the length of time (40 seconds) he could have water poured down his throat. By the end he was seen counting down the seconds with his fingers. It’s said by people who have read the transcripts of his confessions that some of his information led to the arrests of leading jihadis. But Khalid Sheikh Mohammed later told the Red Cross that he also gave false information so as to confuse the Americans. Crucially, he failed to answer questions about the location of either al-Zawahiri (despite having met him the day before his capture) or bin Laden’s courier, Abu Ahmed al-Kuwaiti, the man who led the Americans to Abbotabad. Other detainees told the Americans that al-Kuwaiti had been well known to Khalid Sheikh Mohammed for years. So the waterboarding not only fed al-Qaida’s narrative, it was also ineffective.

Other examples of counterproductive Western policies abound. Take the 2013 drone strike on the Taliban leader Hakimullah Mehsud. Mehsud was known to have deployed suicide bombers all over Pakistan, yet his death evoked a wave of sympathy in the country. It was simultaneously a close-battle victory for the US and a deep-battle defeat. Although many Pakistanis were happy that Mehsud was no long threatening them, their relief was outweighed by the thought that the US’s use of drones in Pakistan was an unacceptable breach of sovereignty and a national humiliation. Sympathy for Mehsud was punctured only when a journalist revealed that his house in the tribal areas was worth $120,000 (a huge sum in that part of Pakistan) and included extensive gardens with orchards.

At a stroke the news dented the Taliban’s carefully burnished image as selfless holy warriors renouncing worldly comforts for their faith. Some jihadis evoke the kind of admiration that Western socialists feel for the volunteers who fought in Spain. YouTube videos of fit young men, brave, idealistic and pious, washing themselves in mountain streams, give jihadism a romantic air. Bin Laden was perfectly placed to take advantage of this appeal, having given up his riches for a life of hardship and struggle. You can’t help thinking that the US would have been well advised to use the drones not to kill Mehsud but to leaflet his admirers with images of his luxurious lifestyle.

The carefully honed Robin Hood image is only part of the story. Ryan’s stated purpose in Decoding al-Qaida’s Strategy is to identify other aspects of radical Islam’s support base, the better to equip US strategists for the deep battle. The book consists of summaries, translations and analysis of important al-Qaida texts including al-Zawahiri’s Knights under the Prophet’s Banner; The Administration of Savagery by Abu Bakr Naji; A Practical Course for Guerrilla Warfare (actually not an ideological text but a description of military tactics) by Abd al-Aziz al-Muqrin; various articles by Abu Ubayd al-Qurashi including Fourth Generation Warfare, which argues that al-Qaida should conceive of itself as a revolutionary vanguard; and finally a perennial favourite on many jihadi websites, The Call to Global Islamic Resistance by Abu Musab al-Suri.

Ryan’s survey pins down crucial elements of al-Qaida’s appeal. Even many of its detractors in the Middle East would accept that the organisation is trying to respond to the humiliations meted out to the Arab people by colonial European powers, the US, Israel and, according to al-Zawahiri, the United Nations, the multinationals, internet providers, the global news media and international aid agencies. All these stand accused of using puppet regimes in the Middle East to continue the colonial project by other means. As I travelled around the Middle East during the Arab Spring, the word that most often cropped up in the slogans in various capitals was not ‘freedom’ – the one the Western media recognised and highlighted – but ‘dignity’. The failure of the Arab Spring has reinforced al-Qaida’s case. For a moment it looked as if people power rather than jihadi violence would topple the authoritarian regimes bin Laden railed against. Today those hopes have been dashed. Indeed, the Egyptian army’s successful assault on the democratically elected Muslim Brotherhood government may turn out to be as useful to al-Qaida as the US invasion of Iraq. It’s already a familiar argument that only the jihadis have the resolve and drive to sweep away entrenched dictatorial regimes.

Of course al-Qaida faces challenges as well as opportunities, and as with other extremist groups, one of its biggest is maintaining ideological unity. There are divisive issues: should the focus be on the near enemy (authoritarian regimes in Arab countries) or the far enemy (the US)? During the Taliban years in Afghanistan there was a split between those who favoured protecting the regime and those who wanted to use Afghanistan as a platform for global attacks, even if it meant jeopardising the mullahs’ rule. Al-Qaida members do not agree about how much of their energy should be devoted to killing Shias.

There has also been a long debate about hierarchy and the extent to which al-Qaida should be a hierarchical organisation. Many Western writers have tried to describe al-Qaida’s structure. The central leadership is often likened to the board of a multinational company overseeing local franchises. The franchisees have to stick to at least some al-Qaida policies and in return can use the brand name. In another parallel from the business world, al-Qaida is said to have affiliates rather than fully-owned and controlled subsidiaries. Bin Laden’s overarching innovation was to give jihad a global dimension. Given that very broad objective, he and now al-Zawahiri could tolerate all sorts of local movements with far narrower goals. What harm if a Tunisian salafi opposing the Ennadha leader Rashid Gannouchi, for example, claims to be from al-Qaida? Al-Zawahiri might have more pressing priorities but is hardly going to object if someone fights the Tunisian branch of the Muslim Brotherhood in his name. The uncertainty as to who is and who isn’t an al-Qaida member has another advantage. Western media correspondents have long since tended to label any jihadi-style attack as the work of al-Qaida. All the way from the 1996 Khobar Towers bombing in Riyadh to the Mumbai attack in 2008, al-Qaida has initially been held responsible and subsequently found to have played no direct role.

Take the Madrid train bombings in 2004. Despite conflicting evidence, the Spanish judicial authorities have concluded that there was no direct order for the attack from al-Qaida’s leadership in Pakistan. But there is no doubt that al-Qaida inspired the attack. Al-Qaida tracts were found on the plotters’ computers. One article actually discussed the merits of targeting Spain as a way of forcing the withdrawal of Spanish forces from Iraq – an objective that was achieved. So was the Madrid train bombing an al-Qaida attack? To what extent could a national-level jihadi group trying to follow al-Qaida’s ideas be accurately described as part of the organisation? It’s a question that has sparked a surprisingly vitriolic debate among Western experts. Some, such as Bruce Hoffman, have argued that the organisation’s potency derives from its highly disciplined and competent central leadership; others, such as Marc Sageman, have emphasised al-Qaida’s capacity to inspire individuals or ‘bunches of guys’ to act autonomously. This debate in Western academia echoes internal discussions within al-Qaida, some of whose founders explicitly considered whether to establish a hierarchical structure of party workers or just a loose network of sympathisers.

In Peshawar in the early 1990s Abu Musab al-Suri gave a series of lectures, now famous in jihadi circles, in which he advocated decentralised, loose affiliations that could more easily evade Western security systems. By the time of 9/11 he was arguing that al-Qaida was not a group but rather ‘a call, a reference, a methodology’, which would survive even if its leadership was eliminated, as he fully expected it to be. These questions preoccupy Jacob Shapiro. The difficulty facing a terrorist leader, he argues, is to ensure that his subordinates apply just the right amount of violence to the right targets. Only then can the organisation maintain popular support and high morale among its membership while simultaneously persuading the government it is fighting that it’s sufficiently threatening to be taken seriously. If a group kills too many civilians, popular support drains away; if it uses too little violence, the government will dismiss it as irrelevant and its cadres will drift off to more active outfits.

On Shapiro’s account the Provisional IRA was remarkably adept at calibrating how much and what kind of violence to use. Having launched a series of spectacular attacks in Northern Ireland it moved onto economic targets in London. Just in case the government hadn’t got the message, at a crucial point in the talks process it launched a mortar attack at Heathrow which, Shapiro suggests, was deliberately set up so that the bombs did not detonate. It helped persuade the government that concessions might be a good idea. Once the Provisional IRA had achieved a settlement the breakaway Real IRA showed far less subtlety. Its first bomb attack in Omagh killed 29 civilians, alienated public opinion and put the organisation out of business for the following three years.

The phenomenon of excess violence is structural. Many of the junior ranks of terrorist organisations are so highly motivated that they want to use more violence than the leadership thinks wise. And there is another inherent problem. Some volunteers sign up not as a result of genuine political commitment but rather for the sense of empowerment that comes with carrying out violent missions. Zealous recruits of this kind have a tendency to filch the organisation’s funds. It’s an issue, Shapiro finds, that has been faced by every terrorist outfit all the way from pre-revolutionary Russia to Gaza. Leaderships, then, need to exercise control over targeting and money. Shapiro sees the dilemma clearly: the mechanisms required to exert control, such as semi-judicial processes to hand down punishments to wayward members, expenses claim sheets and documents in which new members commit themselves to following the leadership’s policies, all make a group vulnerable. If the security services find the documents or computer files, the result can be long terms of imprisonment for all concerned.

Shapiro believes that these general rules apply to al-Qaida just as much as to any other group. The organisation was so effective before 2001 because it was in the highly unusual position of having a place in which it could operate – Taliban-run Afghanistan – under a state security apparatus that left it alone to recruit, train and process thousands of fighters. The leadership itself was able to select the highly capable group of men who carried out the 9/11 attacks. As the Americans subsequently discovered, al-Qaida at this time had generated volumes of paperwork. There were even employment contracts with clearly laid-out vacation policies. It was just that no one in the Afghan security agencies was looking for them. After 9/11 all that changed. Al-Qaida’s role in resisting the US forces in Iraq showed the problems it faced in a hostile security environment. A year after the invasion a Jordanian jihadi, Abu Musab al-Zarqawi, had established the Group of Monotheism and Jihad. In the course of 2004, after discussions with the al-Qaida leadership in Pakistan, the group was renamed al-Qaida in Iraq. Zarqawi, who had a deep hatred of Shias, set about killing them, often by beheadings that were filmed and posted on YouTube. His idea was that the Shias would eventually fight back against him, a development that would radicalise Sunni opinion, thereby increasing his support base. The al-Qaida leadership in Pakistan realised that this was a terrible plan. Far from trying to provoke a civil war by having Zarqawi organise the mass murder of Iraqi civilians, they wanted al-Qaida in Iraq to focus on the occupying US forces. A letter from al-Zawahiri to Zarqawi seized by the Americans showed the al-Qaida leadership trying to rein in the Iraqi commander. ‘Among the things which the feelings of the Muslim populace who love and support you will never find palatable,’ al-Zawahiri wrote, ‘are the scenes of slaughtering hostages … We are in a battle and more than half this battle is taking place in the battlefield of the media.’ This was not what Zarqawi wanted to hear and his sectarian violence continued to the point at which, far from being radicalised, most Sunnis were disgusted and turned against al-Qaida in Iraq, in some cases fighting against the organisation.

Ryan argues that al-Qaida cannot prevail in the long term because it is undermined by a fundamental strategic contradiction. While it believes in a fluid, unconventional, borderless global jihad, it also needs to establish an al-Qaida state in the heart of the Muslim world as the first step towards the restoration of the Caliphate. But if northern Syria, for example, did emerge as the new Afghanistan – a place where the lack of central government means that al-Qaida will be able to recruit, train and process volunteers with little fear of interference from a security agency – the US would probably overcome its hesitation about intervening. Needless to say, the arrival of Western troops would bring great benefits to al-Qaida, which would once again deplore the crusaders’ hatred of Islam and their unquenchable desire to dominate the Middle East by force. If it did get drawn into Syria, in other words, the US would face a dilemma of its own, neatly encapsulated in the headline of the cover story of an al-Qaida Arabic-language journal in Afghanistan in 2009: ‘Disastrous Occupation or a Humiliating Withdrawal?’

Yet any American discomfort will be outweighed by bigger problems for al-Qaida. As it expands its operations around the world its battered central leadership will in all likelihood be left on the sidelines, unable to stop local fighters committing acts of brutality that will alienate local support bases. As Edward Snowden has revealed, the West’s ability to intercept communications is so advanced that controlling junior ranks over such great distances while maintaining security is virtually impossible, and the real question is how many more Zarqawis there will be. Whenever jihadi forces sympathetic to al-Qaida have taken charge of an area they have been so grossly violent that the people have rejected them.

The Swat Valley in Pakistan exemplifies the process and shows why it will be hard for al-Qaida to win the deep battle of ideas. Ground down by decades of corrupt and dysfunctional rule from Islamabad, the people of Swat wondered briefly if the jihadis might be a better option. In 2008, after a series of blunders by the Pakistan army, some local jihadis broadly sympathetic to al-Qaida managed to win power in Swat and proceeded to whip or behead anyone they didn’t like the look of. When the army eventually responded and regained control, it was grudgingly welcomed by local residents, who had concluded that a corrupt and dysfunctional government is preferable to a religious and violent one. The same process may be about to play itself out in Syria, parts of North Africa and some countries where the Arab Spring has failed. Many more close battles, in other words, may have to be fought and lost before the deep battle is concluded.