In a previous post, I argued that the organization of Al Qaeda declined even as the movement it championed remains robust. No single factor explains Al Qaeda’s problems since the September 11th attacks, but I believe the most consequential reasons for this decline include its underestimation of the U.S.-led counterterrorism campaign and the associated loss of its haven and global infrastructure; its killing of Muslim civilians; and the lack of a strong base among the people it claims to represent.

Al Qaeda anticipated, and may have even sought out, U.S. retaliation for 9/11. The ineffective U.S. response to the 1998 embassy bombings in Kenya and Tanzania and the 2000 attack on USS Cole convinced Al Qaeda leaders that it could strike the United States and emerge stronger as a result. The United States might even be deterred from future intervention. Even if the United States intervened directly, Al Qaeda had the Soviet campaign in Afghanistan as a model of an ineffectual U.S. response. The group expected the United States to become bogged down in Afghanistan quickly. It expected to galvanize ordinary Muslims with the resulting struggle, to rally jihadist groups to Al Qaeda’s standard, and to trap the United States in a quagmire that would lead to its humiliation and ultimate collapse.

But as terrorism scholar Barak Mendelsohn points out, “To some extent, the 9/11 attacks were too successful, leading the United States to react with much more force than Bin Laden expected.” The 9/11 mastermind, Khalid Sheikh Mohammad, put it more bluntly as he watched the Twin Towers fall: “Shit!” he exclaimed, “I think we bit off more than we can chew.” By the end of 2001, U.S. and Afghan forces had toppled the Taliban, scattering Al Qaeda without a haven. Al Qaeda lost more than 80 percent of its members in Afghanistan in the final months of 2001. The Taliban’s fall represented a triple loss for Al Qaeda: it lost an openly supportive government; became hunted and therefore could no longer train and proselytize on the scale it did before 9/11; and suffered damage to its reputation—the world’s only (in jihadist eyes) Islamic government had fallen, with Bin Laden’s recklessness to blame.

The global intelligence and law enforcement campaign against Al Qaeda’s infrastructure further limited the group’s operations. Before 9/11, Al Qaeda cells operated discreetly in Europe, Asia, and even the United States, with law enforcement expending limited efforts to disrupt them. After the attacks, the world went on counterterrorism overdrive. Shortly after 9/11, the United States was working with over 100 countries on counterterrorism. In 2005, a senior CIA official testified that assistance from foreign intelligence services contributed to virtually every capture or killing of a suspected terrorist outside Iraq. Many jihadist groups resented Al Qaeda for putting them on the receiving end of the U.S.-counterterrorism campaign and saw little reason to shift their focus from local regimes to the West. The jihadi strategist Abu Musab al-Suri lamented that the 9/11 attacks cast “jihadists into a fiery furnace. … A hellfire which consumed most of their leaders, fighters, and bases.”

As Afghanistan descended into civil war and Pakistan emerged as complicit, or at least tolerant, of some Al Qaeda presence, the U.S. drone campaign grew as a new United States weapon of choice. By the time President Barack Obama took office in 2009, the President relied on drones as a vital counterterrorism tool. Drone strikes in Pakistan increased from a mere handful in 2007 to more than 120 in 2010. Critics of the drone campaign emphasize that drone attacks lead to civilian casualties and that the strikes independently do not constitute a long-term solution. However, the consistent loss of leaders devastated Al Qaeda’s core. Documents found in the raid on Bin Laden’s compound included a warning from a senior Al Qaeda leader: the deaths of experienced operatives produce “the rise of lower leaders who are not as experienced as the former leaders and this would lead to the repeat of mistakes.” A tip sheet found among jihadists in Mali advised militants to avoid drones by maintaining “complete silence of all wireless contacts,” to “avoid gathering in open areas,” noting that leaders “should not use communications equipment” and to take strenuous measures to root out spies. These steps hindered Al Qaeda core’s communication with far-flung cells and affiliates.

Yet this campaign had limits—and Al Qaeda survived, though it hardly thrived. Some remnant of the Al Qaeda core found a haven in Pakistan, sheltered by sympathetic tribal and jihadist groups. From these sanctuaries, they directed attacks on London, Madrid, and other Western sites. Iran also allowed some Al Qaeda members to flee to its territory, creating a mixed relationship of sanctuary and support combined with arrests and deportation depending on Iran’s relations with the United States and regional regimes at any given time. Some countries, notably Syria, facilitated the flow of fighters to Al Qaeda’s Iraq affiliate after the U.S. invasion in 2003. After 2011, when unrest broke out in Syria, the Assad regime released Al Qaeda members from jail and otherwise created a permissive environment for jihadists in an effort to tarnish the opposition and claim that the Syrian state sided with the West against radical Islam.

The United States could not eliminate the entire network and infrastructure that fostered foreign fighters and promoted jihadist networks. Thus, new causes could rely on existing ideas and networks. Preachers, often funded from the Gulf states, occasionally embraced new causes, and enough of the network remained to generate new flows of jihadists to conflicts, such as Somalia and especially Syria after 2011.

Al Qaeda viewed itself as a vanguard that would awaken and lead the Muslim community against its foes—and here the group simply failed. Though anti-American sentiment grew after 9/11, the group’s spectacular terrorism rarely helped Al Qaeda directly. Ordinary Muslims did not rally to the group’s cause. Al Qaeda “has lost the struggle for Muslim hearts and minds,” argues Middle East analyst Fawaz Gerges.

This failure stands in contrast to the experience of terrorist groups like Hamas and Hizballah, which enjoy considerable local support. Both are integrated into the broader Palestinian and Lebanese Shi’a populations of their countries. Hamas is the de facto ruler of Gaza, and Hizballah is part of the Lebanese government. Both began as Islamist movements committed to “national liberation” against Israel, enabling them to draw on anti-foreign sentiment, as well as religious fervor. They provide social services, education infrastructure, and a political voice—and they both claim to defend their communities and act as the legitimate vehicle for using violence against an enemy. Both Hamas and Hizballah’s nationalistic element allows them to draw deeply on local support, which allows them also to rely on the populace to replenish their ranks and for support when they face severe pressure. Neither, of course, is a movement actively seeking followers from other countries (and Hizballah’s transnational reputation among Sunnis is tarnished due to its embrace of the Assad regime). In contrast, Al Qaeda lacks a similar base because of its heavy reliance on foreign fighters and shallower roots in local communities—a fact which made the group ill-prepared for the loss of many cadre to robust U.S.-led counterterrorism efforts.

This lack of support became apparent after the 2003 U.S. invasion and occupation of Iraq, when the group continued to reel from its devastating post-9/11 losses. Al Qaeda’s claim that the United States aimed to subjugate the Middle East seemed prophetic. The failure to secure Iraq and the subsequent civil war there gave the jihadist movement a new area of operations—one endowed with popular support from mainstream and radical Islamists. Respected scholars in Egypt and Saudi Arabia called for jihad against the invaders. In private correspondence, Zawahiri described the invasion as a “blessing” that allowed for jihad “in the heart of the Islamic world.” Al Qaeda and jihadist movements historically achieved success when they could harness anti-foreign sentiment.

Because of Iraq’s importance, Al Qaeda could not avoid the struggle against the United States and hope to stay relevant. But Al Qaeda lacked a presence in Iraq. As a result, it cooperated with foreign jihadists in Iraq led by Abu Musab al-Zarqawi, the Jordanian whose organization in 2004 took on the name “Al Qaeda in Iraq” (AQI) ; this was a seeming victory for Bin Laden. Thousands of foreign fighters flocked to Iraq to fight under Zarqawi’s banner; Al Qaeda, once again, appeared triumphant.

But despite the broad popularity of the anti-U.S. message, the affiliation proved costly to Al Qaeda. AQI’s killings and oppression of fellow Muslims tainted the jihadist cause, leading many hardline ideologues to excoriate the group. Senior Al Qaeda leaders warned Zarqawi about the Algerian experience, noting that militants there destroyed “themselves with their own hands by their alienation of the population with their lack of reason … oppression, deviance, and ruthlessness.” It was only the mix of discrimination and brutality on the part of the Iraqi government and the civil war next door in Syria that enabled the group to get back on its feet.

The Islamic State successfully swayed the local population more than Al Qaeda did. By establishing a caliphate and governing parts of Iraq and Syria, the Islamic State established roots in local society that will prove hard to undo. However, if Al Qaeda’s experience offers any guide, then a combination of a global campaign, direct strikes against the group, the destruction of its haven, and the Islamic State’s own barbarism will diminish this latest jihadist threat—though it will not destroy it completely. Once it is down, however, effective governance and stability are necessary to limit terrorist attacks and keep it from returning.