In the aftermath of the Algerian hostage crisis, David Cameron issued an ominous warning of the continued threat from terrorism. But is al-Qaida more, or less, dangerous than before?

Last week the world took another step towards succumbing to an existential threat. Again.

Speaking in the aftermath of the spectacular seizure and siege of an Algerian gas refinery by Islamist extremists 10 days ago, David Cameron warned of how "we face a large and existential terrorist threat from a group of extremists based in different parts of the world who want to do the biggest possible amount of damage to our interests and way of life".

There was little further detail, leaving it unclear if the prime minister was referring to al-Qaida, the group founded by the late Osama bin Laden 25 years ago. Or possibly al-Qaida-type groups in the middle of the Saharan desert. Or maybe other offshoots around the world. Or possibly the ideology of al-Qaida.

However, the broad thrust of what he was saying was obvious: if you thought the threat from al-Qaida, however defined, had gone away, you were wrong. It is here, and will be here for decades to come. And it endangers the very foundation of our societies. The intervening week, one imagines, replete as it was with a range of shootings, bombings, arrests and court judgments across the world all involving Islamist extremism, has not improved things.

Such rhetoric was once familiar. We heard much of it in the immediate aftermath of the 9/11 attacks and through the months before the invasion of Iraq in 2003. But as the years have passed however, such pronouncements of imminent danger became rarer. The public naturally learned to be suspicious of rhetoric raising fears that appeared unreasonable and unfounded. We all learned enough about the complex phenomenon of contemporary Islamist militancy to be able to challenge the sillier claims ourselves. Policymakers recognised that any exaggeration, particularly of the "global" nature of a threat that their own security services were increasingly seeing as local, simply played into the hands of the enemy.

So Cameron's words last week, echoed elsewhere, were unexpected.

Rather like al-Qaida's own rhetoric in the wake of the changes wrought by the Arab spring, they sounded dated; at worst, they were an indication of wilful ignorance, a nostalgia for simpler times when leaders could promise "iron resolve" against a threat without provoking widespread scepticism. They have however usefully provoked a new debate on two very old questions, both still urgent and important: what is al-Qaida? And is it more or less dangerous than it was?

Answering the first question is, for once, relatively straightforward. Islamist militancy is a phenomenon going back much further than the foundation of the group al-Qaida by Saudi-born Osama bin Laden in 1988. There have been waves of revivalism in the Muslim world since the days of the Prophet Muhammad. These have frequently come in response to external challenges, whether political, social, cultural and military. Intense and very varied reactions were provoked by European colonialism in the 19th century from Afghanistan to Algeria, from Morocco to Malaysia and beyond. The end of European colonialism in the Muslim world in no way diminished the immediacy of that challenge nor the venality, brutality and incompetence of local regimes. In the late 1980s and early 1990s, scores of different violent extremist movements, in part products of a massive new interest in "Islamism" across the Muslim world, were waging armed struggles against local governments in the name of religion.

Al-Qaida (usually translated as "the base") was founded – in Pakistan towards the end of the war in Afghanistan against the Soviets – to channel and co-ordinate the dispersed efforts of these movements into a single campaign. It believed that striking at a universally accepted global enemy, the US, would lead to the destruction of "hypocrite" unbelieving regimes across the Muslim world in the short term and, eventually, the creation of a new ill-defined and utopian religious rule. This latter goal was long-term, a cosmic struggle, possibly indefinite and certainly undefinable in terms of time.

Aided by a range of external factors, al-Qaida was to some extent successful in achieving its less abstract aims, striking the US hard and drawing together an unprecedented network of affiliates in the late 1990s. This then helped – particularly by the response to the 9/11 attacks and other operations – disseminate its ideology further than ever before in the noughties.

The high point, however, was reached around 2004 or 2005. Even as it appeared to peak, the wave of extremism was receding. Since then, the central leadership of al-Qaida has suffered blow after blow. It is not just Bin Laden who has been killed or rendered inactive, but pretty much everyone else in the senior and middle ranks of the organisation. Ayman al-Zawahiri, the leader of al-Qaida central, may be an effective, utterly dedicated and experienced organiser but he lacks Bin Laden's charisma. Saif al Adel, the only other veteran leader remaining, lacks his stature and may not be at liberty at all but detained in Iran.

Key players who few, beyond specialists, had ever heard of – such as the very capable Libyan Atiyah Abd al-Rahman – have gone. British security officials describe "al-Qaida central" as being "hollowed out", largely by the controversial drone strikes. Equally damaging for the group, al-Qaida's training infrastructure is minimal, certainly compared with the dozens of fully fledged camps that were in use on the eve of the 9/11 attacks. Back in 2008, according to interrogation documents, handlers were forced to admit to new recruits coming straight from Europe that their facilities unfortunately bore no resemblance to those depicted in recruiting videos.

Nothing has improved since. Volunteers are fewer than before. There are younger members rising up the thinning ranks, but this is promotion by default not merit.

Equally damaging has been the rejection by successive communities over the past two decades. Almost every attempt by al-Qaida central to win genuine popular support has failed – in Iraq, Pakistan, Saudi Arabia and elsewhere. Polls show approval ratings for Bin Laden peaking around 2004-5 and then steep decline. This is particularly true when communities have direct experience of extremist violence or rule. The al-Qaida brand is irremediably tarnished. Even Bin Laden was apparently thinking of relaunching the group under a new name, his correspondence reveals.

The terrorist siege of Mumbai had no links with al-Qaida. Photograph: Sebastian D'souza/AP

The two most spectacular attacks in recent years – in Algeria and the strike on Mumbai by Pakistan-based militants from the Lashkar-e-Taiba organisation – were carried out by entities that have, in the first instance, tenuous connections with al-Qaida's senior leadership and, in the second, none at all. This indicates the degree to which the remnant led by al-Zawahiri have become, at best, only one player among many.

The result is that the centripetal force the group once exerted has gone and we have returned to a situation similar to that of the old "pre-al-Qaida" days with a whole series of different local groups involved in local struggles with negligible central co-ordination.

There are major differences with the previous period, of course. Decades of violence have led to much higher structural levels of radicalisation and polarisation. The technology and tactics used by all protagonists in these current "shadow wars" has evolved. Then there are the consequences of the Arab spring – for the Sahel and Syria and elsewhere. But, nonetheless, the unthinking use of the term al-Qaida, as has so often been the case in the past, obscures rather than illuminate the real chaotic and fractured, if still dynamic, nature of modern Islamist militancy. This is something Cameron's own security services will have told him.

Of course a threat remains. But the big attacks – those that could potentially pose something a little closer to "an existential threat" – are unlikely. These would need to be in a major European or US city or involve at least one passenger jet. If British intelligence, despite having a team devoted for months to checking and rechecking every possible potential lead, could not come up with a single credible threat to the London Olympics last year and their US counterparts were confident enough to declare a similar lack of immediate danger during the recent presidential campaign, it appears fair to assume that bombs in London or New York are a fairly distant prospect for the moment. The biggest threat to airplanes comes from a single highly proficient bombmaker in the Yemen.

The location of the major spectacular attacks appears closely related to al-Qaida's ability to focus the dispersed energies of contemporary Sunni Islamist extremism. Through the 1990s, attacks were restricted to targets – in Pakistan, Algeria, Egypt, Saudi Arabia and elsewhere – which were distant from western populations, with the exception of the first abortive plot to bomb the World Trade Center in New York in 1993. US troops who were attacked in Somalia in that year in the famous "Blackhawk Down" episode had simply strayed into someone else's war.

By the late 1990s, US interests were being attacked, but in east Africa or the Yemen. It was only through the first six years of the past decade that the violence approached the west – first in Indonesia, Morocco, Saudi Arabia and Turkey, then in Madrid and London. But since, the dynamic has reversed, tracking the new weakness of the al-Qaida senior leadership. The big attacks still come – but in Islamabad, Mumbai, Kabul, Baghdad, and now in the deserts of the Sahara. Nor do they strike targets that resonate throughout the Muslim world. A gas refinery in southern Algeria is not the Pentagon.

Partly this is due to vastly improved security precautions and competent intelligence services that co-operated much more effectively.

Intermittent attempts to down airplanes have been defeated, if only just. Hundreds of potential troublemakers have been stopped long before they even begin to contemplate actually perpetrating a violent attack. MI5 officials say that, in part due to closer collaboration with a range of other agencies and particularly the police, they are able to head off possible threats much earlier. One compared their operations to the famously tedious stonewall tactics of the Arsenal team 20 years ago. "It's boring but it works," he said.

There is, of course, the fear of a "lone wolf", a solo, self-radicalising extremist. The example most often cited is Mohamed Merah, the French-Algerian who killed three soldiers as well as three Jewish schoolchildren and a teacher last March.

A spokesman for Mokhtar Belmokhtar, the man who orchestrated the recent refinery attack in Algeria, told French media on Monday that France could expect "dozens like ... Merah and Khaled Kelkal" who would spontaneously rise up to kill and maim.

Islamist militia leader Mokhtar Belmokhtar, who orchestrated the Amenas refinery attack in Algeria. Photograph: AP

But real lone wolves are extremely rare. Kelkal, who carried out a series of attacks in France in 1995, plugged into a broader network of militants run and recruited by Algerian groups active at the time. Merah did the shooting on his own but came from a family steeped in extremist versions of Islam and anti-Semitism, had been to Afghanistan and Pakistan to train and was, French and Pakistani officials say, connected to Moez Garsalloui, a high-profile known Belgian militant, now dead, who had been recruiting widely and was well-known to intelligence services. Merah was thus not only part of an old style of terrorism – recruits making their way to the badlands of Pakistan to get trained and then returning to carry out attacks – but was also much less effective than predecessors such as those responsible for the 7/7 attacks in London. The number of people making that journey is now a fraction of the levels of six or seven years ago. Back then, scores, if not hundreds, made their way to the Afghan-Pakistan frontier to fight alongside the Taliban or other groups. Now the number is in the low dozens, according to intelligence officials in Pakistan, the UK and elsewhere.

The other fear is of a new generation of veteran militants returning from the battlefields of the Sahel to wreak havoc in the US or, more realistically, Europe. There are some reports that Canadian or even French passport-holders were among those who attacked the refinery. However, there are two reasons to be relatively sanguine.

Islamist fighters from the Islamist group Ansar Dine in Mali. Photograph: AP

First, the facilities available for training in the region are minimal and there would seem to be no reason why extremists graduating in terrorist studies from there would be better able to carry out effective mass casualty attacks than men such as Merah.

Second, we are yet to see a wave of violence involving veterans of much more longlasting and extensive violence elsewhere in the Maghreb or the core of the Middle East. British intelligence officials pointed to the experience of the horrific conflict in Iraq when asked about the possibility of veterans of the current fighting in Syria, where extremist religious groups are playing an increasingly significant role, posing a threat to the UK. Only one attack – the abortive 2007 London and Glasgow strikes – has been definitively linked to someone involved in that previous conflict, and he was not a former fighter. Iraqi veterans have proved dangerous in Saudi, even in Afghanistan and in the Maghreb. But that is not the same as posing a direct existential threat to the west. There seems, the officials say, to be no reason why the Syrian theatre should produce a greater threat today than the Iraqi theatre has done. Nor, indeed, Mali.

Does this all mean that Islamist militancy will simply die away? Of course not. A phenomenon with such long and complex roots will evolve rather than disappear. That is what is currently happening in this new post-al-Qaida phase. Wherever the various factors that allow the "Salafi-Jihadi" ideology to get traction are united, there is likely to be violence. Extremists do, as Cameron said, "thrive when they have ungoverned spaces in which they can exist, build and plan" and the aftermath of the Arab spring has not just opened up new terrain but also exacerbated existing problems of lawlessness and criminality. Flows of arms from Libya have made a bad situation worse.

And if you take the fighting in Mali and the attack on the refinery, and add it to a list of all the incidents occurring around the globe involving extremist Islamist violence, it is undoubtedly a frightening picture.

In the last few days there were arrests in the Philippines, anti-terrorist operations in Indonesia, deaths in Pakistan (due to infighting between extremist groups), air raids in Afghanistan on suspected al-Qaida bases, battles in the Yemen, shootings and executions in Iraq following the release of a video showing brutal executions, reports of trials in the UK and Germany as well as fighting in Mali.

But does this all add up to al-Qaida 3.0, more dangerous than ever before? There's a simple test. Think back to those dark days of 2004 or 2005 and how much closer the violence seemed. Were you more frightened then, or now? The aim of terrorism is to inspire irrational fear, to terrorise. Few are as fearful today as they were back then. So that means there are two possibilities: we are wrong, ignorant or misinformed, and should be much more worried than we are; or our instincts are right, and those responsible for the violence are as far from posing an existential threat as they have ever been.

• This article was amended on 29 January 2013. The abortive attacks on London and Glasgow took place in 2007, not 2006 as originally stated.