From terabytes of chatter to snippets and wire taps, information floods into the security services. Deciding what should be given priority can be the difference between safety and attack

It is the eternal conundrum facing every intelligence official. How to filter which of the tens of thousands of snippets of information, the terabytes of chatter, tipoffs, sightings or wiretaps, to prioritise. How to decipher the weak signals of embryonic plots or telltale tradecraft of Islamist operatives. Such decisions underpin the safety of every European state.

During the aftermath of the November Paris attacks, western intelligence officials suspected that Brussels was both a probable source and target of a follow-up atrocity. The failure to thwart Tuesday’s attacks exposed serious weaknesses in Belgium’s intelligence assessment capability. Effective intelligence work requires intensive manpower, both to analyse data and to gather it. Counter-terrorism experts believe Belgium can do neither adequately.

Man named as Fayçal C charged over 'core role' in Brussels bomb attacks Read more

Yan St-Pierre, counter-terrorism adviser for the Modern Security Consulting Group, a private intelligence firm with headquarters in Berlin, articulates the scale of the relentless data challenge facing Europe’s security services. In Germany, he said, the federal police alone are bombarded with around 10,000 items of intelligence every day – email, sightings and telephone tipoffs concerning potential criminal suspects.

“If you start combining the other agencies, plus the wiretapping, the electronic surveillance, then you are talking about hundreds of thousands of terabytes of information every day on a 24-hour scale,” he says.

Sifting through such huge amounts of information requires, according to experts, a lot of skilled individuals. Belgium’s intelligence agency has 600 staff (only a third as many as neighbouring Holland) monitoring a population of 11.25 million, and the country has the highest per capita number of foreign fighters in Syria of any European nation. According to Belgium’s justice ministry, 451 fighters have left the country, of whom 117 have returned. One senior US official has briefed, however, that there could be several hundred more people who are not known.

Meanwhile, around 820 Britons have left to fight in Syria, of whom around half have returned. Yet the security apparatus monitoring them is vast compared with Belgium’s – almost 20 times the volume of people are employed in the UK to examine intelligence compared with Belgium. The workforce of the security services, MI5 and MI6, stands at around 7,000, with another 9,000 employed by the government intelligence agency GCHQ, and these, together with S015, Scotland Yard’s counter-terrorism unit, yield a total of around 18,000 individuals.

Even then, prioritising intelligence is a demanding business. Michael Clarke, former director of the Royal United Services Institute, said: “All intelligence agencies have fairly acute resource shortages; trained human resource is, by definition, short compared to the size of the problem.

“Anything that gets very high priority will be acted on immediately, but even things given a fairly high priority may get left for 48 hours or a week because there aren’t enough analysts. In the case of Belgium, information can bounce around before it is picked up.”

Following a single suspect covertly 24 hours a day requires a team of up to 36 security service officers. Even in the UK, only a finite number of suspects – surprisingly low, according to sources – can be placed under such intensive surveillance at any particular time.

St-Pierre said a lack of manpower had bred an over-reliance on technology, which is unable to decode the constantly evolving language, codes and behaviour of the latest generation of terrorists.

“Terrorists adapt,” he says. “We’re missing out on the undercover agent who has infiltrated or the double agent within the cell who is able to analyse the information and feel the more subtle inflections of terrorism, radicalism and preparation. By not having that human component, you end up with a flood of information and not enough people to analyse it. You either oversee information or do not react to it, even if only by a few hours.”

The horror of Brussels: 'My memories stop at 9.10am. When they come back, I have blood on my face' Read more

The lack of sufficient human resources and the use of technology to plug the gaps has one particularly glaring consequence: a reliance on past attacks as an indicator to future atrocities. A sudden change in tactics can sidestep all known warning signs.

“Prioritisation is based on the implicit algorithms you have as to what has happened in the past, or what you think is building out there,” says Clarke. “By definition, every time there is a radical change in tactics – as with 9/11 – it undercuts all the prioritisation that you previously had as indicators.”

St-Pierre said the dependence on algorithms meant pressured intelligence officials might miss clues. “A lot of it [intelligence] is filtered through keywords. Five or 10 years ago, you had one person in charge of 10 cases; now, that same person would be in charge of double that and, overworked and overwhelmed, things will be overlooked.”

One disturbing element of the new Islamist threat is the ability of Isis to inspire lone wolves or “clean skins” – individuals who are not known to the intelligence agencies – to commit attacks.

Clarke said: “Because they are an inspiration to amateurs, the sheer numbers that could commit acts of barbarity is an issue. The fact they are suicide bombers is problematic; 60% of all the planning [before a terror attack] goes into trying to get away afterwards. They have not great tradecraft but are popping up everywhere.”

After the Brussels attacks, we’re starting to develop a coping strategy | Jonathan Freedland Read more

St-Pierre said the recurring theme of terror suspects having been previously known to European police forces, often for petty offences, underlined the problems of prioritising individuals who had previously come to the attention of the authorities.

“They say: ‘Well, we knew about him but he was a small fish.’ If you had the manpower and resources to pay more attention to that small fish, you would be able to do more effective work. A lot of them are small-time criminals who are dismissed.”

St-Pierre recommends that Belgium’s intelligence agency should be increased fivefold to around 3,000, while other analysts believe that the £300m of extra spending to upgrade the country’s security capabilities in the wake of the Paris attacks may need to be increased further.