Last Tuesday, in the wake of the latest terror atrocity to strike Britain, the former head of MI5 Dame Stella Rimington recalled just how primitive intelligence gathering used to be. Addressing a conference of security officials in west London – four miles from London Bridge where the terror attack had taken place three days earlier – Rimington recounted an anecdote about how her spy training in the 1970s involved infiltrating a local pub to eavesdrop on targets.

Over the four decades since then, intelligence gathering within Britain’s security services has evolved beyond comparison. Eking out a lead is no longer an issue – instead extraordinary volumes of information are relentlessly harvested electronically. The worry, according to experts, is whether they are acquiring too much.

The information-collecting machine grew even larger when the Investigatory Powers Act passed with little fanfare last November, handing UK intelligence agencies a comprehensive range of tools for snooping and hacking unparalleled in any other country in western Europe, and even the US.

What is already clear following last Saturday’s attack, during which three attackers killed eight people and injured almost 50 in an eight-minute rampage, is that Britain’s security services had collated a surfeit of reliable, well-sourced material on the perpetrators. Of the London Bridge attackers, Khuram Butt, 27, had been reported to the anti-terror hotline in 2015 and investigated by MI5 for his highly public ties to the banned al-Muhajiroun network.

Another, Youssef Zaghba, 22, was interrogated by Italian police, who told UK intelligence he was at risk of radicalisation. He was also added to the Schengen Information System, an EU-wide database that gives UK police details of 8,000 jihadis in Europe.

The pattern was repeated in the two attacks that preceded the latest atrocity. The suicide bomber Salman Abedi, 22, who carried out the Manchester attack was known to MI5 and categorised under its prioritisation matrix as P4 – priority 4 – which denotes suspects who might be at risk of re-engagement but are deemed not to be planning an attack and therefore downgraded as a security risk.

Khalid Masood, 52, who carried out March’s Westminster Bridge attack using an almost identical modus operandi to the London Bridge attack, was also classified in the P4 tier at the time, essentially regarded as an Islamist but not a threat.

So why did they all slip through the net? Some security experts warn of an analytical deficit in the heart of the government’s intelligence infrastructure, claiming a lack of human resources to decode and contextualise the myriad snippets of information, terabytes of chatter, tipoffs, sightings and wiretaps that cumulatively help to form the modern intelligence picture.

The victims of the London Bridge attack. Photograph: AFP

Although the immediate political fallout of the London attack focused on Theresa May’s cuts to policing, reductions in the number of staff who analyse intelligence is perhaps the area most deserving of scrutiny. Professor Philip Davies, director of the Brunel Centre for Intelligence and Security Studies, believes the UK’s security apparatus is suffering from what those in signals intelligence call information overload.

Davies said: “The good news is we’ve got lots of information, but the bad news is that we have got lots of information. We’ve always known GCHQ [Britain’s electronic surveillance agency] struggled with information overload but I think we’re going to have to be realistic that MI5 and SIS [MI6] are being confronted with information overload in terms of scale and complexity.

“We have been cutting national analytical capability for 20 years. The collection of information has increased but if you cut back on analysis you get overload. If you have tens of thousands of reports, thousands of subjects of interest, you’re going to need a lot of Mark 1 human brains making sense of this who can get inside the mind of an adversary who has a different worldview. It’s about getting inside the enemies’ doctrinal loop. What are they thinking? How are they planning?”

Although a detailed breakdown is difficult to ascertain, less than a quarter of MI5’s 4,000 staff are believed to be involved in analysis, and a far smaller number are sited among the 2,500 staff of MI6 – as few as 100, according to Davies. Although it is true that most of the 8,000 employees in GCHQ can be categorised as analysts, they have to evaluate a vast daily data stream that runs to countless terabytes. A single terabyte is the equivalent of 1,024 gigabytes, with a gigabyte the equivalent of 1.5 million WhatsApp messages

According to Davies, the number of analysts within defence intelligence, which has a counter-terrorism component, has been cut from 600 to 400 since the end of the cold war even as the terror threat has increased following UK military involvement in Afghanistan, Libya and Iraq. The number of analysts inside the Cabinet Office’s Joint Intelligence Organisation, which “leads on intelligence assessment and development of the UK intelligence community’s analytical capability”, currently stands at only 35.

Salvatore Sinno, global chief security architect at Unisys, which has worked on counter-terrorism with UK, European and US government institutions for more than 30 years, said that even with the combined might of algorithms, artificial intelligence technology and advanced data analytics to assist, calling the shots on potential terror suspects remained a daunting challenge. “The intelligence service deals with hundreds of phone submissions a week alone, and if you combine this with search engine and social listening analysis – a monumental data set begins to form.”

John Chirhart, federal technical director of the US security firm Tenable, warned that the human touch should never be underestimated when assessing intelligence. “Humans are still a vital part of the process because they provide something computers can’t – context.”

Paul Calatayud, chief technology officer of the US security intelligence company Firemon, said that despite the constantly evolving technological landscape that means more communications can be gathered and examined quickly, making sense of the material itself remains a primary obstacle to security.

“The biggest challenge facing intelligence agencies is the vast amount of information they have to process. Data is siloed and collected at various stages; it has to be moved before it can be processed. Limited talent and resources also play a role. Collecting is the primary challenge, but when this information is collected, the real work begins.”

In other words, the process of prioritisation. Another shared fact from this year’s three UK attacks is that at some stage an active decision was taken to downgrade the perpetrators to the lowest of the four tiers that rank MI5’s current list of 3,000 subjects of interest. At the apex of the pyramid of cases known to the intelligence agencies is the category P1 – priority 1 – which is reserved for individuals linked to information indicating “attack planning”. These cases – rarely more than two dozen at a time – require mammoth resources, typically 24/7 covert surveillance. New leads are picked up, the behaviour of a suspect changes. Individuals are moved up and down the chart depending on the intelligence received, a perpetual process of review that is officially discussed each Monday at Thames House, the headquarters of MI5.

Messages of condolence at London Bridge. Photograph: Carl Court/Getty Images

In the wake of the London Bridge and Borough Market rampage, MI5 has announced it will review its handling of events to ascertain whether lessons can be learned. The attack in Manchester had, according to sources, already prompted intelligence agencies to examine procedures for prioritising suspects.

Sources say that the internal inquiry will forensically look at what was known during the build-up to the London attack, whether the prioritisation structure is sufficiently flexible and what clues can be gleaned from the behaviour of the three attackers before the outrage.

Raffaello Pantucci, director of international security studies at the Royal United Services Institute, believes the intelligence agencies should re-examine their prioritisation mechanisms in relation to the ever-expanding pool of suspected assailants. Beyond the 3,000 currently on the list, a pool of around 20,000 individuals are categorised as former subjects of interest whose risk remains subject to review by MI5 and its supporting agencies.

Pantucci raised the possibility of an external panel being created to evaluate prioritisation decisions taken by the intelligence agencies on a case-by-case basis. “I wonder if a unit needs to be developed that focuses on those changes, with a range of people looking at the decisions – someone from the policing side, someone from welfare, that evaluates the judgment made.”

The data is there. But, on at least three fateful occasions, the expert analysis has gone missing.