It was the moment spying in Britain changed for ever.

When, on the morning of July 7, 2005, people staggered out coughing and smoke-blackened from London Underground stations, the first reports suggested nothing more sinister than a power surge.

But within minutes, the roof of a bus was ripped off by an explosion in Tavistock Square, and the truth was horrifyingly clear. Four separate bombs had brought carnage to the capital. London was under terrorist attack.

At the headquarters of MI5, director general Eliza Manningham-Buller gathered her staff in the atrium and told them their worst fears had been realised.

Scroll down for video

Security: An aerial image of the British Government Communications Headquarters (GCHQ) in Cheltenham

In Cheltenham, GCHQ director Iain Lobban was stepping anxiously into the building after rushing back nearly 100 miles from London. Yet for the first few days, the authorities could only speculate, believing perhaps that a team had come into the country, planted their explosives and left. They were wrong, and forensic analysis soon showed the perpetrators were four British-born suicide bombers. No one had prepared for this – including the spies.

Then, two weeks later, it nearly happened again, raising the chilling possibility of waves of home-grown suicide bombers. The failed attack on July 21 – when only the detonators of four bombs aimed at commuters exploded and a fifth bomber escaped without triggering his device – raised another, altogether different problem: the culprits had escaped and might strike again.

It was time for the security services to respond, and so began a very modern manhunt, the basis of the digital tracking that is at the root of so much controversy today.

It was only after 7/7 and 21/7 that GCHQ fully realised that monitoring the vast flood of communication that flows around the world was the key to intelligence. It led to a new digital phase of the war on terror – and the birth of techniques later exposed by whistleblower Edward Snowden, a computer specialist who worked for the US’s National Security Agency (NSA).

Horror: A victim of the July 7 bombings when four separate bombs brought carnage to the capital in 2005

Last week, David Anderson QC, the independent reviewer of terrorism legislation, said in an official report that GCHQ should retain the powers to carry out this sort of bulk collection data, but called for new laws to oversee its use.

The bulk interception of communications, its supporters say, has proved to be crucial in the war on terror. In the decade since 7/7, only one person has been killed by terrorism in the UK. But if the authorities have a better grasp on what our adversaries including Islamic State, Al Qaeda and hostile nations are doing, it comes at a price. And the price is this: the power of computers and data is changing what many people can find out about all of us.

If you journey beneath GCHQ’s ‘Doughnut’ – the ring-shaped building in Cheltenham that houses Britain’s secret codebreakers and communication intelligence specialists – you find a cavernous computer hall, stretching to 100,000 sq ft.

The Doughnut is the descendant of Bletchley Park, where the great innovation in the Second World War had been the melding of human ingenuity with the power of modern technology to crack German codes such as Enigma.

That remains the model today. Above the computer hall, the mathematical heirs of Alan Turing face the same task of understanding a system and its weaknesses. ‘My job is a combination of maths and computer programming and just being crafty at problem-solving,’ one twentysomething mathematician explained.

GCHQ is packed with people who understand modern technology but the irony is that, until the 7/7 bombings, the organisation was bad at understanding the way the younger generation lived their lives, partly because no one was allowed internet access at work or to bring in a phone. But as the organisation became focused on finding home-grown suicide bombers, it realised it needed a system to investigate the richness of the trail people left in their digital lives, building a pattern of terrorist behaviour, and then asking a computer who else matched it.

Tracked: July 21 suspects Muktar Said Ibrahim (left) and Ramzi Mohammed are arrested on July 29, 2005

This was done using the ‘digital exhaust’ – the unique fingerprint that all users of smartphones and the internet leave behind. Exploiting this resource required a new, powerful, top-secret capability employing data, telecoms and computing at GCHQ.

It was classified to the highest level and authorised by the Prime Minister, and remained secret until whistleblower Snowden unveiled the vast extent of British and US governmental cyber-spying.

He revealed that the secret intelligence allies could tap emails, monitor internet use and map the locations of mobiles of almost anyone. But the key to the new programmes was the ability to filter the vast mass of information about internet use. For instance, take a huge pile of data about internet traffic and ask who was communicating between Britain and Pakistan. That may reduce the online population to a few tens of thousands. But if you ask the computer again to find which of these people has also visited certain extremist websites, you are down to perhaps a few hundred. And who of those had social connections to known extremists? You are now down to a handful.

Much of this could be done by looking not at the content of their communications but the ‘metadata’, the information about the origin and destination of internet traffic, rather than the content.

Joint teams of GCHQ, MI5 and MI6 operatives now work together on terrorist targets and share leads – a move pioneered after 7/7.

Whistleblower: After 7/7, new digital phase of the war on terror was exposed by Edward Snowden (pictured), a computer specialist who worked for the US’s National Security Agency

Intelligence officials are adamant that what they built works. The building of this capability at GCHQ relied on two key developments. One was the legal authorisation to get hold of the data. The other was technology.

It was only in the year or so before 7/7 that systems became available to carry out this kind of ‘data mining’. To perform this analysis, you need as much metadata as possible to be sure your target is in there and to make it easier to do the kind of pattern analysis that helps you understand what normal looks like and what makes a terrorist look different.

A system called Tempora was designed to slow down the flood of global data by storing information – an email address, a phone number or other search terms – and filter out unwanted traffic.

According to reports, GCHQ was running about 40,000 selectors and the NSA 31,000 across one data set, looking, perhaps, for emails from Russia to Ukraine that are encoded or calls to a specific number. This means that of the millions of communications passing through GCHQ each day, only a few thousand are examined by analysts who are presented with an index – like the results of a web search – from which they pick the relevant items.

Smartphones have aided the spy business enormously. In the past, placing a tracking device or a bug was risky. Now most people carry a multi-purpose device that can provide more details than just their location or what they are saying.

Analysts can build up a digital fingerprint of a target based on how they move online and what devices and identities they use. This can then be used to track them in real time without the need for a surveillance team.

The missing suspects from 21/7 had abandoned their old phones. Even so, they were hunted down through telephone records and the ‘contact chaining’ technique the US had begun using after 9/11 with its bulk collection programme of phone records. British investigators cross-referenced calls made by the suspects to known contacts in a hunt for groups of new numbers that were also contacting each other. Within 24 hours, they had worked out what phones the men were now using, meaning the bombers could be traced to within a few yards.

Two men, including Muktar Said Ibrahim, were arrested in West London on July 29. Another was held in Italy.

It was a notable success. Yet it required access to data on a scale never seen before in Britain.

Last week’s report from David Anderson was a significant moment for the spies. It provided independent confirmation that the analysis of vast amounts of internet and phone data that GCHQ pioneered after 7/7 had proved genuinely valuable in protecting the public against more terrorist attacks.

IS has learned to adapt to this new world. On his first day at GCHQ, the new director, Robert Hannigan, warned that the terrorist group presented a new challenge in terms of its use of technology. Its younger generations of followers have become more skilled at encouraging violent jihad through more clandestine means, and at hiding their trail. He also claimed there was ‘no doubt’ that foreign fighters had learned from the Edward Snowden leaks. Now some IS members are said to use encryption software that is easily available from the internet and scrambles the contents of text and voice messages. It reduces the messages to gibberish which Western code-breakers struggle to crack. The spies say they have seen their targets shift away from using communications platforms they know the state can listen in to, and instead are choosing devices that are harder to trace.

Target: Our phones can become spies – not just for intelligence agencies but also the companies that sell us advertising

If our smartphones are at the forefront of helping the authorities to keep us safe, there seems little doubt they are helping other forces too. Our phones can become spies – not just for intelligence agencies but also the companies that track us to find out what we are doing and sell us advertising. And the result is insight into our everyday lives on a scale that would once have seemed unimaginable. David Anderson has recommended greater legal safeguards over what the state can do with its new powers. But the private sector has fewer controls.

Computers and digital surveillance have transformed espionage, but they can be a double-edged sword.

They are used, for example, by MI6, the intelligence service that recruits agents, to help find its targets. MI6 might want to get into the Iranian nuclear programme and so may use computer databases to ask who has access to the programme, what their links are to the outside world, who their friends are, and how they might be contacted.

Alerts could be set so if any of those targets are travelling out of the country, the spy service can be informed immediately. The technique of analysing ‘digital exhaust’ is also being used in counter-espionage – catching spies. The Russians are reported to be returning to typewriters in their embassies.

One of the biggest concerns that spies have about Snowden is not what has been revealed to the public, but the fear that because he is now in Moscow, the Russians may have been able to get hold of his treasure trove of secrets, only part of which has ever been disclosed. That would make it easier for them to evade British and American intelligence and to find Western spies.

Once, a faked passport might have been enough to build a cover story for an MI6 officer travelling to meet agents. Now he or she might have to get through biometrics and database checks at the border. Furthermore, their fake identity will need to have a presence online, complete with a history. Judicious use of a search engine can be enough to show that someone is not who they say they are. When MI6 ran a test using Google to see how long an officer’s cover could stand up against a suspicious foreign official, the answer was about a minute.