Five years ago, Rob, a 38-year-old father of two, was fitting kitchens and bathrooms for a living. Now he is a digital spy. As one of GCHQ’s army of cyber analysts, he monitors global counter-intelligence targets in countries he cannot disclose for national security reasons.

“You’re always looking for that key or that nugget that’s going to really help progress the operation,” he says, before adding proudly that his work often makes the headlines.

“I could be sitting eating my tea at home, and the news will be on, and I want to turn around to my wife and say, ‘I helped on that one.’ It feels great. But then you realise you can’t share it with anybody outside of work.”

Rob is not your typical GCHQ intelligence officer. Although he began his career in the Royal Air Force, he quit the military in 2007 to become a builder. It was only when he saw a recruitment ad on the back of a bus in Cheltenham — home to GCHQ’s gigantic circular headquarters — that he decided to join Britain’s biggest intelligence service.

At first he was conscious that his background was very different from that of many of his colleagues. “One of the things I felt on day one . . . I was in a room full of graduates — very, very clever graduates — maybe a couple of people who had come from industry and then there was me, a builder. I didn’t feel inadequate but I felt a bit of an outsider.”

As the centenary of its creation approaches, the UK’s largest intelligence service is having to rethink the way it recruits the spies of tomorrow. Faced with sweeping technological change and a rapidly evolving security threat, GCHQ now plays a far more wide-ranging role in the everyday safety of British citizens than was probably ever imagined when it was founded in 1919 as the Government Code and Cypher School.

Despite being one of the UK’s most secret organisations, GCHQ agreed to allow me to spend a total of six days inside its headquarters in Cheltenham and two of its other highly classified sites at Scarborough, in North Yorkshire, and Bude, on Cornwall’s north coast.

Over the past year I have interviewed 20 people, the majority of whom used only their first name or a cover name to protect their identity. At all times, I was escorted by members of the agency’s press and security staff.

The picture that emerged is of an organisation still heavily bound up in its traditional work of secretive code-cracking and surveillance, but also braced for another wave of technological change that is thrusting it and its staff of 6,000 people into the spotlight.

As the nature of intelligence work becomes increasingly digital, GCHQ is no longer a passive collector and distributor of intelligence, but is transforming into a key player in offensive combat operations.

“In the past, you could characterise what we did as producing pieces of paper which we handed to government who could take action,” explains Tony Comer, GCHQ’s historian and one of just seven people allowed to speak publicly on its behalf. “Now we are the ones actually taking the action.”

Almost three decades after the birth of the world wide web forced GCHQ to rapidly shift from cold war-era listening posts to a digital surveillance and security service, the arrival of artificial intelligence and machine learning, the internet of things and the sheer scale and complexity of modern online communications is upending the agency again, forcing it to rethink how it delivers its expanding mission.

It took 15 years for GCHQ to work out that the arrival of the public internet was a bigger event [than the end of the cold war]

At the same time, the advancing cyber threat posed by hostile states such as Russia, North Korea and China, as well as international criminal gangs, is pushing GCHQ further into the front line.

In the coming months, Britain will launch a new offensive cyber force, made up of more than 2,000 people, which will build significantly on existing powers to initiate online operations that can degrade or destroy computer networks and have real-world effects, such as turning off energy grids or water supplies. While no decision has yet been made public, the force is expected to be led by GCHQ.

“Look at the nature of warfare today,” says Sir David Omand, a director of GCHQ between 1996 and 1997. “It’s almost inconceivable that any armed conflict won’t have some form of cyber dimension.”

GCHQ, Cheltenham. The agency was founded in 1919 as the Government Code and Cypher School. As intelligence work becomes increasingly digital, GCHQ is no longer a passive collector and distributor of intelligence, but is transforming into a key player in offensive combat operations © Getty

Last year, Jeremy Fleming, the agency’s current director, took the unusual step of revealing it had conducted a major offensive cyber campaign against Isis, suppressing the Islamist terror group’s online propaganda machine and hindering its ability to co-ordinate attacks.

While he did not disclose details of how the agency achieved this, he hinted that specialists from GCHQ had used new cyber capabilities to deny terrorists’ online services and destroy equipment and networks.

Earlier this month, a Channel 4 documentary went further, disclosing how sound analysts at GCHQ had identified the British-born Isis terrorist Mohammed Emwazi, known as Jihadi John, by matching disguised voice recordings from a 2014 video to British intelligence files.

Both examples serve to illustrate the growing power the state can now command through remote online spying methods. Yet in the face of this digital transformation, one pressing question for GCHQ is how to recruit the best people ahead of intense competition from Silicon Valley and the City.

“We just can’t compete with Apple or Amazon [in terms of pay],” Elizabeth, a senior recruitment officer, tells me. “So a lot of our focus is on recruiting people with the right motivation.”

Until about 25 years ago, Britain’s spying agencies were kept largely under wraps. When parliament passed the 1994 Intelligence Services Act, it was the first official acknowledgment that MI6, the foreign intelligence service, and GCHQ even existed.

Until then, gaps in the public’s understanding were filled in by novelists such as Ian Fleming, the creator of James Bond who worked for British naval intelligence, and John le Carré and Graham Greene, who both spied for MI6. More recently, hit TV shows such as Spooks and Bodyguard have dramatised the work of MI5, the domestic security service.

When I have asked senior officials from these two agencies whether there is any truth in such portrayals, they often laugh but stop short of saying they are completely made up. For top-secret organisations, it can pay to play along. Whenever a James Bond film is released, MI6 job applications rise.

In 2017 Alex Younger, the current MI6 chief, even took the unusual step of writing a letter to The Economist, saying he favoured the “quiet courage and integrity” of Le Carré’s central spying character George Smiley over “the brash antics of 007”.

Compared with the glamorous, mysterious world of MI6, GCHQ’s more technical work has offered fewer opportunities for fiction or drama. The organisation is still best known for the story of Enigma and how a heroic, eccentric bunch of mathematical geniuses led by Alan Turing cracked Nazi Germany’s encrypted military messages.

“In an organisation of people who are deeply introverted, the work we do is difficult to dramatise in an interesting way,” admits Comer, who joined in the 1980s as a linguist, still a very typical route into GCHQ.

“Our starting point is radically different from people who work in humint [the spying buzzword for human espionage],” he says. “If you work in human intelligence, you have got an individual who has access [to information]. And either because that person is venal or idealistic, you will play on that person to deliver the information you want.”

By contrast, GCHQ “is about understanding the technology of communications: how do they work? How might we exploit that signal? And how can we access that modem or app if we ever needed to?”

© Pâté

The history of GCHQ can be broken into three distinct periods. It was formed in 1919 to exploit Britain’s successes in intercepting German radio and telegram signals during the first world war. The second world war brought the industrialisation of decryption and surveillance. Then, after the Allied victory in 1945, the agency’s best minds and growing technical capabilities were switched to fighting the cold war, combating Soviet intelligence and intercepting Russian radio signals.

The organisation, which had grown out of the UK’s military establishment, became, over time, a more civilian operation, assuming the name GCHQ (Government Communications Headquarters) from 1946, and moving to its first official base in Cheltenham in 1951.

Almost 40 years later, the fall of the Berlin Wall would precipitate a deep identity crisis. As the Soviet Union collapsed, the onset of the digital revolution forced a complete overhaul of methods and techniques.

Although GCHQ had been tapping into data flows and collecting satellite communications long before the arrival of the internet, the agency was slow to understand the scale of the change it was confronting, says Comer. “It took 15 years for GCHQ to work out that the arrival of the public internet was the bigger event [than the end of the cold war].”

As Richard Aldrich points out in his 2010 history of GCHQ, the growing dependence of banks and businesses on secure computing created a pressing demand from government ministers for greater online security for citizens. This was something GCHQ was uniquely well placed to advise on but it ran counter to the needs of a secret intelligence-gathering organisation seeking to exploit those weaknesses. “For GCHQ, this was a paradigm shift,” Aldrich writes.

The other huge change was the types of people who might be useful to the agency in the age of the internet. Over the past three decades, cyber security specialists and hackers have emerged as the next generation of codebreakers.

GCHQ admits it cannot compete with the private sector when it comes to pay, although its entry-level salary for graduates of £26,500 is not hugely out of line with other sectors such as technology or even banking, according to data from the Institute of Student Employers published last year. The broader challenge is how to retain these employees as they develop their skills.

Senior directors say they have to appeal to a sense of national duty to keep hold of the best talent. “Attraction isn’t the problem,” says Fleming, who is the agency’s 16th director and its most senior spymaster. A former MI5 deputy director, he took over from Robert Hannigan in 2017.

“The brand is so strong but we are in competition and, certainly, when you are looking at the very high-end skills areas, we can have difficulty around difficult languages, around some aspects of cyber security and some high-end aspects of engineering.

“I don’t have all of the same levers that a private-sector organisation has,” he continues, “particularly around pay. But I have loads of levers that they don’t have. People come here because they want the mission and to feel part of something special.”

This year, GCHQ is looking to add between 600 and 800 new people, driven mainly by the expansion of its National Cyber Security Centre, which opened in 2016 and provides advice and support for the public and private sector on how to deal with cyber security threats. Next year it will also open a new base in the centre of Manchester.

Although the agency hit its targets for recruitment in 2016-17, hiring 567 people against a target of 550, the year before that it fell short, hiring 490 against a target of 640. At the same time, the churn of staff, including people leaving for higher-paid private-sector jobs, is about seven per cent each year, say agency recruitment officials.

© Pâté

One of the ways GCHQ hopes to attract new hires and convince existing staff to stay is by publicising its willingness to embrace people from a wide range of backgrounds. In 2015, the Cheltenham “doughnut” was lit up in rainbow colours to support the LGBT community, and the agency is trying to recruit more black and Asian staff. It also actively seeks people who class themselves as neurodiverse, with conditions including autism, Asperger’s syndrome and learning difficulties such as dyslexia, dyspraxia or dyscalculia.

Officials believe people with such conditions can approach difficult problems from completely different angles. GCHQ even has a well-established club, the Think Differently group. Its chair, Mike, a senior project manager, says he landed his current role despite being both dyslexic and dyspraxic and having problems with his short-term memory.

Mike says his memory problems can turn routine tasks, such as remembering how to get to a certain meeting room, into an ordeal. But he adds: “I’m great at problem-solving.”

We don’t just take the top 10 out of Cambridge. We want mathematicians who know how to make maths useful and that’s a wider set of skills

GCHQ aims to recruit up to 15 mathematicians a year to join an elite unit tasked with solving some of the agency’s most perplexing problems: cryptography, cyber security and how to analyse enormous data sets. The legend of Alan Turing still attracts new talent, officials say, even if they quickly learn that some of their breakthrough discoveries may take years to come to public attention.

In 1973, Clifford Cocks, a member of the maths community at Cheltenham, developed the public key encryption algorithm that forms the basis of the secure internet today. It was not until 1997 — some time after the technology had been developed by other scientists — that the classified information about Cocks’s earlier discovery was made public.

Meg (not her real name) is one of GCHQ’s current crop of maths specialists. Nearly 50 years after Cocks’s discoveries, she and her colleagues are developing encryption solutions to protect online communications from powerful new quantum computers, which could soon render today’s methods obsolete. “It’s not just a threat to our communications,” she says, “but to the whole world.”

Meg joined GCHQ after graduating with a maths degree from Durham University in 2007. Now she is part of the team involved in identifying future maths recruits. “We don’t just take the top 10 out of Cambridge because that’s not necessarily the cohort we want,” she says. “We want good mathematicians who know how to make maths useful and that’s a wider set of skills than just maths.”

The importance of maths to GCHQ is underlined by her boss. “I will continue to be one of the nation’s biggest employers of mathematicians,” says Fleming. “I am really sure of that.”

The journey from the centre of Bude to GCHQ’s station on the rugged edge of the north Cornwall coast takes you up and down a breathtakingly steep, single-track road. Reaching the end of the final climb, my taxi emerges, gears screaming, into open terrain revealing the incongruous sight of almost 30 giant satellite dishes sitting slap-bang in the middle of wild countryside.

GCHQ bought the site, a former military base, in 1969, just as the first commercial telecoms satellites were being launched. Its proximity to the Goonhilly satellite receiving station, once owned by BT and just 100km along the coast on the Lizard peninsula, was one reason for its development.

But the lack of signals interference offered by its coastal position and the clear view across the Atlantic to the US eastern seaboard also played a part. GCHQ’s Bude base has a history of close collaboration with the National Security Agency (NSA), providing it with raw intelligence as part of the Five Eyes alliance that also includes Australia, New Zealand and Canada.

Holidaymakers who come here for the beautiful scenery and sandy beaches might be surprised to learn of the significance of Cornwall to the UK’s spying agencies. As well as Bude, the county provides the main landing points for some of the undersea cables that carry the majority of the world’s international internet traffic.

GCHQ’s base in Bude, Cornwall, with its 29 satellites. An employee who maintains them says: 'While powerful satellites are slow to set up, expensive and difficult to maintain, they remain an important tool in GCHQ’s arsenal' © Getty

Clem, a former propulsion technician in the RAF who joined GCHQ 40 years ago, is responsible for maintaining the 29 dishes and protecting them from the high winds and salty air that batter them on a daily basis. Compared with other modern collection methods, satellite dishes are expensive. The biggest one at Bude is 32 metres high and cost £3m.

By now, he admits, GCHQ might have expected to be winding down its satellite-interception operations as radio signals become less important and more and more internet data travels via submarine cables. But one of the challenges the agency faces is the sheer complexity of how information now moves around the world.

A terrorist target being tracked by GCHQ may move not only between different web applications such as WhatsApp or Signal and multiple electronic devices, but also from 4G satellite networks to underground cables. A single packet of crucial information could take dozens of routes before it reaches the recipient.

To give the intelligence agencies the best shot at intercepting the information therefore requires a wide range of collection methods. “While powerful satellites are slow to set up, expensive and difficult to maintain, they remain an important tool in GCHQ’s arsenal,” says Clem.

The next generation of GCHQ spies will need a completely different range of intelligence weapons if they are going to keep up with the rapid advancements in technology. Fleming says the agency is well placed to deal with this.

“There are some threads you can follow from 1919 right through to today,” he argues. “And those threads are about using technology in our workspace and being really innovative around analysis. Technology is at the heart of society in a way it hasn’t been at previous points in our history.”

Some months after my trip to Bude, I am back behind the barbed-wire fences at Cheltenham to meet Paul, a senior director who leads a team looking at how GCHQ develops its next-generation artificial intelligence capabilities.

He and his colleague Steve have created a new programme that uses machine learning to identify the best internet access points for collecting data. It is based on the open-source algorithms developed by Google’s artificial intelligence arm DeepMind to teach a computer how to win at chess. If successful, it could transform the way GCHQ gathers internet data.

At the moment the decision on where to position the bearers, or “clips”, that enable the transmission of data between networks, is taken by GCHQ operatives, depending on the location of the target and the type of device and network they are using.

But, taking a lead from the way DeepMind taught its algorithms what winning looked like, thereby making the computer’s chess tactics less predictable and more human-like, GCHQ hopes that eventually the system will learn the most productive places to harvest communications metadata (the location, time and type of message rather than the actual content, which is commonly protected by end-to-end encryption).

One of the ways GCHQ hopes to attract new hires is by publicising its embrace of people from a wide range of backgrounds

Paul explains: “What the machine can learn is that there are patterns that are better than me just plugging into the internet and hoping. From the computer’s perspective, it’s playing and it knows what good looks like. It knows intelligence reports are being generated [from the data collected] and it is trying to create different rules and guess better than a human would ever do.”

GCHQ says the project, which has been running for six months, requires much more testing before it can be rolled out. Despite what Paul and Steve describe as promising results, the move would raise difficult questions around the potential for humans to be taken out of the decision-making process on where to intercept data.

Steve says that before such a programme is introduced, it has to be signed off by GCHQ’s legal and policy directors to “fully understand what the position is around a machine taking a decision [of where to tap into the internet] . . . Right now, there is a person who will say, ‘OK, that’s the recommendation, I will do it.’”

“There’s a debate to be had,” adds Paul. “But I think many members of the public would probably find the idea that we were minimising intrusion to privacy and maximising the chances of finding bad people by automating a very technical collection system probably quite attractive.”

GCHQ is also exploring ways of using AI to process enormous biometric data sets, which could be a big advantage in helping identify patterns of behaviour among targets and criminal suspects. At the same time, new biometric technology is killing off old spycraft techniques.

Paul cites an example of a UK developer who has created a programme that can identify people from the small “envelope” of skin beneath the eyes. Software like this might, he says, be able to identify a person even if they are wearing a balaclava.

More generally, CCTV and other increasingly common facial-recognition technologies mean the James Bond approach of slipping across international borders with a false name, false passport and a bunch of whizzy gadgets is becoming virtually impossible.

The point was well demonstrated last year by the sensational unmasking of two Russian military intelligence officers accused of carrying out the nerve-agent attack on the MI6 double agent Sergei Skripal.

The attack, which the UK says was carried out with the approval of the Kremlin, was shocking not only for the ruthless and reckless use of a prohibited chemical weapon on the streets of Britain, but also for the brazen lack of care the two Russian intelligence officers displayed when it came to concealing their identity.

Increased computer power means that even if spies calculate they can evade detection now, future technologies may be able to identify their activities. “Human beings can’t easily pretend to be something they’re not,” says Paul. “You can fool other human beings but it’s much harder with a computer.”

GCHQ’s growing powers and higher profile over the past 20 years have inevitably brought the agency into conflict with privacy campaigners, who argue that the bulk collection of computer data and communications is a potential breach of human rights.

In 2013, former NSA officer Edward Snowden revealed the existence of a GCHQ programme called Tempora, which enabled the bulk storage of all internet traffic the agency could harvest. Since then, legal challenges through the UK and European courts have sought to check the agency’s mass monitoring of public communications.

Last September, 14 human-rights and investigative-journalism groups won a legal challenge at the European Court of Human Rights against the UK’s previous legal system that oversaw bulk interception.

The victory prompted Snowden to tweet from Russia, where he has claimed asylum: “For five long years, governments have denied that global mass surveillance violates your rights. And for five long years, we have chased them through the doors of every court. Today, we won.”

The UK government has taken steps to restore public trust in the intelligence services since the Snowden revelations, including the 2016 Investigatory Powers Act, which requires all spying agencies to now seek ministerial and judicial approval before targeting someone’s data. But privacy campaigners Liberty say the steps still allow the state to “hack computers, phones and tablets on an industrial scale”.

“The government must urgently reassess the invasively wide powers it has to snoop on our lives, and develop a proportionate surveillance regime that better balances public safety with respect for privacy,” says Megan Goulding, a Liberty solicitor.

Ask those who work for GCHQ what they think of Snowden and other whistleblowers such as Katharine Gun — who, in 2003, revealed how the NSA hacked members of UN delegations in the run-up to the Iraq war — and their view is pretty unequivocal.

“[Snowden]’s an idiot. An absolute idiot,” says Rob, the career-changer. “He set all the progress GCHQ made against fighting the bad people back years and gave them the upper hand again.”

“What Snowden did was very bad,” adds Comer. “Targets we were interested in, we lost track of completely.”

Everyone I spoke to at the agency insisted it is not capable of hoovering up the vast quantities of data and online communications that campaigners say it can collect and store. “The public perception is that we’re listening in on everyone’s phone calls, browsing everyone’s internet activity and reading everyone’s messages, which is not the case at all,” says Will, a 25-year-old trainee analyst.

Does he ever feel guilty snooping around someone’s digital profile? “No. Because everything you do is for a reason. I trust in the organisation and the wider government to be correct in that process.”

Steve, a senior security official who sits in on a number of my interviews to ensure no one breaks the UK’s Official Secrets Act, chips in: “People think we’re cowboys and we’re not.”

Anyone wanting to join GCHQ must face an initial interview designed to test their knowledge of technical subjects — such as how a smartphone works or the cyber security risk posed by antivirus software.

Back in Bude, I meet a group of the agency’s interviewers, whose job it is to find the spies of tomorrow. They explain that they are seeking people who want to look inside software systems and devices and really understand how they work. Regretfully, they admit many applicants lack the depth of technological understanding and computer-engineering skills so badly needed to meet GCHQ’s growing cyber demands.

“They are consumers of technology,” says Pete, one of the interviewers. “People come in thinking their technical competence level is sometimes a lot higher than it actually is. Many years ago, if you knew something about the internet, you knew something about the underlying internet. In this day and age, you probably know a lot more about superficial services like Twitter.”

The [deeply intrusive] vetting is the first hurdle on an endurance course. People don’t understand that it carries on through your life of working in the intelligence services

Perhaps the most searching question is left to the very end of the interview. All would-be recruits are asked: “As GCHQ is a British intelligence agency, you will inevitably come into contact with sensitive material which may test your ethical or moral conscience. How do you feel about that?”

The interviewers say most candidates fully understand the type of organisation they are joining. But if they didn’t, the deep vetting process that follows, which can take up to a year, acts as a sobering reminder of the commitment they are making.

The process is deeply intrusive, asking searching questions about a candidate’s sexual history, friends, family and financial situation. Anything that might compromise a potential GCHQ employee can mean the end of that person’s application.

“It’s fair to say when I joined, my vetting officer knew more about me than anyone else alive,” says Meg, the mathematician. “It was not nice discussing your sexual history with a stranger. Nobody likes doing that, especially when you are 22.”

Maria, one of GCHQ’s vetting officers, tells me that the process doesn’t stop once someone joins the organisation. It carries on throughout an officer’s career at the agency, with staff revetted every seven years. “It’s [just] the first hurdle on an endurance course,” she says of the first deep-vetting round. “That’s one of the things people don’t understand. It carries on through your life of working in the intelligence services.”

Her colleague Al adds that strict zero-tolerance policies around drugs can also put people off: “They are making a bit of a lifestyle commitment when they join. From the day you sign and accept that, you accept you will never take drugs again.”

The culture of secrecy can also be a deterrent. Staff are told never to discuss what they do at work even with their closest family and friends. “My parents found that very difficult,” admits Meg. “They didn’t know what they were and weren’t allowed to say. One year they wrote their annual Christmas letter for friends and family and while they had a big paragraph on each of my sisters, all they had about me was one line saying I was fine. It was as if I was a heroin addict or something.”

Doesn’t that just add to the intrigue, I ask, wondering if secretly she enjoys the unique status of being a spy. “But I don’t feel that word applies to me,” she replies. “Even when I did work in the Sigint (signals intelligence) mission I wouldn’t have described myself as a spy.”

It’s a line I hear again and again from the many people I meet inside GCHQ. “The vision in many people’s minds is the James Bond stuff but that’s not what we do,” explains Steve, the senior security officer. “We’re not special people. We’re normal people doing our special jobs.”

Yet as our lives become increasingly interconnected through 5G mobile networks and the internet of things, the role of GCHQ and spycraft is becoming much more important. The international row over the Chinese telecoms group Huawei and its potential involvement in building Britain’s 5G infrastructure is one more timely case in point.

Paul believes that if GCHQ is to successfully meet the enormous challenges facing the intelligence agency, it has to move away from the idea of single individuals cracking problems towards teams of technologists working together to solve seemingly impenetrable digital puzzles.

The stakes, he says, have never been higher. “Ultimately this is about us being better than the terrorist and being better than Moscow. If we’re not better than they are, they will overcome our defences.”

Sample questions from the GCHQ maths entrance test These three questions are drawn from a test that candidates at GCHQ’s assessment centres sit. It is one of several exercises designed to allow people to show their mathematical aptitude. Click here for the answers. Q1 — Alice and Bob play Rock, Paper, Scissors until one or the other is 5 wins ahead. They generate their moves at random, so, in each round, “win”, “draw” or “lose” for a given contestant are equally likely outcomes. After 10 rounds, Alice is 1 win ahead. After 13 rounds, one or the other is 1 ahead. At round 20, one of them attains the 5-win lead and the game ends. What is the probability that Alice is the ultimate winner? Q2 — 353, 46, 618, 30, 877, 235, 98, 24, 107, 188, 673 are successive large powers of an integer x modulo a 3-digit prime p. Find x and p (without using a computer). Q3 — (X1, Y­­1), (X2, Y2), . . ., (Xn, Yn) are n ordered points in the Cartesian plane that are successive vertices of a non-intersecting closed polygon. a. Describe how to find efficiently a diagonal (that is, a line joining 2 vertices) that lies entirely in the interior of the polygon. b. Repeated application of this process will completely triangulate the interior of the polygon. Estimate the worst-case number of arithmetic operations needed to complete the triangulation.

David Bond is the FT’s security and defence editor

Follow @FTMag on Twitter to find out about our latest stories first. Subscribe to FT Life on YouTube for the latest FT Weekend videos