The naval base at La Spezia in northern Italy is in an advanced state of decay. The grand Mussolini-era barracks are shuttered; the weeds won their battle with the concrete some time ago. But amid the crumbling masonry, there is an incongruously neat little building, shaded behind a line of flags, with smartly outfitted security men behind its glass doors. This is Nato’s Centre for Maritime Research and Experimentation (CMRE). As one battleship after another has been removed from what remains of the Italian navy, and the base is wound down, the centre is preparing for a new kind of marine warfare amid the wreckage of the old.

In a line of workshops along the quay, technicians tinker at the innards of the next generation of naval weapons. They may look like large bright yellow torpedoes, but they are in fact underwater drones, capable of being remote controlled on the surface and taking autonomous actions in the deep. Several will be able to stay submerged for months, eventually for years, only surfacing to report an encounter with an enemy submarine.

The CMRE at La Spezia is not alone in this field. It has far bigger, better funded – and much more secretive – counterparts in the US, Russia and China. But the technicians here insist they are working on the state of the art.

Facebook Twitter Pinterest The Nato research vessel Alliance. Photograph: CMRE

Marco, a wiry Italian-American in jeans and a white T-shirt (he cannot give his surname for security reasons), was working at a US Navy robotics workshop in Connecticut when he was seconded to La Spezia. He had never heard of CMRE and was surprised by what he found. In their laid-back, Mediterranean manner, the small mixed team of specialists on the Ligurian coast between Genoa and Pisa have made some striking advances. “No one knew what was going on here. They were very quiet about it,” Marco says. “Because it is smaller, with a tight group of people from across Nato, there is far less red tape.”

Marco is working on a partially disassembled robot on a trestle, its innards laid out on a bench in a row: motor, on-board computer and sonar equipment. The basic model is made by Florida Atlantic University for research and oil rigs; Marco and his team take these apart and improve them, giving them new propellers and batteries that allow them to stay submerged for longer, and a computerised brain that will enable them to hunt for submarines in deep water.

For half a century, big missile submarines, known as boomers, have been arguably the most decisive weapon systems in modern warfare – the queen on the strategic chessboard – because of their capacity to remain unseen until the critical moment, unleashing enormous destructive force without warning. Now that dominant position is under threat. A submarine can hide from a few noisily obvious ships and planes, but it is harder to hide from a swarm of small, virtually undetectable drones. The robots being developed here can potentially be made cheap and expendable, and capable of being deployed in large numbers to cover vast expanses of sea. Once fully developed, they could tilt the balance of power beneath the waves – much as airborne drones are already doing in the sky. It is unclear how far other countries have got with underwater drone technology; it is known that the Russian navy is working on it intensively.

Facebook Twitter Pinterest An engineer on the Nato research vessel Alliance. Photograph: CMRE

The implications of these advances are far-reaching for all military powers, but none more so than the UK, which depends on the invisibility and stealth of submarines for its Trident nuclear missiles. The government is in the process of placing a £31bn gamble that its submarines will stay invisible for the foreseeable future – a bet that might be splitting the Labour party but is little debated outside it. Yet these developments could drastically change the debate: from whether an independent British nuclear deterrent is good, bad or necessary, to whether Trident would even function as a deterrent in the long term.

***

Hovering above all this is arguably the biggest threat of all – cyber warfare: the great wild card that can turn the world’s most advanced technology against itself with a few well-placed lines of code.

In a hotel in the Estonian capital Tallinn, 400 soldiers and civilians are taking part in Nato’s biggest ever cyber war game, Locked Shields. At ranks of computer screens, young men with crew cuts in camouflage fatigues sit interspersed with teams of male and female hackers in green and yellow T-shirts, most in their early 20s, many with piercings and tattoos.

In a separate conference room, a team of hackers in red T-shirts are sequestered away. They have researched the latest cyber weapons in circulation on the darknet and are throwing them at competing teams taking part here and remotely, from various Nato capitals and bases. The aim, says Colonel Artur Suzik, the Estonian army officer who is the game’s host, is “to put these people under stress”.

Facebook Twitter Pinterest The green team at Locked Shields, Nato’s biggest ever cyber war game. Photograph: Henrik Duncker for the Guardian

Estonia learned the importance of cyber defence the hardest way possible. In 2007, it became the target of the first concerted state-on-state cyber attack, when Moscow decided to show the small former Soviet republic that it was still under Russia’s shadow. The assault unfolded in late April and early May. The servers of the country’s banks were hacked, forcing them to close down all but essential operations, and move to proxy servers in Lithuania. Without a shot being fired, a nation’s entire financial infrastructure was forced into exile.

At the same time, mass text messages were sent from an anonymous source to Estonia’s Russian-speaking minority, telling them to drive very slowly through the city centre at a certain time of day. The drivers kept moving, so technically no crime was committed, but it brought Tallinn to a virtual standstill. Then the telephone numbers of vital government services all started ringing at once, nonstop, as they were swamped by robot calls.

It was a frightening early display of the vulnerability of modern societies to this new form of aggression. It was also an example of what has now become known as hybrid warfare, in which the dividing line between war and peace is blurred, and acts of war are shrouded in ambiguity and deniability. Since 2007, there has been a steady drip-drip of small probing attacks on Estonia, continuing at a low enough intensity so as to be absorbed into daily life. “It is a constant background noise,” says Hillar Aarelaid, who was chief of Estonia’s cyber defences in 2007. “It’s like being by the sea and hearing the waves.”

In anticipation of the next big attack, Nato’s Cooperative Cyber Defence Centre of Excellence has been set up in Tallinn, and links have been established between the military and a civilian infrastructure. Most of the Estonian computer experts and hackers currently sitting in this hotel ballroom would be called up in a crisis.

Facebook Twitter Pinterest Coloured strands show the game’s changing patterns of attack. Photograph: Henrik Duncker for the Guardian

If the 2007 incident was destabilising, it now seems as rudimentary as a zeppelin attack. The weapons available to hackers today are far more sophisticated and powerful, menacing even the most heavily guarded networks. The Locked Shields exercise in Tallinn is designed to anticipate what the next onslaught might look like: a complex scenario in which an imaginary country, Berylia, which looks very like Estonia, comes under a surprise attack by both strategically placed explosives and an escalating cyber assault focused on its most sensitive industry, a drone manufacturer. The assailants are anonymous, but seem to be working for Berylia’s bitter rival and neighbour, Crimsonia, an imaginary state that closely resembles Vladimir Putin’s Russia.

In a corner of the ballroom, a bank of large screens shows how the game is unfolding, a graphic portrayal of what a wired society looks like when it begins to break down in the face of a well-organised cyber offensive. Within a few hours, Berylia has lost control of all its drones. Their trajectories can be seen drifting around a map of the northern hemisphere on a screen on the wall. On another screen, what looks like a big ball of multicoloured electronic wool constantly rearranges itself. Each coloured strand represents a new computer connection being made; it is a shimmering picture of the battlefield, showing the ever-changing patterns of attack.

A cyber security team from Nato headquarters in Belgium is eventually declared the winner, having built the most secure systems, and rebuilt them fastest. What we have seen in this ballroom is what warfare will look like in the future, Aarelaid tells me, standing by a row of tables offering a wide range of caffeinated drinks to keep the hackers awake and on edge. “The Arab spring started on the internet. This is the new reality. Everything from now on will be accompanied by cyber [warfare] or triggered by cyber.”

***

In an operations room at the Nato command compound in Mons, Belgium, is another bank of screens, this time depicting near-constant real-life attacks, in the form of red lines of data. Ian West, head of cyber security at Nato’s Communications and Information Agency, puts the success of its team in the Locked Shields exercise down to the experience gleaned here. “Every single day, we are operational, experiencing attacks and defending against them,” the former RAF officer says.

West’s agency logs around 200m suspicious events a week. Many of those are automatically discarded by filters, but that still leaves 250-350 serious cases each week against Nato HQ and bases around the world, each of them requiring intervention from the 200-strong multinational group of security analysts and programmers gathered here. There are many more attacks on the national infrastructures of member states.

West is an enthusiastic collector of Napoleonic memorabilia and his office houses musket balls, tunic buttons and hat badges salvaged from the nearby battlefield at Waterloo. Yet the form of warfare he now pursues is further from these relics than Napoleon was from the swords and spears of the iron age. “This is largely an unseen war,” he says. “We talk about different wars that might take place in the future – cyber attacks are something we are dealing with now. People hear about it only when something big happens, but it is because of this team, and others like it in other countries, that big things don’t happen all the time.”

In the operations room, the red lines show the nature, location and persistence of each assault. A set of algorithms has eliminated the false alarms and insignificant attempts. Scanning the rest, security analysts look through the lines of code for patterns and vulnerabilities in the system. “If someone knocks on your door once, you think, ‘That’s annoying’,” says one of the analysts on duty. “When they knock 10 times, then you wait for the bugger to come back to see what he’s doing.”

Facebook Twitter Pinterest Ian West, head of Nato’s Communications and Information Agency, in an operations room at its command compound in Mons, Belgium. Photograph: Emmanuel Fradin for the Guardian

Right now, the greatest constraint on Nato’s ability to defend itself against attack is the scarcity of security specialists. The Russian and Chinese security establishments are known to have corralled networks of hackers. In China, the now infamous Unit 61398 of the People’s Liberation Army was discovered, in 2013, to have been running an almost constant cyber-offensive against western companies and governments for seven years, from a 12-storey building in Shanghai; the offensive involved thousands of English-speaking hackers. A mass networked assault on Nato infrastructure from China two years ago is believed to have been the work of the same unit; more recently, there have been constant attacks on Nato from hacktivist groups such as CyberBerkut, backing Russian intervention in eastern Ukraine.

“Many people think of hacktivists as 16-year-old youths sitting in their bedroom,” West says, “and of course there are many like that. But when you put enough of them together and under the control of a few people who know what they are doing, they can become a formidable army.”

***

Bob is an ex-hacker in his late 20s with “Middle Eastern roots”, and lives in London. This is as much as he will tell me about his background, since Bob (not his real name) does not want his identity revealed and will only speak to me over the phone, from an undisclosed location. A Nato recruit, Bob is, in his own words, “from the darker side”.

“When I was in my teens, I was cracking games and breaking into sites,” he says. “I was good at programming and I started wondering how to make a website do what you wanted it to do. There was a certain company that had games zones, and winners would get six months’ supply of their products. I was bored and playing around, and I found a bug in the game. I gave the name and address of a neighbour, and a few days later I saw them deliver him six months of baby food.”

Ultimately, hacking became so easy that it stopped being fun; he also found that “the ethical side has a guaranteed paycheck every month, which is nice”. First Bob did security work for banks, spotting vulnerabilities; then, in 2006, a friend told him about a Nato job.

As an alliance, Nato can only play defence

As an alliance, Nato can only play defence. It can hire experienced hackers to assess its systems’ vulnerabilities, but only if they have gone through a recognised course and earned the unlikely sounding qualification of “certified ethical hacker”.

When Bob joined Nato, he did an interview over the phone and moved to Mons, where he led “red team” exercises against alliance defences, using the latest hacks being passed around the underground forums. “It was never nine to five,” he says. “I was online 20 hours a day, looking at forums to see what people are doing and what are the new threats. I worked with close incident response, figuring out attacks and learning how sites got compromised. It comes down to technique and instincts. You have to find someone who can deal with an incident under pressure, and not get paranoid and block access to normal people. The trouble with most industry and government agencies is that they don’t work with the hackers. They need their skills but don’t trust their own guys. They’re scared of them.”

Drawing on Estonia’s experience of 2007, the UK set up a Joint Cyber Reserve unit in 2013, which is intended to recruit hundreds of hackers as reservists, irrespective of whether they have been convicted of hacking offences. The aim is to sharpen the British military’s ability to bolster its defences, identifying potential “trapdoors”. The unit will also be involved for the first time in cyber attack, after the government indicated a willingness to strike back against intruders or even take pre-emptive action, compromising the networks used by terrorist groups such as Islamic State.

Last year, a new unit was formed to further update the army’s psychological operations. The 77th Brigade is modelled on the celebrated Chindits, second world war commandos who operated behind enemy lines in Burma to win over “hearts and minds”. The commanding officer, Brigadier Alastair Aitken, intends to bring in creative figures from the arts, who know about constructing narratives, with the aim of countering jihadist recruitment and radicalisation online. Using social media, it has been dubbed the “Facebook Brigade”, but it is unclear how different it will be from traditional military propaganda. “I don’t think the army have quite got their head around this new battlefield,” one military source says. “The scale of what needs to be done and what it has to change is way beyond what the 77th is meant to do.”

In the wake of criticisms that the UK’s cyber fortifications are inadequate, George Osborne announced in November that the government would spend nearly £2bn on cyber defence and establish a National Cyber Centre, led by GCHQ. As well as data collection, mass surveillance and monitoring terrorist threats, the UK is prepared to carry out offensive operations. Officials say these might include infecting and disconnecting enemy computers, or creating “real world effects” by targeting computer systems that control civilian infrastructure – the power grid, for instance.

Facebook Twitter Pinterest Lieutenant Colonel Jens van Laak and Locked Shields’ host, Colonel Artur Suzik. The war game’s aim, he says, is to put people under stress. Photograph: Henrik Duncker

Meanwhile, the national defence debate remains focused on Trident. According to former defence secretary Des Browne, Britain has not even begun to make a comprehensive assessment of its vulnerability. “When it comes to our strategic security, it appears that we are willing to bet the farm on having one undetectable and assuredly reliable nuclear armed submarine at all times,” says Browne, now vice chairman of Nuclear Threat Initiative, an arms control advocacy group. He argues that Trident’s effectiveness can no longer be taken for granted. “Cyber attacks are already able to undermine the reliability of our nuclear command, control and communications,” he says. “No longer can we guarantee that the weapons will work as we designed them to do when we reach for them.”

Critics point in particular to the Royal Navy’s decision to install a variant of Windows XP as the operating system on its missile-carrying Vanguard-class submarines. It was cheaper than the alternatives, but Windows for Submarines, as it is called, is also more vulnerable to malware as it comes off-the-shelf. This also means there are more bugs in circulation that could affect it, and every time a submarine comes to port and gets a software patch, it is newly vulnerable.

But the Ministry of Defence insists that Trident “remains safe and secure. Submarines operate in isolation by design, and this contributes to their cyber resilience. We take our responsibility to maintain a credible nuclear deterrent extremely seriously and continually assess the capability of our submarines to ensure their operational effectiveness, including against threats from cyber and unmanned vehicles.”

Peter Roberts, a former Royal Navy officer now at Royal United Services Institute, tells me that British technicians are well aware of the potential software vulnerabilities and have instituted special safeguards. He says predictions of the submarine’s demise as a stealth weapon are premature.

“None of this anti-submarine technology has been perfected,” he says. “And what you are not able to do with drones is get them to work together, because of the problems of communications underwater. I can’t see a breakthrough in the next 15 years, and you are never going to see the whole ocean. We are talking about a water space that covers two-thirds of the world’s surface. This is not a needle in haystack. It’s way beyond that.”

***

In La Spezia, the technicians are working on overcoming these current limitations. One is how to sustain autonomous robots under the sea for long periods, but that is close to being cracked, with combinations of solar and the power generated by a drone’s descent through the water. The next big issue will be getting the drones to “see” and communicate through the oceans. Experiments are under way into laser-based techniques for finding objects underwater; for the time being, the lemon-coloured robots in La Spezia use sonar like their manned precursors.

Operating in “blue water”, the open ocean, is a challenge, but Kevin LePage, the American principal scientist at CMRE, believes it is just a matter of time. “I think it will be used in blue water. The technology is completely applicable.”

The trouble with most industry and government agencies is they don't work with the hackers

His Portuguese colleague, Emanuel Coelho, adds that the capabilities of anti-submarine drones will be amplified by intelligence. Big submarines can be picked up in coastal water and then tracked by relays of drones and aircraft. “Submarine forces are getting very nervous,” Coelho says, “because they never know when they are being detected or not.”

The renewed Trident deterrent is not due to be in service until the early 2030s. By then, the oceans are almost certain to be swarming with anti-submarine drones from many rival powers, with far higher capabilities than the ones being trialled at La Spezia. “Trident is old technology,” says Paul Ingram, executive director of the British American Security Information Council, based in London and Washington DC. “The submarines are big, they’re expensive, with very long lead times. The technology chasing them will be will be 30 or 40 generations on by the time they hit the water.”

He believes the uncertainties about Trident are just one part of a much bigger revolution, in which behemoths such as missile submarines and aircraft carriers are being outpaced. In this new world, technology is largely developed in the civilian sector, where the pace of innovation is fast and could be available to small nations as well as terrorist groups. In other words, history could be on the side of the swarm.