Even though it has become a part of daily life, the internet itself remains a cloudy idea to most people. It's nebulous in a deeper way than previous leaps in home technology. Take the radio. Nobody knew how that worked, but you could picture invisible waves of electromagnetic particles arriving from the distance like the surf. Or TV… well, nobody understood that, except that it was like the damn radio only the waves were more complex and hence delivered pictures, too. There was something going on there you could picture, even if falsely. But the internet is just there. It is all around us, like the old idea of luminiferous ether. No antenna. No waves – at least, none of the kind readily understood. And it contains not just a voice or picture, but… the whole world and everything in it: pictures, sounds, text, movies, maps, art, propaganda, music, news, games, mail, whole national libraries, books, magazines, newspapers, sex, along with close-up pictures of Mars and Jupiter, your long-forgotten great-aunt Margaret, the menu at your local Thai restaurant, everything you ever heard of and plenty you had not ever dreamed about, all of it just waiting to be plucked out of thin air.

Behind his array of three monitors at the headquarters of SRI International in Menlo Park in California [the research institute originally founded by Stanford University], Phil Porras occupies a desk in the birthplace of this marvel. Inside the very large computer he gets to play with, Porras creates a network of "virtual computers". He sees the internet not in some vague sense, but as something very real, comprehensible and alarmingly fragile.

By design, a portion of the virtual ranch he surveys is left unfenced and undefended. It is thus an inviting target for every free-roaming strain of malware trolling cyberspace. This is his petri dish, or honeynet.

These are not physical machines, just individual operating systems within the large computer that mimic the functions of distinct, small ones. Each has its own IP address. So Phil can set up the equivalent of a computer network that exists entirely within the confines of his digital ranch.

Normal folk tend to use the terms "virus" and "worm" interchangeably, while the "Geek Tribe" defines them differently. The overarching term "malware" refers to any programme that infects a computer and operates without the user's consent. For the purposes of this story, the difference between a "virus" and a "worm" is in the way each spreads. To invade a computer, a virus relies on human help such as clicking unadvisedly on an unsolicited email attachment, or inserting an infected floppy disk or thumb drive into a vulnerable computer. A worm, on the other hand, is state of the art. It can spread all by itself.

The new arrival in Phil's honeynet was clearly a worm, and it began to attract the Tribe's attention immediately. After that first infection at 5:20pm on Thursday 20 November 2008, there came a few classic bits of malware and then the newcomer again. And then again. And again. The infection rate kept accelerating. By Friday morning, Phil's colleague Vinod Yegneswaran notified him that their honeynet was under significant attack. By then, very little else was showing on the infections log. The worm was spreading exponentially, crowding in so fast that it shouldered aside all the ordinary daily fare. If the typical inflow of infection was like a steady drip from a tap, this new strain seemed shot out of a fire hose.

Its most obvious characteristics were familiar at a glance. The worm was targeting – Phil could see this on his log – Port 445 of the Microsoft Windows Operating System, the most commonly used operating software in the world, causing a buffer at that port to overflow, then corrupting its execution in order to burrow into the host computer's memory.

Whatever this strain was, it was the most contagious he had ever seen. It turned each new machine it infected into a propagation demon, rapidly scanning for new targets, reaching out voraciously. Soon he began to hear from others in the Tribe, who were seeing the same thing. They were watching it flood in from Germany, Japan, Colombia, Argentina and various points around the United States. It was a pandemic.

Months later, when the battle over this worm was fully joined, Phil would check with his friends at the University of California, San Diego (UCSD), who operate a supercomputer that owns a "darknet", or a "black hole", a continent-size portion of cyberspace. Theirs is a "slash eight", which amounts to one 256th of the entire internet. Any random scanning worm like this new one would land in UCSD's black hole once every 256 times it launched from a new source. When they went looking, they found that the first Conficker scan attempt had hit them three minutes before the worm first hit Phil's honeynet. The source for their infection would turn out to be the same – the IP address in Buenos Aires. The address itself didn't mean much. Most internet service providers reassigned an IP address each time a machine connects to the network. But behind that number on that day had been the original worm, possibly its author but more likely a drone computer under his control.

The honeynets at SRI and at UCSD were designed to snare malware in order to study it. But the worm wasn't just cascading into their networks. This was a worldwide digital blitzkrieg. Existing firewalls and antiviral software didn't recognise it, so they weren't slowing it down. The next questions were: Why? What was it up to? What was the worm's purpose?

The most likely initial guess was that it was building a botnet. Not all worms assemble botnets, but they are very good at doing so. This would explain the extraordinary propagation rate. The term "bot" is short for "robot". Various kinds of malware turn computers into slaves controlled by an illicit, outside operator.

Imagine your computer as a big spaceship, like the Starship Enterprise on Star Trek. The ship is so complex and sophisticated that even an experienced commander like Captain James T Kirk has only a general sense of how every facet of it works. The ship contains many complex, interrelated systems, each with its own operator, performing routine maintenance, exchanging information, making fine adjustments, keeping it running or ready. When idling or cruising, the ship essentially runs itself without a word from Captain Kirk. It obeys when he issues a command and then returns to its latent mode.

Now imagine a clever invader, an enemy infiltrator. He knows the ship well enough to find a portal with a broken lock overlooked by the ship's otherwise vigilant defences – like, say, a flaw in Microsoft's operating platform. So no one notices when he slips in. He trips no alarm and then, to prevent another clever invader from exploiting the same weakness, he repairs the broken lock and seals the portal shut behind him. He improves the ship's defences. Ensconced securely inside, he silently sets himself up as the ship's alternate commander. The Enterprise is now a "bot". The Enterprise continues to perform as it always did. Meanwhile, the invader begins surreptitiously communicating with his own distant commander, letting him know that he is in position and ready, waiting for instructions.

And now imagine a vast fleet, in which the Enterprise is only one ship among millions, all of them infiltrated in exactly the same way, each ship with its hidden pilot, ever alert to an outside command. In the real world, this infiltrated fleet is called a "botnet", a network of infected, "robot" computers. The first job of a botnet-assembling worm is to infect and link together as many computers as possible. Thousands of botnets exist, most of them relatively small – a few tens of thousand or a few hundreds of thousands of infected computers. More than 1bn computers are in use around the world and, by some estimates, a fourth of them have been joined to a botnet.

Most of us still think of the threat posed by malware in terms of what it might do to our personal computer. When the subject comes up, the questions are: how do I know if I'm infected? How do I get rid of the infection? But modern malware is aimed less at exploiting individual computers than exploiting the internet. A botnet-creating worm doesn't want to harm your computer; it wants to use it.

Botnets are exceedingly valuable tools for criminal enterprise. Among other things, they can be used to efficiently distribute malware, to steal private information from otherwise secure websites or computers, to assist in fraudulent schemes, or to launch Dedicated Denial of Service (DDoS) attacks – overwhelming a targeted server with a flood of requests for response. If you control even a minor botnet, one with, say, 20,000 computers, you own enough computing power to shut down most business networks. The creator of an effective botnet, one with a wide range and the staying power to defeat security measures, can use it himself for one of the above scams, or he can sell or lease it to people who will. Botnets are traded in underground markets online. Customers shop for specific things, say, 50 computers that belong to the FBI, or 1,000 computers owned by Google or Bank of America or the military. The cumulative power of a botnet has been used to extort protection money from large business networks, which will sometimes pay to avoid a crippling DDoS attack. Botnets can also be used to launder money. Opportunity for larceny and sabotage is limited only by the imagination and skill of the botmaster.

If the right orders were given, and all bots in a large net worked together in one concerted effort, they could crack most codes, break into and plunder just about any protected database in the world and potentially hobble or even destroy almost any computer network, including those that make up a country's vital infrastructure: systems that control banking, telephones, energy flow, air traffic, health-care information – even the internet itself.

Today the most serious computer predators are funded by rich criminal syndicates and even nations. Cyberattacks were launched at digital networks in Estonia by ethnic Russian protesters in 2007 and in Georgia before Russia attacked that country in 2008; and someone, probably Israel or the United States (or both), successfully unleashed a worm called Stuxnet in 2010 to sabotage computer-controlled uranium centrifuges inside Iran's secretive nuclear programme.

The threat may be virtual, but the consequences would be all too real. A successful computer attack could compromise nuclear reactors, electrical grids, transportation networks, pipelines – you name it. Earlier this year, the Pentagon formulated its first-ever formal cyberstrategy, which found that a cyberattack on America originating in another country would be considered as much an act of war as dropping bombs on Buffalo, one that would justify a traditional military response. It is, of course, always easier to tear something down than to build it up, easier to break into a computer than to protect it, so the good guys work at a constant disadvantage. The tide of malware is relentless.

Phil had no way to stop the spread of this new worm. He could only study it. And he could tell little about it at first. He knew roughly where his first sample had come from, and that it was something unrecognised. He knew it was a genius of a propagator. The first step in dealing with any new malware is to "unpack" it, to break it open and look inside. Most malware comes in a protective shell of code, complex enough to keep amateurs from taking a close look, but Phil's Menlo Park wizards were pros. They had invented an unpacking programme called Eureka that cracked open 95% of what they saw.

When they tried it on the new worm, it failed.

Sometimes when Phil was stymied like this, he'd just wait for one of the anti-virus companies to meet the challenge. But the Conficker worm, as it came to be known, was flooding in so fast that waiting was not an option. By mid-December, three weeks after Conficker first appeared, the worm had burrowed into well over 1m computers worldwide. It had spread silently for six days before it began regularly trying to connect with its botmaster, who could have been hiding behind any of the 250 domain names the worm generated afresh each day. Such a large infection became a noisy presence on the internet.

Yet still it had attracted no attention outside the Tribe.

The problem was already known to be bigger than that in the security community, but since there is no such thing as an agency charged with protecting the internet for its own sake, concern about Conficker proceeded from a variety of narrower motives. The anti-virus industry was worried about protecting its customers, but was also mindful that the growing lists of bots represented a potential gold mine of new customers – since the malware disabled security updates, each bot was a prime mark for remedial software (the botnet itself was a valuable list of unprotected computers). The telecoms folks were interested in protecting their vital networks from DDoS attacks. Microsoft wanted to safeguard its customers and reputation, while researchers like Phil Porras at SRI had more of an academic interest, figuring out what this latest wrinkle meant.

The work demanded aptitude, but also years of experience. Most of the elite übergeeks today were the first generation to grow up with computers and have absorbed an intuitive fluency with networks. They work for software companies, research labs, security firms, telecom companies, government, or internet service organisations. Whatever the overarching agendas of their employers, these guys (and they're mostly men) were viscerally drawn to fighting Conficker. This was intellectual combat, pitting the best good-guy minds against the best bad-guy minds.

It is one of the peculiarities of modern times that as industrialised nations depend more and more on computer networks for everything, relatively little thought has been given to protecting them. The US spends billions on its military, not just to protect its own borders, but to project force anywhere in the world on short notice. Yet the telecommunications networks that increasingly undergird every aspect of modern life, not to mention the military itself, are shockingly vulnerable to infiltration and sabotage, not just from pranksters and cybercriminals, but from the very nations the United States are likely to confront as enemies.

The ad-hoc group that formed to combat Conficker reached out repeatedly to government agencies, including law enforcement, the military, the intelligence community, and every other agency you might expect to have an interest in protecting the computer networks of the nation (not to mention the world). They eventually succeeded in getting reps from the alphabet soup – NSA, DOD, CIA, FBI, DHS, etc – to sign on as members of the private chat channel where they co-ordinated strategy, but throughout the effort the feds would remain lurkers; they logged in and listened, but rarely made a peep. Over four months in December 2008 and January, February and March 2009, as Conficker assembled the largest botnet in the world, government, which would seem to have had the largest share of overarching responsibility, played a shockingly minor role. At first the übergeeks assumed the feds were constrained by the need for secrecy: you know, protecting official tactics and methods. Surely behind the scenes there was a sophisticated, well-funded clandestine official apparatus – everyone has seen the gleaming, dark glass and metal, see-everything/hear-everything sets Hollywood dusts off for its espionage blockbusters. What the anti-Conficker group discovered was deeply disillusioning. The real reason for the feds' silence was… they had nothing to offer. They were in way over their heads.

So the battle was in the hands of this odd and uniquely talented collection of volunteers. Given the esoteric nature of the combat, it lent itself less to the analogies of earthbound warfare than to the fantastic. It called to mind DC Comics's Justice League of America or, better still, X-Men. What were superheroes, after all, but those with special powers? Marvel's creations were also invariably outsiders, not just special but mutant, a little bit off, defiantly antisocial, prone to sarcasm and cracking wise, suspicious of authority, both governmental and corporate. There is not one of the übergeeks involved who had not, at one time or another in his life, realised that he could run rings around the safeguards and defences of most computer systems.

As the threat mounted, working with the X-Men became a mark of status. Here was a band of warriors for the internet, which is to say warriors for civilisation. There were only a few hundred people in the world capable of the work.

By the end of December, the X-Men were regularly pulling all-nighters, trying to stay one step ahead of the evil botmaster.

TJ Campana, senior manager for investigations for Microsoft's Digital Crimes Unit, was working until 10pm most nights in his office up in one of Microsoft's Redmond sprockets. His boss would stop by, surprised to see him in so late.

"What are you doing?" he'd ask.

"Conficker."

"Everything OK?"

"Well, the internet's melting. We're just keeping it from melting completely."

The bad guys behind Conficker, its unknown botmaster, would prove to be worthy adversaries. They were villains in the truest sense, talented programmers bent on using their powers for evil. And the world war was about nothing less than the soul of the future, the soul of the new global mind. As for the X-Men, what could be cooler than to be right in the middle of it, showing off your chops?

This is an edited extract from Worm: the story of the First Digital World War by Mark Bowden, published at £16.99 by Grove Press UK. To order a copy for £13.59 with free UK p&p click on the link or call 0330 333 6846

Mark Bowden will be speaking at TEDxObserver on 10 March. Tickets for the event in London are sold out, although there are some tickets remaining for simulcasts in six venues around the UK. TEDxObserver will also be streamed live on theguardian.com. For full details of the event, go to theguardian.com/tedx