There are two big mysteries at the heart of the Edward Snowden story. First, why did he do it? That is, why did he do it: here was a relatively nondescript, unassuming twentysomething, with no apparent political backing, popping up out of nowhere to take on the world's most powerful security organisation. By incurring the wrath of the US government Snowden knew he was risking a lifetime in jail. Even the journalists who worked closely with him were confounded by his bravado, or naivety, or perhaps both.

Second, how did he do it? Snowden wanted the world to know about the newfound and mindboggling capacity of the NSA and its international partners to hoover up private information, allowing them to snoop on almost anything or anyone. Snowden nicknamed this surveillance operation the "Panopticon", after Jeremy Bentham's all-knowing, all-seeing prison system. Yet that same organisation failed to notice when Snowden, a mid-level contractor, made off with its own darkest secrets, seemingly blind to the most glaring security threats in its midst. What was going on?

Luke Harding's breathless page-turner, which reads more like a spy thriller than a piece of dry political analysis, does its best to answer these questions. Harding gives us Snowden's backstory, which is not as straightforward as it might appear. Yes, he was a pretty regular guy, good with computers, interested in girls, a bit flaky perhaps and something of a drifter, but no more than the average. Trawling the extensive record of Snowden's online activities, where he posted for more than a decade on every subject under the sun as "TheTrueHOOHAA", Harding suggests he might have been a bit of a Walter Mitty. But isn't that the point of the internet age, that it makes Walter Mittys of all of us? What's striking is not so much the range of Snowden's fantasies as the depth of his political commitment. He emerges as a committed Republican, a libertarian, a huge fan of Ron Paul, a gun lover and believer in national security with a tendency to suggest that anyone who thinks otherwise deserves to be shot.

This produces some startling moments. Writing in 2009 after the New York Times has leaked secret information about covert US action against Iran's nuclear programme, "TheTrueHOOHAA" rages against whistleblowers, WikiLeaks and anyone who would betray their country for the sake of airy-fairy liberal principles. "Those people should be shot in the balls," he writes. But the real clue to his motivation comes later in the same exchange, when he wails: "Obama just appointed a fucking POLITICIAN to run the CIA." (The politician in question was Leon Panetta, who had once been Bill Clinton's chief of staff.) As a libertarian, what really gets Snowden's goat is the thought of government getting its tentacles into everything. He has no problem with spying and secrecy in their place (in Iran, for instance). What terrifies him is the idea that no one is setting limits to it all. Like many supporters of Ron Paul, Snowden would like to go back to the gold standard, because he thinks letting politicians print money is a recipe for inflation and ultimate global ruin. He sees the politicisation of surveillance as part of the same pattern: evidence of a system spinning crazily out of control.

Despite dropping out of college and a failed interlude in the army (he broke both legs in a training accident), Snowden's tech skills eventually got him good defence jobs, first at the CIA, then at the NSA, and finally at a private firm, Booz Allen Hamilton, which serviced the NSA's computer systems. At some point he went from loyalist to whistleblower – Obama's election seems to have tipped him over the edge. While Democrats were complaining about government overreach during the Bush years, Snowden was able to hope that regime change in the White House would signal a return to proper oversight. But when Obama morphed from a critic of the security state in opposition to its number one enabler in government, Snowden concluded that any safeguards were gone. In his eyes the entire US government was now operating outside its constitutional remit. What alarmed him about the NSA's activities was that no one was in charge: this had become a system that was, as he put it to journalist Glenn Greenwald, "automatically ingesting" vast amounts of human communications, indiscriminately, blindly, idiotically. It was a monster, and it was taking over the world.

In a way, Snowden's own experiences confirmed that he was right. The monster was so big and so unwieldy that it didn't notice what was going on. It was as if it had no time for old-fashioned security checks in the brave new world of big data. One of the most astonishing revelations in Harding's book is that Snowden had already blotted his copybook at the CIA, where a row with a superior had him marked down as unreliable. But when he transferred to the NSA, no one thought to pass on the personnel file, so they employed him without checking his backstory (perhaps they simply looked at "TheTrueHOOHAA" posts and decided, mistakenly, that he was one of them). At Booz, Snowden was able to scrape vast amounts of data off the NSA computers, using what now appears to have been relatively old-fashioned technology, without anyone detecting what he was up to.

Among his trawl was a series of internal PowerPoint presentations in which the NSA outlined its new capabilities and its eager readiness to use them. There are two ways to read these. One is that they are evidence of an organisation that now has terrifying technological reach, able to cross all borders and access any information it chooses, often co-opting the tech industry along the way (Google, Facebook and other titans of Silicon Valley found themselves implicated in what was going on, to their horror). The other is that, like many PowerPoint presentations, they contain a fair amount of boastful corporate bullshit. "The Mission Never Sleeps" is the ironic heading of one of the files that Snowden stole. Snowden is probably right that what's really scary is the thought of so much power being in the hands of people with so little idea of what it means. When it turned out that the NSA had been bugging Angela Merkel's phone, with disastrous political consequences, no one could say what the point had been. As John McCain told Der Spiegel, the only plausible explanation is that "they did it because they could".

Once Snowden broke cover in Hong Kong the surveillance state lumbered into action. The authorities hadn't been much good at detecting what he was up to, but now they were determined to limit the damage. They weren't much good at that either. The second half of Harding's book describes, in sometimes hilarious detail, the cack-handed attempts of various security services to put on the frighteners. In America, journalists and editors were alternately brow-beaten and threatened by various high-up officials. In Britain, in the most bizarre episode of all, two heavies from GCHQ supervised the destruction of the Guardian's hard-drives that were thought to contain the illicit files. Greenwald's partner was detained and searched under anti-terrorism laws by British police officers at Heathrow, who were on the hunt for more Snowden material. The Americans persuaded the French to bar the plane of Bolivian president Evo Morales from their airspace, on the suspicion that he had smuggled Snowden himself aboard (Snowden was by now holed up in a Moscow airport).

None of it worked. The material, once it had escaped into the public domain, could hardly be put back in the bottle: no amount of smashed up machinery can stop the spread of information (the NSA people, who devote much of their time to breaking encryption codes, might be expected to know that). Snowden was eventually granted temporary asylum in Russia, the very last thing the Americans wanted. Still, the spooks must be allowed to play their silly games. Harding writes that in the immediate aftermath of Snowden's revelations, construction crews appeared during the night outside the offices of the Guardian and the homes of its reporters, "taxi drivers" got mysteriously lost, "window cleaners" began loitering outside meeting rooms. Those trying to report the story found their lives inconvenienced – and occasionally they got a little scared – but it hardly put them off what they were doing. You could call it the Chris Christie rule of politics: it's easier to start traffic jams than it is to prevent them, but that doesn't help anyone.

Harding slightly overdoes the plucky journalists versus the overweening state: his is an insider's account that suffers from the vice of all such accounts in bigging up the experience of the people who were actually in the room. You had to be there. What they were doing was extremely important, but some of the excitement of being at the heart of world events reads overegged on the page. Also, for a writer telling a story whose details depend on understanding how tech works, he sometimes seems hazy on the basics: Tim Berners-Lee, the creator of the web, is described more than once as "the inventor of the internet", which is bit like calling Henry Ford the man who invented roads (the web is a system of documents accessed via the internet). Still, this is a riveting read and it unravels the mystery better than anything that's been published so far.

Yet by following the conventions of the political thriller – with its heroes and villains, its nods to John le Carré and All the President's Men – Harding perhaps does his tale a disservice. What is so astonishing about the secrets that Snowden revealed is how much in the dark everyone turns out to be. No one really understands what it all means. The pace of technological change and its extraordinary reach mean that a lot of this stuff is entirely new: this isn't Nixon's world any longer but it's not Deep Throat's either. Snowden is a quirky figure – a distinctive product of the American right, in ways that some of his European champions on the left ought to find uncomfortable – but he is also a thoughtful one. He is correct in thinking that something has fundamentally changed in our relationship to power. He would like to turn the clock back to the late 18th century when the American constitution said what it meant and meant what it said – the "originalist" dream. That's not going to happen. It's not even clear that we can turn the clock back to the late 20th century. This is a new world and a scary one.

In Britain the political debate about how we are going to regulate this new world has barely got going. In the US (and even more in so Germany) Snowden's revelations have kickstarted an angry discussion about how to tame the monster. Though it's unlikely to be enough to satisfy Snowden, some members of the US Congress have begun to bare their teeth. But here in the UK we still have politicians such as Sir Malcolm Rifkind, a standard-order product of the late 20th-century British establishment, assuring us that his parliamentary intelligence committee has got GCHQ under wraps. It doesn't. Rifkind has instinctively closed ranks with people whose capacity to abuse their powers he can barely comprehend. This week Ed Miliband included the need to take on the security services as part of his agenda to confront "unaccountable power". It will be a long haul.

• David Runciman's history of democracy in crisis, The Confidence Trap, is out from Princeton.