I can understand the attraction of being a spy. There's the mystery, the play-acting, the adrenaline-rush of covert operations, the delicious fear of being found out. You are privy to information others don't have, which gives you power over them. In some ways it is "like being God", in the words of one of the Metropolitan police spies featured in this book; especially, perhaps, if you are someone who feels your life is rather pathetic otherwise. Of course you can also become a spy for principled and ideological reasons: because you genuinely believe you can do good this way. Wartime, or even cold war spies are likely to come into this category. But that doesn't seem to have been the case in any of the examples chronicled here; of men (and one woman) who infiltrated radical movements in Britain in the 1990s and 2000s, with new names (often taken from dead children, in case anyone checked the registry of births), long hair and beards (they called themselves "The Hairies"), and lots of sleeping around, to preserve credibility. They simply liked the thrill. (And the sex, one imagines.)

This, however, is one of the problems. For most people this sort of work is anathema. Espionage involves deception and betrayal, usually of people you have pretended to befriend, and – in at least one of these cases – women you have fathered babies with. It's a sordid business, and doesn't attract the most virtuous of people. So how can you trust their information? How do you know they're not provoking crimes, just to get the kudos for revealing them? (There are examples of this here.) How can you trust the authorities, with all this secretive power in their hands, not to use it for their own purposes – for example, to spy on and try to discredit their own enemies? (This also happened here: agents were charged with digging up dirt on critics of the police, and even on the family of Stephen Lawrence, whose case was notoriously mishandled.) Lastly, how can you trust them to stay loyal; not to go rogue – to blab to the newspapers?

That's how Guardian journalists Paul Lewis and Rob Evans got hold of this story: from a couple of ex-undercover agents who decided to spill the beans, apparently because they came to feel guilty about what they had been doing, which I suppose is to their credit. Another ex‑spy, Mark Kennedy, sold his story to the Mail on Sunday. It is interesting to read how different the penitents' stories in this book are from Kennedy's. The former had come to the realisation that the men and women they had been spying on were reasonable folk – kindly, harmless idealists in the main – and had seen them brutally beaten up by the police. They had obviously bonded with them. But Kennedy painted an entirely different picture, of filthy, scavenging, wild-eyed terrorists from whom he was hiding, fearing for his and his children's lives. That made him the victim, of course. One of them must be lying, but that's the nature of the beast.

It always has been. None of this is new. That is not to say it has always gone on. For much of the 19th century it didn't, partly because the British authorities were fully aware of the dangers; and partly because any hint of "domestic espionage" at almost any level was regarded as unethical. Worse: it was seen as French. "Spylessness" was a fundamental feature of Britain's national identity. (Those who wish to use history to establish the nature of "Britishness" might ponder that.) Even if it worked it wasn't justified. "I would rather half a dozen people's throats should be cut in Ratcliffe Highway every three or four years," said one MP (a Lord) in 1811, "than be subject to domiciliary visits, spies, and all the rest of Fouché's contrivances." (Ratcliffe Highway was a notorious London crime spot, Fouché the chief of the Parisian secret police.) Political spies especially were among the most excoriated villains of popular culture. Governments couldn't risk their use for many years after that.

That was one of the reasons. A more philosophical one was that domestic espionage undermined popular confidence in the authorities, making the exercise of authority more difficult. If they don't trust us, why should we trust them? These restraints gradually dissolved during the 20th century, partly under the cover of the world wars, and in the fever of paranoia after the Russian revolution. That came to a head in the late 1960s and 1970s. The farcical plot against Harold Wilson, who was suspected of being a Soviet mole, happened at exactly the time as the Special Demonstration Squad (SDS) – motto "By Any Means Necessary" – was secretly set up. In retrospect, the year 1968 can be seen to mark the beginning of Britain's Great Reaction, which has gone on (and on) since then. In Britain – this is obviously not true of everywhere – it is the right wing that has generally been most willing to use underhand methods against its perceived enemies, and to defend these methods openly: "If you have done nothing wrong you have nothing to fear." It is also the right who tend to cry "conspiracy theorist" at anyone who suspects this kind of thing. In the view of Evans and Lewis, the conspiracy theorists have been "not nearly paranoid enough".

The question remains whether these methods can be in any way justified by their results. The circumstances of today are different from those of the 19th century. It is arguable that there are more subversive dangers now: though there were plenty of things that the Victorians could have painted as subversive if they had been so inclined. Perhaps the switch from being the most proudly "spyless" of nations to possibly the most spied on – surveillance cameras, GCHQ, Prism, industrial "blacklists" and the like – had to come. But it is difficult to see the SDS fitting into this rationale. IRA and Islamist terrorists are one thing. (Bankers might have been another, if only we had realised where the real danger lay.) But tree-huggers, veggies, anti-imperialists and animal rights campaigners – the SDS's main targets? Come on!

With all this going on, it may not be surprising that popular trust in government is breaking down. It is what the Victorians would have expected. We're no longer so surprised by revelations such as this, which is sad. But the real change will have come if we are no longer outraged by them – then we really will have changed as a nation. With luck, the grotesqueries revealed in this powerful book will fire some of the old free and radical British spirit again. But don't wait up.

• Bernard Porter is the author of Plots and Paranoia: A History of Political Espionage in Britain 1790-1988