There's nothing new about British governments spying on their own citizens. From the time of Elizabeth I's spy chief Francis Walsingham to the legendary agent provocateurs of the years after Waterloo to the bugging and blacklisting of the postwar decades, espionage against domestic dissenters has long been a staple of British statecraft. For most of the last century, the secret state targeted the left, trade unionists and peace campaigners, along with Irish republicans and anyone else regarded as a "subversive" threat.

That was all supposed to have been consigned to history after the end of the cold war, when MI5 declared it had abandoned counter-subversion and switched its focus to the threat of jihadist terror attacks. But, if anything, the apparatus of official snooping and spooking has grown even more inflated than in the days when the state faced a real political challenge from both within and without.

It's now not just the security service and police special branch that spy on environmental campaigners and anti-war protesters, but an array of police intelligence units set up to keep tabs on those designated "domestic extremists", including through covert informants and intercepts. And as the Guardian's reports of the past few days have shown, these outfits don't just monitor activists, they work hand in glove with private companies, using anti-harassment legislation and pre-charge bail conditions, to prevent them from continuing to demonstrate and protest.

What began with injunctions against violent animal rights activists has now reached the point where hundreds of non-violent protesters are banned from going near arms factories or power stations, travelling to particular areas or even communicating with each other – without being charged with any offence. Last year, protesters at an academy school in south London were banned by injunction from handing out leaflets or even speaking outside the premises.

The Association of Chief Police Officers, which runs the intelligence units, claims that they only target groups that break the law – for instance, by peacefully occupying a power plant or taking secondary industrial action – or operate "outside of the normal democratic process". In fact, Acpo is itself an unaccountable private body, while protests and demonstrations are of course an essential part of the democratic process.

"Domestic extremism" is the subversion of the new surveillance state, though without even the spurious definition the cold war term was given. And just as MI5 used to claim it never targeted peace organisations or trade unions but the subversives within them, so the police intelligence apparatus insists it's only interested in "extremists", not the groups they're part of.

The home secretary Alan Johnson this week sneered that if the police wanted to use the term "domestic extremism" he "certainly wouldn't fall to the floor clutching my box of Kleenex". But by blurring the lines between the civil and criminal law and publicly branding those who take part in demonstrations and direct action, the police and the Home Office are in effect criminalising political dissent.

That is even more true of Britain's Muslim community, where the line the authorities are busy blurring is between political protest and terrorism. Dozens of British Muslims are due to appear in court today charged with public order offences over the angry demonstrations against Israel's war on Gaza in January. Several were arrested months after the event in dawn raids by police who broke down the doors of their family homes. In February, nine British Muslims taking part in George Galloway's Viva Palestina aid convoy to Gaza were arrested on the motorway under the Terrorism Act. They were eventually released without charge. But the impact on support from the rest of the community was naturally chilling.

Last week, reports in the Guardian and by the Institute of Race Relations highlighted how the government's £140m Prevent programme, which is supposed to mobilise Muslim community opposition to terrorism, is being used for what Liberty's Shami Chakrabarti calls the "biggest spying programme in Britain in modern times". Schools, community groups and colleges are required to provide information on everything from the opinions to the sex lives of Muslims not even suspected of involvement in violence.

Underlying the abuse of the programme has been a dangerous shift in official counter-terror policy which, in parallel with the wider police surveillance of protest group, now targets "non-violent extremism", rather than simply those who might want to launch bomb attacks on buses and tubes. The idea is that, as Ed Husain of the government-funded Quilliam Foundation puts it, non-violent Islamists – rather than western wars in the Muslim world – provide the "mood music" for terror groups and spying on them is "good and it is right".

In reality, both the mass surveillance and the government's decision to widen its target from the violent to the elastic McCarthyite catch-all of "extreme" is spreading fear and mistrust, intimidating Muslims from taking part in mainstream politics and undermining the very people who can most effectively challenge those drawn towards indiscriminate violence.

Intelligence is anyway notoriously unreliable, because it cannot be properly tested as evidence – whether on the grand scale of weapons of mass destruction in Iraq or in more routine injustices, such as the 2006 raid in London's Forest Gate, in which police shot an innocent man on the basis of groundless intelligence about a chemical bomb.

That's one of the unwitting messages of the new official history of MI5 by the loyal historian Christopher Andrew. While clearing a faction of the security service of having plotted against Harold Wilson, despite overwhelming evidence to the contrary, Andrew gives credence to absurd claims that the pre-eminent 1970s trade union leader Jack Jones was a paid KGB agent – this on the account of the same defector who once claimed to general ridicule that the former Labour leader Michael Foot had been a Soviet agent codenamed Boot.

Which is a timely reminder of the self-serving tendency to fantasy among intelligence organisations. Unleashing such people on those exercising their right to protest or take part in non-violent politics has got nothing to do with the defence of the democratic process – it's an assault on democracy.