Fresh revelations in the Guardian have delayed the publication of an official report into the work of undercover policemen in the UK. The report was ordered after Metropolitan police officer Mark Kennedy earlier this year exposed the widespread police penetration of legitimate protest groups. His evidence revealed miscarriages of justice when evidence was suppressed. But it now gets worse, as it appears that false evidence was used in trials and police officers have perjured themselves to maintain their cover.

The latest piece revealed that in 1997 Jim Boyling, posing as Jim Sutton, a committed member of the Reclaim the Streets campaign, was arrested and prosecuted for his part in the protests. He allegedly maintained his cover even when questioned in a court of law under oath. Other former secret policemen have approvingly said that this was a normal procedure to build up the credibility of an undercover agent.

Boyling's double life began in 1995, and the timing is interesting. Until 1994, MI5's F2 section had played the leading role in investigating UK political "subversives", those citizens deemed to be a threat to national security. The original justification for this work came after the exposure of the notorious Cambridge spy ring in the 50s and 60s, when MI5 was tasked to identify Soviet moles within the British establishment. However, this justification was pretty threadbare by the late 80s when the Communist party in Britain became largely defunct. At that point MI5 decided that Trotskyists could also a big threat, despite the fact that there was zero likelihood of any Trotskyist group being backed by the USSR. But why let the facts get in the way of a good spy investigation?

After the Berlin Wall had been down for a few years, MI5 finally threw in the subversive towel and closed down F2. Any residual responsibility for monitoring such groups was passed on to the police special branches. And this is where the current scandal begins.

The police also faced a massive reduction in their role, as a large part of the work of special branch sections across the country had involved reporting on local "subversive" groups. The Metropolitan police SB also had a dedicated unit, the special duties section (SDS), which ran police officers into political groups. Top brass were in a quandary: what to do with all this specialist knowledge and expertise? Well, it appears that work was made for idle SDS hands, and the unit began to infiltrate single-issue protest groups.

It appears to have been around this time that the control of the SDS, which morphed into Mark Kennedy's now-notorious national public order investigation unit (NPOIU), was transferred from the notionally accountable MPSB to the wholly unaccountable private limited company that is the Association of Chief Police Officers (Acpo).

So, let's get this straight. We had a bunch of secret policemen running around the country spying on their fellow citizens with no legal justification and absolutely no political oversight or public accountability. This would be difficult to justify even if our national security were under direct threat, but how precisely was the Reclaim the Streets campaign an existential threat to our nation?

If the investigation ever resumes, it needs to ask some hard-hitting questions: when did Acpo take over command of the undercover cops, what was the legal justification for this work and will there be a review of all cases involving such agents provocateurs?

Crucially, how on earth is it justifiable in a democracy for secret police to work undercover, gather intelligence on political campaigners exercising their democratic rights, act as agents provocateurs, and perjure themselves in court? The last time I checked, perjury carried a seven-year prison sentence, but I'm willing to bet Jim Boyling will not be back in the dock.

The UK likes to think of itself as a functioning democracy, but any country that condones spies infiltrating legitimate political protest groups is closer to a Stasi-style police state.