If hacking someone's voicemail is a gross invasion of privacy, what words are left to describe agents of the state with fake identities having sex with women they're spying on? One activist who had a child with the undercover police officer Bob Lambert has offered four words: "raped by the state". She is among a group of women activists currently fighting attempts by the Met to sabotage their quest for truth and justice. If phone hacking provoked anger, the use of police spies should chill.

But police spies stealing the identities of dead children and duplicitously sharing the homes, beds and lives of women is only the latest in a string of damning scandals about the Metropolitan police: Stephen Lawrence, and the Macpherson report's subsequent conclusion that the Met is institutionally racist; a stop-and-search policy that discriminates against black people; deaths in police custody; the shooting of Jean Charles de Menezes; the unlawful killing of Ian Tomlinson; the treatment of protesters as social problems to be contained; the stitching up of a Tory heavyweight.

Each scandal is examined in isolation, treated as the action of rogue officers. But together they suggest an institutionally rotten system. Londoners need a force devoted to protecting their security, which treats all sections of the community equally, and which enjoys the consent and trust of everyone. Currently they do not have one, and so it must be built on new foundations.

This is a suggestion that will infuriate some, not least Met officers. Easy for a columnist, issuing grand proclamations behind the safety of his desk. Met officers, on the other hand, are taking rapists and killers off the streets, putting their lives in danger as they do so. More than 3,000 British police officers are injured a year; about 800 seriously. But this is not about individuals: it's the system that is the problem, and it traps good and bad officers alike.

The government has finally announced an inquiry into police spies, driven on by the revelation that a police force supposed to be solving the murder of Stephen Lawrence was actually spying on his grieving family. But Doreen Lawrence is right to state that police failings go to "the highest level", and the Macpherson report's damning conclusion – that the Met is "institutionally racist" – is as true as ever.

Doreen Lawrence, the mother of Stephen Lawrence, 'is right to state that police failings go to the highest level'. Photograph: Andy Rain/EPA

I've never been randomly stopped and searched by a police officer, but I've met plenty of young black men who have. The experience varies: sometimes officers are almost apologetic, other times full of intimidation and aggression. The evidence shows that black people are significantly less likely to use drugs, and yet black Londoners are six times more likely to be stopped on suspicion of possession. It is difficult to conclude that this is anything but racism.

It is not just black Londoners who have described the Met as "the biggest gang around here": senior officers have self-described as such. "You might have 100 people in your gang," publicly declared Chief Inspector Ian Kibblewhite, of Enfield police, in 2012. "We have 32,000 people in our gang. It's called the Metropolitan police." But a "gang" does not serve a community: it has a turf, a demand for prestige and status, a desire to smash enemies.

When Andrew Mitchell was stitched up by Met officers, the lesson was frightening and instructive. The number of officers involved – including PC Keith Wallis, jailed for falsely claiming to have witnessed the infamous bicycle incident – must give pause to those who think it is a story of "bad apples". If an upper-middle-class Conservative cabinet minister can be stitched up, what hope for the rest of us? It is a point he has passionately and rightly made himself.

A story of conspiracy and cover-up is all too familiar, although other victims do not enjoy anything approaching the power and influence of a Conservative chief whip. There have been 82 black and minority ethnic deaths following contact with the Metropolitan police since 1990, and not a single successful prosecution. Among them is Sean Rigg, a black musician who died in Brixton police station in 2008; four years later, an inquest jury found that police had used unnecessary force against him. It was in stark contrast to initial police claims, and – after a prolonged fight by Rigg's family – three officers were arrested on suspicion of perjury.

When the newspaper vendor Ian Tomlinson died after being thrown to the ground in 2009 at the G20 protests by PC Simon Harwood, the initial police narrative – faithfully repeated by so many news outlets – blamed protesters, claiming that officers coming to his help were bombarded with "bricks, bottles and planks of wood". It was all lies, and symptomatic of a force that saw protest as something that had to be contained, not facilitated. Young people had been patronised as the apathetic "X Factor generation": when they mobilised on the streets, they were met with batons and kettles.

What would a new police force look like? That should be left to a royal commission – headed by an independent figure, not an establishment patsy – which calls evidence from all sections of the community. Structures, training, forms of accountability: all need to be designed from scratch. It needs to be a body stripped of prejudice and bigotry, that defends hard-won democratic freedoms, as well as protecting people's security. It is all over for the Met, and time to debate the police force that London deserves.