For some time now, England and Wales have had a semi-decriminalisation programme for cannabis. And it has ended up criminalising more cannabis users than ever before.

But it doesn't criminalise all cannabis users: it primarily targets people who are young, black or Asian. It is a story of muddle-headed government initiatives, skewed police incentives, racism, drug wars and the old, old habit of treating white people more leniently than everyone else.

In 2004, when cannabis was made Class C, cannabis warnings were introduced. These were spoken warnings given by a policeman on the street if you were caught with a small amount of weed for personal use. Five years later the drug was returned to Class B, but the cannabis warnings remained. This effectively gave the police discretion in how they treated cannabis possession. The result of this discretion was the disproportionate targeting of black and Asian youths.

Cannabis warnings are now the first step on the 'escalator' system of options available to the police if you're caught. It goes like this:

First time: Cannabis warning. Informal, verbal, doesn't form part of your criminal record. This is the least harsh option available to a police officer if they catch you with drugs.

Second time: Penalty Notice for Disorder (PND). Issued to anyone who has already got a cannabis warning within the last 12 months. Results in an £80 fine which must be paid in 21 days. Like cannabis warnings, they don't contribute to your criminal record. Also like cannabis warnings, they are issued there and then on the street.

Third time: Caution. Police can use this instead of sending someone to court for prosecution. Anyone accepting a caution must admit their guilt and it does figure on their criminal record. Cautions are preceded by an arrest and being taken to the police station.

Fourth time: Charge. The police and the Crown Prosecution Service (CPS) can charge an individual for possession and send them to the courts.

This is the best-case escalator scenario, where you are being treated as leniently as possible. The police can, if they want, charge you on a first offence. But that's not how it's supposed to work.

Nevertheless, it sounds reasonable. Clearly, this was partial, informal decriminalisation. Criminal penalties continued, but someone caught with cannabis was not expected to face severe criminal sanction. So even when cannabis was reclassified as Class B in 2009, the police had the freedom to maintain its previous classification in all but name.

What this policy has really done, though, is suck more people into the criminal justice system.

When the cannabis warnings were introduced, Association of Chief Police Officers (Acpo) chairman Andy Hayman said there should be a "presumption against arrest" and urged police officers to "use their discretion".

It worked for a while. The number of cautions issued dropped 40% in the first year. But by 2005 they were rising again. By 2011, the number of people found guilty of cannabis possession in court exceeded the number for the year before the warning scheme was introduced.

The graph below shows how many more people came into contact with the criminal justice system for cannabis. Partial decriminalisation had merely added a host of new police powers to the armoury.

What caused it? How did a method brought in to make the drug system more liberal do the opposite?

The problem is that a cannabis warning is classified as a 'recorded crime outcome'. That means that it is classified as a crime that has been detected and resolved. On the books, the verbal cannabis warning given to a teenager is equal to a murderer being brought in after months of painstaking detective work.

When the coalition came to power one of the first things it did was get rid of the target culture in the police force. But that only gets you so far. Humans are humans and police are police. Officers who want to get on in their careers still want to get their numbers up.

Cannabis crimes are a low-hanging fruit. They're an easy way to make it look as if you are a productive and hard-working member of the force.

Niamh Eastwood, executive director of Release, says:

"Many police officers on the street will proactively go out looking for people who are in possession of drugs for their own personal use, usually cannabis, in order to demonstrate their effectiveness to their superior officers. "This activity is most concentrated in areas of deprivation, where young people, often those from black and Asian backgrounds, are continually stopped and searched in the hope that small amounts of drugs will be found and an outcome achieved. "In our view the police have been given ample opportunity to change their practices and have failed."

Police officers go into areas they know they're likely to catch a bunch of kids with some drugs on them and they search people. The first occasion may just be a cannabis warning, but on the next visit it'll be a penalty notice, a caution – or straight to court. This informal model of decriminalisation didn't really decriminalise anything, it just introduced a new soft method of bringing young black and Asian kids into the criminal justice system.

The stats speak for themselves: seven out of 1,000 white people are searched for drugs. That doubles to 14 per 1,000 for those who are mixed race, like some weird kind of ethnic arithmetic. It's 18 per 1,000 for Asians and 45 per 1,000 for black people.

Black people are then charged for possession of cannabis at five times the rate of white people in London. For cannabis warnings, the rate is three times.

It's not just the police. The CPS is also responsible. It is absolutely not in the public interest to stuff the courts and the prisons full of people with some weed in their pocket. Its own guidance says so. But in practice, that's what they do.

Cannabis cases, like drug stop-and-search, provide low-hanging legal fruit. They have a very high rate of successful prosecution. You only need forensic confirmation that it's cannabis and a police officer to say they found it on the kid. Job done. No-one takes a young black kid's word over a police officer. The jury will almost always convict. They don't have much choice.

By 2010, the CPS was bringing more prosecutions for drugs than in any year since the Misuse of Drugs Act in 1971, and this was mainly driven by cannabis possession. Every year, between ten per cent and 15% of all indictable offences brought before the courts are for drug possession.

The people who get prosecuted are young – often very young. One in ten of them are between the ages of 15 and 18. And they are disproportionately black. Black people are sentenced at a rate of 4.5 times that of white people for drug offences. They are subject to immediate custody at five times the rate of white people.

The effect on those found guilty is shameful and well-documented. They are often thrown in jail, at the cost of £40,000 a year to the taxpayer. Once in there, they are statistically much more likely to go back to jail. They meet more criminals, try more things and learn a few others. With a criminal record, they find it harder to get a job. Without a job, they are more likely to turn to crime. And to benefits, while we're at it. Putting people in jail when you don't have to is a very, very bad way to reduce a deficit.

But the negative effect of this system isn't just in these worst-case scenarios. It starts earlier, on that first contact, when a young black kid is stopped and searched by a policeman for drugs.

This aggressive approach to disadvantaged communities defines many people's first and lasting impression of the police. It leads them to believe – often rightly – that they are bullies and thugs, coming into their neighbourhoods, taking control of them for a small amount of weed in their pocket, and starting a process which will, a few steps down the road, ruin their life chances. Of course, the white middle class kids are smoking that weed too. But they're not in the neighbourhoods the police use to get their numbers up.

It creates a generation of kids who don't trust the police, who are less likely to report a crime, to get in contact as a witness, or even join the force. It creates a generation who will laugh in your face if you mention 'community policing'.

What is the solution? It is easy. Stop counting cannabis warnings as a 'recorded crime outcome'. Remove the incentive for police to criminalise youth. It could be done with a flick of the pen and the situation would be much improved.

Theresa May is currently committed to reducing the number of stop and searches. If she is serious about it, this would be a very good way to do it.