Several generations have now lived under the shadow of the Misuse of Drugs Act 1971, including police officers like me who became increasingly disillusioned with its effects. Despite all the money and effort poured into the so-called "war on drugs", the inexorable spread of drugs and the accompanying damage is powerful testament to failure. What we are doing is not only very expensive and misdirected activity, but actively counterproductive and harmful.

As a young constable in London, I was shocked when I saw the "pit", a hospital room used for the temporary storage of the latest collapsed "junkie" picked up from the pavements of the West End. After minimal treatment they awoke and staggered off, back to their next hit, hoping it was not going to be their last. Some ended up in the mortuary.

If your child was found in possession of drugs, would you want them to be arrested, charged and convicted (with all the stigma that entails) or advised, supported and treated if necessary? Every drug user is someone's child and, sadly, often the victim of emotional, physical and sexual abuse.

Drug-taking blocks the pain and yet we ostracise and criminalise rather than understand and support. "Drugs are bad, ban them!" is an easy mantra, but it ignores the history of alcohol prohibition in the US and our own recent experience of spending more than £10bn a year on the criminal justice system and losing more than £15bn to crime that has merely accompanied the rise in the drug trade. The criminals make around £6bn a year. They are the success story.

I suppose that I arrested as many "druggies" as anybody on the team and the thumbnail of cannabis found in the bottom of their pockets found its way on to the charge sheet as a matter of routine. Sometimes detectives came back to the police station with a few pot plants they had found on some hippie's window ledge. After a few weeks of healthy, well-watered growth on the crime squad's own windowsill, this now very impressive evidence arrived on the evidence bench of the magistrates' court.

I commanded or oversaw many anti-drug operations. In one London council estate we arrested almost 30 street dealers in a co-ordinated swoop, motivated by a desire to tackle overt street-dealing in heroin and crack cocaine.

Some undercover officers put themselves at risk as they immersed themselves in the addicts' lifestyle (showers not an option) and became accepted by the dealers. Others were at risk of falling off ladders as they assumed the role of observant decorators. The evidence was so good that all those arrested pleaded guilty. And one building ended up with five coats of paint.

A bigger operation in an East Anglian city targeted more than 100 street dealers. It was hailed as a great success by politicians, much as any large seizure of drugs, or police "crackdown", is celebrated as evidence of the success of the "war on drugs". Within days the dealers were back. If success were measured by the volume of arrests and drugs seized, you could conclude that the police had done well; however, judged on success in containing the market and reducing harm, the outcome is quite different.

It all seemed so pointless; what were we achieving? The enthusiastically spun revolving door of criminal justice took in and spat out users and dealers, often addicts themselves, to deal again. Men and women, arrested for little more than youthful experimentation, emerged with lives forever tainted by a conviction.

Nowhere in the country is free from drugs and the associated crime epidemic. Criminals continue to make huge profits, corroding and corrupting public and private lives. They target each new generation of children and create addicts who are ostracised, become diseased and die unnecessarily.

More recently, I have been working abroad and the problems that exist worldwide are recognised at the highest levels, with most acknowledging the harmful unintended consequences of the current approach. A huge criminal market (with enormous financial incentives) has been created using corruption and violence to make its huge profits.

Efforts to destroy crops only destroy peasant farmers' livelihoods and the environment, while the poppy fields and coca plants spring up elsewhere, with producers adapting to meet the demand. Growing other crops is futile if the demand for drugs remains.

Our limited resources are directed towards this futile "war" while public health, which is clearly the first principle of drug control, remains an impoverished baby brother. Prevention and treatment, surely, should come first.

Finally, users are excluded and marginalised from the social mainstream, tainted with a moral stigma, and often unable to find treatment even when they may be motivated to want it. The biggest growth of HIV/Aids outside Africa is in injecting drug users.

Unless we face these unintended consequences head-on, we will continue to be mesmerised by the many paradoxes of the drug problem. We can do things differently. In Boston in the 1990s the US police successfully concentrated on reducing the number of murders as a greater priority than pursuing futile efforts to reduce the scale of the illegal market. In Portugal, decriminalisation of possession of all drugs since 2001 has unblocked a hopelessly overcrowded court and prison system, and evaluations of this approach have shown a broadly positive impact on recidivism and social reintegration and a significant cost saving to the government.

The Swiss people voted by a two-thirds majority last year to ratify their successful heroin prescription programme as official government policy. For 15 years, heroin has been prescribed in special clinics under controlled conditions, resulting in less crime, death and disease and fewer new users. After this "medicalisation" heroin is no longer cool. Importantly, of the previously hopeless individuals many now hold down a job and live normal family lives. All we have managed is three trial runs, obviously successful, involving just over 100 heroin users. This is good news, but we must move more quickly.

As we wring our hands and close our eyes to the lessons from abroad, delay in expanding heroin prescribing will inevitably lead to more people who will die, contract HIV and Hepatitis C, continue to commit crime and prostitute themselves to feed their habits.

The different approach in East Anglia offered prolifically offending addicts a choice between treatment and arrest. They almost invariably chose treatment, and detectives were surprised to learn that not only did this save time and precious resources, but it was also the most effective way of tackling burglary they had ever seen. We thought and acted in new ways and achieved better results, for everybody.

Prosecuting users is misguided and counterproductive; prosecuting dealers without tackling demand or their profits does not work. If the money wasted on misinformation, low-level enforcement and condemnation had been spent on tackling the underlying causes, so many blighted lives could have been different. There are other options, but sadly we cannot hold a rational public debate as serving officers or politicians who dare challenge the "war on drugs" orthodoxy justifiably fear being pilloried by our national press.

Politicians will not even conduct a cost-effectiveness analysis of the current approach. The drug policy thinktank Transform has calculated savings of up to £14bn a year if drugs were controlled and regulated. It's not as if we could not do with the money.

So, where are we? Law enforcement spending is up, criminal profits are up but drug use is also up. The game's up!

We know that we must change and we also know that police officers like to make things happen. This is the time for police leaders throughout the world to challenge the status quo and focus resources on serious, organised criminals, not blighted users, and to focus on harm reduction not some pie-in-the-sky dream of a drug-free society. Where they lead, politicians will follow.