Forty years ago, Richard Nixon declared "all-out war on drugs", and enlisted the support of global allies. Since then the war on drugs has cost the United States US$1 trillion, drugs are cheaper and more available, the profits for trading them greater than ever and the violence of the entire culture of prohibition escalating. By all of its own criteria, the "war" has failed utterly.

So why on earth are we still fighting it?

While Australia is still an enthusiastic front-line ally, many countries have beat a hasty retreat. Mexico and most South American nations have recently abandoned the policy. Earlier this year, US Senator Jim Webb introduced a bill seeking to reform the entire criminal justice system, saying the nation is "wasting billions of dollars and diminishing millions of lives" on ineffective strategies such as the war on drugs. The bill has the backing of a growing band of police dissenters. One such group, Law Enforcement Against Prohibition (LEAP), believes the policy is a disaster for law enforcement in the US.

One of LEAP’s key advisors is former Police Chief Norm Stamper. After 34 years fighting and losing the war on drugs — three of them as chief of police in Seattle — Stamper left the force. He now blogs on Huffington Post and travels extensively, making LEAP’s case for legalisation. He calls the war on drugs "the most devastating, dysfunctional, harmful policy since slavery". Stamper was in Sydney to speak at the Festival of Dangerous Ideas alongside fellow contrarians Germaine Greer and Christopher Hitchens, but I caught up with him in the tiny auditorium at the NSW Users and AIDS Association in Surry Hills.

I asked Stamper why, given its catastrophic failure, the war on drugs still has overwhelming political support.

Almost all US States have become addicted to federal money to fight drugs, Stamper told me. "The toughest nut to crack will be the prison industrial complex — especially the private ones. You get paid per prisoner per day. Why would you want to reduce crime?" And the public prison system is also invested in the status quo. "Powerful unions have strong memberships in the prison system. They tend to present any policy that reduces incarceration as a threat to job security and their own survival."

No wonder the United States, with 5 per cent of the world’s population, has 25 per cent of the world’s prisoners. Of these, 500,000 are in prison on drugs charges — 1200 per cent more than in 1980.

Before you shake your head and mutter "only in America", the situation in Australia is appalling, too. The Australian Drug Law Reform Foundation conservatively estimates the annual costs of our drug war at $6.7 billion. Official stats are hard to come by, but according to Drugs and Law Enforcement (Spinney Press), in the decade to 2006 at least 50,000 people a year were arrested for drugs, the vast majority of which were for marijuana. In what should be an embarrassment to Australian law enforcement, 84 per cent of marijuana arrests in 2004-5 were for consumption.

It was one such petty bust that made the penny drop for Stamper. Like fellow officers, he’d accepted that possessing a joint was a felony, and that getting promoted meant racking up his arrest sheet. "I’m not proud of it, but the power went to my head," he confessed.

Then he kicked down the door of a 19-year-old kid and, "snatching the weed out of his flushing toilet", busted him. "Taking him down the station for a felony charge, I suddenly saw this kid as a fellow human being, and I’m busting him when I could be out solving real crime."

Stamper came to understand he was playing a role in the deadly narrative established by the prohibition policy of the 1920s and 30s. "Prohibition created the culture of violence. It invented Al Capone and the industry of dealing in illicit products for which there is unstoppable demand. Drive-by shootings, overdose deaths, massive profit and the violent defence of that profit against competitors. The whole framework got started with prohibition."

Stamper sees the horrors of the Mexican drug cartels — who have recently taken to leaving their beheaded victims in public places — as the latest extension of this violence and intimidation. The cartels and their members are now in 230 cities across the US — and everywhere they set up, cops are soon dealing, extorting and kidnapping, just like the cartels.

Some nations are tired of this spiral. After going along with America’s war on drugs for decades, Mexico, Argentina, Colombia, Venezuela, Ecuador and Bolivia have all backed away from a policy of interdiction. All are moving towards decriminalisation in attempts to free up funding and police to deal with the violence and to relieve prison systems bursting at the seams. They’ve done it quietly, so as not to offend the United States.

Stamper feels it bodes well that the US did not stand in the way of this. But he believes decriminalisation, while a useful first step that stops user arrests jamming the system, must be replaced with legalisation. "Only legalisation gets to the heart of the issue; it stops the trafficking, the violence and the corruption, by regulating and taxing the whole process. Then you can truly begin the process of harm minimisation."

It’s hard to fault the man’s determined optimism. But what about the pharmaceutical industry, I ask him, which is a fierce supporter of the war on drugs? Stamper concedes their power, but is sure "their opposition will dissolve once they see there’s a place for them at the table of a regulated market".

I also put to Stamper the commonly expressed fear that legalisation will just result in more users. "There’s a sustained production of misinformation about the real issues, the reason for the violence and corruption," he told me. "We need a massive education strategy to bring the truth to the mums and dads."

In Australia, the Drug Law Reform Foundation and the NSW Users and AIDS Association are recognised leaders on drug policy, education and management. Generally, the approach of drug advocacy groups is to fly under the radar. The Medically Supervised Injecting Centre (MSIC) in Kings Cross is a perfect example. Opened amid a media storm of scare-mongering and conservative moral panic in 2001, it has quietly become established. Why? Because it works. Staff have dealt with 2557 overdoses without a single death. Rather than encouraging drug use, MSIC has enabled health officials to build an accurate picture of drug use in Sydney. This allows them to devise policy based on facts instead of moral positions based on fantasies. It saves lives and saves money.

Tragically, shock jocks and church leaders still pipe a seductive tune for the unthinking — and political leaders remain too meek to confront them. This moronic tune needs to become the real target for anyone, politicians included, interested in ending our outrageously expensive and utterly failed attempt to prohibit drugs.

In the US, Stamper readily accepts that change will be incremental, but with "75 per cent of all Americans believing the war on drugs has failed, we have the critical mass of public opinion. It’s getting the leaders who whisper their support to speak out".