By any standard, the global war on drugs has been an abject failure. Since the “war” was declared by President Richard Nixon in 1971, we have spent over £1tn trying to eradicate drugs from our societies. Yet the criminal market continues to grow, driving unimaginable levels of profit for organised crime. We devote vast police, criminal justice and military resources to the problem, including the incarceration of people on a historically unprecedented scale.

In many parts of the world, drug violence has become endemic. As Mexico’s president, Enrique Peña Nieto, visits the UK, we should remember the estimated 100,000 people killed in Mexico alone since 2006. Yet tragically, the sum total of enforcement efforts against drug supply over the past 40 years has been zero. Efforts at reducing demand have been similarly fruitless. Here in the UK, a third of adults have taken illegal drugs and the gangs are doing a roaring trade. The problem simply isn’t going away.

It’s no wonder that countries around the world are rethinking their approach. Former prime ministers and presidents are now admitting the mistakes of the past and pushing for change, through the Global Commission on Drug Policy and other bodies. Serving politicians are increasingly speaking out, with President Juan Manuel Santos of Colombia leading the way. Senior law enforcement officers are starting to question the idea that we can arrest our way out of this. Perhaps the most startling change has been in the United States, the country that invented the drugs war, where four states have recently voted to tax and regulate cannabis sales. Closer to home, Portugal, the Netherlands, Spain, Switzerland, the Czech Republic and Denmark have all introduced reforms aimed at reducing harm and cutting crime, ranging from heroin prescribing to decriminalisation.

Portugal's reforms have allowed resources to be redirected towards the treatment system

The west is undergoing a tectonic shift; but the UK seems oblivious to it.

And yet we desperately need better solutions in this country. One in six children aged 11 to 15 is still taking drugs; 2,000 people die each year in drug-related incidents; the use of unregulated “legal highs” is rampant.

At the same time, the police are stopping and searching half a million people a year for possession of drugs, prosecutions of users are close to record levels, and prison cells are still used for people whose only crime is the possession of a substance to which they are addicted. This costs a lot of money, which could be better spent on treatment and on redoubling our efforts to disrupt supply. And it wrecks the lives of 70,000 people a year who receive a criminal record for possession and then find themselves unable to get a job.

As an investment, the war on drugs has failed to deliver any returns. If it were a business, it would have been shut down a long time ago. This is not what success looks like.

The idea of eradicating drugs from the world by waging a war on those who use them is fundamentally flawed for one simple reason: it doesn’t reduce drug-taking. The Home Office’s own research, commissioned by Liberal Democrats in government and published a few months ago, found that “there is no apparent correlation between the ‘toughness’ of a country’s approach and the prevalence of adult drug use”. This devastating conclusion means that we are wasting our scarce resources, and on a grand scale.

The standard political response to this is to say that we must stay the course – that if we arrest a few more people, seize a few more shipments, then “victory” remains in our grasp. And conversely, that if we dare to do anything differently then we are playing Russian roulette with people’s lives. The gulf between the rhetoric and the reality could not be greater. The status quo is a colossal con perpetrated on the public by politicians who are too scared to break the taboo.

Facebook Twitter Pinterest ‘The police are stopping and searching half a million people a year for possession of drugs.’ Photograph: Stuart Emmerson/Alamy

So what is the alternative? For this, we should look to Portugal, which removed criminal penalties for drug possession in 2001. Portugal’s reforms have not – as many predicted – led to an increase in drug use. Instead, they have allowed resources to be redirected towards the treatment system, with dramatic reductions in addiction, HIV infections and drug-related deaths. Drugs remain illegal and socially unacceptable, as they should be, but drug users are dealt with through the civil rather than the criminal law. Anyone who is arrested for drug possession is immediately assessed and sent for treatment or education. If they fail to engage, they have to pay a fine.

The Portuguese system works, and on an issue as important as this, where lives are at stake, governments cannot afford to ignore the evidence. We should set up pilots to test and develop a British version of the Portuguese model. The evidence suggests it will be cheaper, more effective at reducing harm and would allow the police to focus their attention where it should be, on the criminal gangs that supply the drugs.

Now is the time for politicians of all parties to count the costs of four decades of failure, and embrace the call for reform. If we really want effective solutions to drug markets and to the harm caused by drug use, it will take political courage and fresh thinking.