Can Britain ever kill its worst taboo? This week’s news of the soaring prevalence of skunk, in place of weaker and less harmful herbal forms of cannabis, is appalling. With other news of prison chaos due to an epidemic of artificial cannabis (spice), government drugs policy is devoid of coherence – and clearly lethal. Deaths from drug misuse are now at an all-time high.

You can tell the state of a society, said Alexis de Tocqueville, by its prisons. Today you tell it by its attitude to drugs. Ten years ago the Labour government recklessly upgraded cannabis from class C to class B, “because of concerns of its impact on mental health”. Then they were minimal. After reclassification, police evidence is that 95% of cannabis used is now skunk. Older forms of “weed” are vanishing.

I regard drugs as the most serious avoidable cause of mental disorder, misery, crime and imprisonment, and have done my time on drug reform committees, including the 2000 Police Foundation inquiry. We advised a downgrading (though sadly not legalisation) of cannabis. Labour’s home secretary, Jack Straw, was horrified, and his successors reacted by upgrading.

All evidence I saw on narcotics led to two conclusions. One is that most psychoactive drugs, which include alcohol and nicotine, are a bad idea and some are unquestionably dangerous. The other is that, since suppression is impossible, criminalisation leaves supply in the hands of crooks and is therefore a menace. It creates crime at every point in the market and makes all drugs more dangerous – as prohibition did alcohol. The only answer is legalisation and control.

British drug consumption has marginally fallen over the past decade, but with a shift from softer drugs to harder ones. The rise is of chemical and synthetic drugs that are easy to adulterate. As for skunk, Marta Di Forti of King’s College London said of this week’s findings: “The increase of high-potency cannabis on the streets poses a significant hazard to users’ mental health.” It induces personality changes, memory loss and deterioration. It can wreck young lives.

Narcotics will never be eliminated – nor have they ever been. But driving them undercover makes it impossible to license or monitor quality. The government cannot even police its own prisons, Britain’s most concentrated market for spice and for the lethal opioid fentanyl. While some police forces such as Durham have, along with most prison governors, de facto decriminalised cannabis by declining to prosecute possession, others are less liberal. Even outside prison, the police over the past five years have reportedly locked up 30,000 cannabis users overnight at a cost of £13.5m. As for spice, last year Manchester police recorded 58 ambulance callouts to victims in one weekend alone. The NHS must sweep up the home secretary’s mess.

A common mistake is to legalise consumption but leave supply criminalised, unlicensed and therefore untested

Every move towards liberalising Britain’s drug laws is opposed by every government. Reform has been proposed by doctors, backbenchers, police chiefs, the press and pressure groups from right and left. Even the Mail on Sunday once featured a Mori opinion poll supporting legalisation. Yet in 2014, when MPs investigated foreign experience with legalisation, the home secretary, Theresa May, was so scared of the findings she sought to censor publication. She reflected a deep cultural aversion to letting other people enjoy, or make a mess of, their own lives. Her attitude to drugs was, like the Saudi attitude to women, an ingrained prejudice, deaf to reason.

British policy is now completely adrift of the rest of the western world. It is light years behind Donald Trump’s America, where half the states have legalised cannabis to some degree. In 1990 Trump himself said: “You have to legalise drugs to win the war.” Even today he supports medicinal cannabis. In California, recreational marijuana will be freely available in licensed premises from next month.

The US cannabis market is predicted to reach $50bn in 10 years. Colorado’s state revenues have already been transformed by an annual $200m in drug taxes, with the money going to schools and clinics. There are even courses for “cannabis sommeliers”.

Canada is bidding to become a major producer of marijuana. In Portugal, the Netherlands, Denmark, Switzerland, Germany – almost everywhere – hard and soft drugs are being brought under state regulation. In the Netherlands there is no market for synthetic cannabis, as the real stuff is legal. Dutch prisons are being closed down, or rented to Norway to keep going.

Of course experience is mixed. A common mistake is to legalise consumption but leave supply criminalised, unlicensed and therefore untested. In some countries consumption has risen, in others fallen. Nowhere has society collapsed. Everywhere, drug-related crime has been reduced.

As it is, the market booms thanks to the internet. The UN drugs agency last year recorded some 700 “new psychoactive substances”, flooding both the open and the dark web. Britain’s response, almost unique, is to turn a blind eye and leave its virtually open market unpoliced. The 2016 Psychoactive Substances Act on “legal highs” made no effort to discriminate good from bad. It just banned everything – except of course alcohol. It pretended that “something had to be done”.

Back in 2002 a certain opposition MP pleaded for an end to Britain’s “lowest common denominator” approach to drugs, to “posturing with tough policies and crackdown after crackdown” – all to no effect. The MP was David Cameron, who went on to oversee the passage of the 2016 act. The truth is that the prevalence of harmful drugs in Britain is not caused by policy. It is caused by politicians.

• Simon Jenkins is a Guardian columnist