Imagine the Afghan war had run for the past 40 years. Imagine 2,000 deaths a year. The enemy remains 400,000-strong, despite 40,000 being taken prisoner annually. The war costs £1bn a month. Casualties vary from time to time, but there is no hope of victory. Were that the case, I suggest public opinion might be exasperated. Parliament might debate the matter. Ministers might review policy. Yet such is Britain's fatuously entitled "war on drugs". Each year governments re-legislate their "war on terror", despite the minimal threat, but reject any need to revise the 1971 Misuse of Drugs Act. They refuse to see if it is working, and do nothing but waste public money.

Home secretaries trumpet idiotic "drug seizures". They pass "awareness" budgets, arrest and imprison thousands of citizens for drug possession and sale. The war has failed. But it continues to immiserate countless families and wreck countless lives. It is stupid, knee-jerk British government at its worst.

There is now a small industry of liberals who spend their time saying so. I am probably one of them, having wasted hours on commissions, inquiries, conferences and lobbies. Worthy charities dole out money for fact-finding trips to California, Switzerland, Portugal and the Netherlands, all with lessons, none of which we learn. The British go abroad, as always, to have their prejudices confirmed. If we were investigating terrorism, the death rate would have ministers racing to the Commons in panic. As it is, the victims of the 1971 act die un-atoned.

I sat on Lady Runciman's Police Foundation inquiry in 2000 which, like her "Runciman Two" this week, reiterated that criminal law had failed to end drug use or reduce harm. It suggested we go easy on cannabis possession and concentrate on treatment for hard drugs. Prison was the wrong place for users or abusers. We dodged the question of supply, as does every drugs report, because that involves discussing manufacture, money and retail. Liberal Britain has always had a distaste for trade.

I hesitate to suggest that Runciman Two is a reprint of Runciman One, but after a decade of inaction I wonder what is different. The police have let up on cannabis possession, as in most countries, largely because they know that the law is unenforceable. This week's report says that government action is immaterial, drug consumption being unaffected by changes in classification, prison sentencing or education. Drug use seems to ebb and flow with price, fashion and, in the case of ecstasy and skunk, perceived harm. None of this stopped the home secretary, Theresa May, beating her chest and howling her rejection of Runciman from the rooftops.

Britain on drugs is where China is on hanging, Saudi Arabia on beating, Russia on censorship and the Taliban on girls' education. Drugs policy is the last legislative wilderness where "here be dragons", a hangover from days when abortion and homosexuality were illegal and divorce expensive. It petrified home secretaries of left and right alike, Jack Straw and Jacqui Smith as much as Kenneth Clarke and Theresa May. So scared was Tony Blair that Alastair Campbell had to order the smothering of the 2000 Runciman report.

Most sane politicians – including David Cameron – advocate reform in opposition, and again after leaving power. A phalanx of Latin American ex-presidents are in favour of cocaine legalisation, the so-called "formers". Yet fear grips the collective brain when in office. The mere word drugs gives every politician the heebie-jeebies and turns libertarians into control freaks.

As Jonathan Haidt has argued in his book The Righteous Mind, political attitudes on most things, certainly drugs, are irrational, rooted in tribe and upbringing. Politicians who stuff their brains with alcohol, nicotine and amphetamines view ecstasy, cannabis and cocaine as dangerous exotics, like the black death or yellow peril, imported from dusky parts to corrupt the young. They shudder at decriminalisation, relying instead on their favourite legislative juju – "sending a message" and washing their hands.

What should be researched is not drugs policy but drugs politics, the hold that taboo has on those in power, and the thrall that rightwing newspapers have over them. This has nothing to do with public opinion, which is now strongly in favour of reform. Most sensible people find the present regime disastrous and want drugs regulated, rather than the wild west that is the urban drug scene today. It is politicians who think "soft on drugs" implies some loss of potency.

Just as few recreations are harmless so are few recreational drugs. To imply otherwise is silly. But the sheer longevity of marijuana use has embedded it in youth culture alongside alcohol. The menace to public health comes from the failure of government to legalise, test and regulate supply, which is what it should do for all narcotics. Over-prescribing of benzodiazepines is now far worse, and more dangerous, than the over-prescribing of heroin in the 1960s, which led to its disastrous banning and proliferation. No one is proposing to ban legal drugs today, so why leave illegal drugs, and their users, to the tender mercy of crooks?

There is no reason in all this. We are dealing with the darkest of bourgeois taboos. Of all the things on which the world has declared "war" in modern times, self-harming substances must be the daftest. Yet the result has been to destroy millions of lives, expend trillions of dollars, and helplessly corrupt sovereign states, from Afghanistan to Colombia. It is the greatest single failure of modern statecraft. It is the dark ages, and we are still in them.