Even Alan Johnson must know his sacking of David Nutt was a mistake. The boast that he was being "big enough, strong enough, bold enough" to make such decisions was a gift to the gods of hypocrisy. If he was that big and strong he would have ignored Nutt and not pretended that an academic lecture on drug classification constituted a "public campaign" against him. Nutt's Advisory Council on the Misuse of Drugs had been humiliated by Johnson and his colleagues, and rendered virtually useless. Leave the guy alone.

It is not the mistakes politicians make that matter, but why they make them. The Labour government's drugs policy must qualify as the worst confection of unreason even in Whitehall's copious canon. This is not for want of advice or research. Few subjects have been more rigorously investigated, not least by Nutt and his collapsed committee.

We know the differential impact of narcotics on the brain. We chart the evolution of schizophrenia in drug users. We can measure harm reduction schemes across Europe. We can even balance the impact of education against deterrence in curbing drug use. When I hear of another committee, conference or seminar on drugs policy I scream: "Don't waste the money: spend it on rehab instead."

Researching drug use is pointless since policy on the subject has nothing to do with evidence, only emotion. It has to do with fear of the unknown, the taboo of other people's escapist narcotics (or worse, those of one's children). Politicians could not care less what experts say – witness this week's smattering of support for Johnson. They care only for the rightwing press, whose editors suffer a similar taboo.

The test was how the Tories reacted to Nutt's sacking. Faced with a home secretary gasping for air, Cameron and his home affairs spokesman, Chris Grayling, rushed forward with oxygen. Parting company with half the cabinet and the weight of scientific opinion, Cameron had a bad attack of funk. He refused to defend Nutt, and asserted his conviction that ecstasy was as harmful as heroin and crack cocaine. This was the same Cameron who, as a backbench member of the home affairs select committee in 2001, had supported Nutt in taking the opposite view. He must know what he said this week was rubbish.

All these politicians accept in private that the law is in chronic need of reform. Yet should they dare murmur so, they seem terrified of being assailed by the Mail, the Sun and the Telegraph. They could handle the House of Commons. They could even carry their constituents. But the rightwing press holds them in thrall, perhaps because they feel powerless before its lash. Might their youthful indiscretions be discovered, or the antics of their children pursued?

Politicians can stand the pressure of corpses piling up in Helmand, but one corpse at a rave would be too much for their consciences. Whenever I have tackled Home Office ministers, from Jack Straw and Charles Clarke to recent, less distinguished holders of the office, the response is the same. Don't even think about it, they cry. We would be crucified by the press. Just say no to drugs reform.

I served on the 2000 police foundation committee on the 1971 Misuse of Drugs Act, the only exhaustive study of the act ever undertaken. It was set up with the government's blessing and members included David Nutt, distinguished pharmacologists and two chief police officers. Our conclusions were mild, embracing a redirection of drugs policy towards harm reduction and a partial decriminalisation of cannabis use.

Polling evidence showed a wide gulf between a public desire for toughness on hard drugs on the one hand; and on the other, two-thirds of opinion that regarded cannabis as "least harmful". An overwhelming majority thought chasing cannabis users was "not a police priority", and a significant majority, from all ages and social groups, favoured cannabis decriminalisation. That was confirmed in other similar polls.

What happened next was a textbook case of Tony Blair's governing style. The home secretary, Jack Straw, went ape, reputedly on the instructions of Alastair Campbell, then at the height of his Downing Street ascendancy. They feared that the slightest welcome for the report's findings might have the government castigated by the rightwing press, of which Campbell lived in perpetual fear. The committee's chairman, Ruth Runciman, was summoned in advance of publication and castigated by Straw in front of his team, until Mo Mowlam had to suggest it might be better if they all read the document first.

When the report appeared it was well received. The Daily Mail, in a front-page editorial, welcomed it and said it had delivered "a mature and serious national debate". The Telegraph was even more favourable and criticised Straw for "misjudging the public mood". The head of the Metropolitan police was supportive.

In other words it was quite untrue that the public and press were opposed to drugs law reform. Realising this, Straw performed a U-turn and was induced, apparently by Campbell, to write an article full of wishy-washy assertions for the News of the World. It warmly welcomed the report and further debate. There was none. The subject was buried.

The incident was a classic example of public policy determined by ministers trying to second-guess Fleet Street. Drugs policy is desperately important. It has the power to wreck lives, families and communities. It underpins a third of crime and 80% of acquisitive crime. Four decades of illegality have done nothing to curb consumption, merely breeding the most lucrative, untaxed product market in Britain. No country has achieved the remotest success with prohibition, but Britain's archaic laws have been the least successful. Go to any deprived area, any difficult school, any failing social service, and the root cause of trouble is drugs.

There is no evidence that the public is averse to reform of the 1971 law, indeed the opposite. Why senior politicians should accord mystical influence to a few irrationalist newspaper editorialists is bizarre. Ministers and opposition leaders disregard the press on war and peace, on indulging banks, and on infringing civil liberties. The media's bluff is called every day on some topic or other – and rightly so. The press, like the pope, can field no divisions.

So what is it about drugs? Britain's deepest social problem is blighted by political cowardice towards an outdated taboo. But who will break the spell?