It is 43 years since Richard Nixon, in need of a public enemy to shore up support for his snarling style of uncompassionate conservatism, declared war on a new target. "America's public enemy No 1 is drug abuse," he declared, warning Congress that the problem of narcotics had "assumed the dimensions of a national emergency".

Having risen to national prominence as an anti-communist campaigner, Nixon's new foe was the counterculture. His stance was widely assumed to be an attack on the hippie culture he so despised, with academics, writers and rock stars promoting the use of hallucinogens, but the media was also full of stories of clean-cut young men returning from Vietnam as junkies.

Nixon pushed new funds towards drug control agencies and backed tougher sentencing and policing. Marijuana, ludicrously identified as a "gateway" drug to heroin, was placed in the most restrictive category. Meanwhile, the United States used its muscle to ensure that the rest of the world joined one of the most futile, destructive and immoral wars the human race ever inflicted upon itself.

Download the new Independent Premium app Sharing the full story, not just the headlines

While the Vietnam War fades into history, thousands of people still die and millions of lives are ruined annually in this insane fight against drugs. Fittingly, given that it was launched by a president who turned out to be a crook, the biggest beneficiaries have been the most lethal gangsters on the globe as they battle over the immense spoils of an illegal trade that crucifies families and corrodes communities.

For more than four decades, the world has been hooked on its own addiction to this ludicrous war. More than one trillion dollars have been wasted on a punitive response to the human desire to get high. Meanwhile, the planet's political leaders ignored the mounting and incontrovertible evidence of their terrible failure: the destroyed families, the decimated cities, the devastated countries along with the improving purity, the falling prices, the widening range of products.

Slowly but surely, the world has begun waking up. It took time: two years before the start of this century, the United Nations stupidly declared that we would have a drug-free planet by 2008, committing member states to eliminate or significantly reduce use of opiates, cannabis and cocaine in a decade. Instead, global opiate use rose by more than one-third, with big rises also for cocaine and cannabis. Last year, the British Medical Journal found that street prices had declined over the past two decades, while potency increased.

As Margaret Thatcher said, you can't buck the market. Like it or not, many people want to take drugs; it is estimated that they are used by 5 per cent of the planet's adults. The finest law enforcement agencies and massive funding are no match for smugglers when there are mark-ups of more than 16,000 per cent. Even in the most well-protected prisons, drugs are available, while the might of American and British militaries failed to stop poppy production tripling in Afghanistan in a decade. What hope of our island nation guarding 12,000 miles of coastline when one year's supply of cocaine for the entire market could fit in a single shipping container?

For libertarians, the state simply has no right to dictate to people what they put in their bodies. Their outrage is all the greater when presidents and prime ministers admit to using drugs, yet governments run prisons crammed with people caught doing the same drugs or selling them, who mostly could not afford decent lawyers. Or when alcohol is socially acceptable, but the use of substances deemed less harmful by scientists is illegal. This hypocrisy is one reason for the dangerous breach in trust between politicians and their electorates, just as it widens the gap between police and the public. Use of drugs is, of course, a victimless crime. Little wonder that chief constables and spy chiefs press the case for reform of our self-harming drug laws.

I have sympathy with these libertarian arguments. But ultimately only one fundamental question should govern drug policy: how can the state ensure that people who use these products do the least harm to themselves and society? If you ignore cultural or historic hang-ups, there can only be one answer – the legalisation and regulation of all drugs.

This idea is often portrayed by ostrich-like opponents as the promotion of a druggie free-for-all. Yet the reality of reform could not be further from this crude caricature. In fact, it is a highly conservative yet progressive cause, an issue unusually popular with younger voters and with the ability to reconnect the Tories with long-lost sections of the community.

Indeed, it is hard to think of another policy with the same potential to challenge popular conceptions of conservatism. As I proposed to the Prime Minister and some of his closest advisers, the issue of drug reform clearly fits the modernising blueprint for party and nation. The idea was toyed with in the early days of David Cameron's leadership, then abandoned amid fear of hostile headlines. Since then, the world – and the British media – has moved on. The UN Commission on Narcotic Drugs is the annual get-together for combatants in the war on drugs. Member states discuss global drug controls and examine the effectiveness of the three key international treaties underpinning their mission.

Two years ago, the Czech Republic questioned the idea of illegality, suggesting that the UN adopt a new approach based on prevention and treatment rather than prohibition. This country has conducted a little-noticed experiment – decriminalising drugs for personal use under Vaclav Havel, then banning them, then decriminalising them again. A major study into this test case found that none of the key arguments for illegality stood up – but vast sums were frittered away that would have been better spent on treatment.

At this year's event in March, the Czechs were joined in pressing for an alternative stance by Colombia, Ecuador, Guatemala, Mexico and Uruguay. These are among the nations most damaged by drugs as feuding gangs fight over profits from transporting cocaine and heroin to North America and Europe. This cancerous trade now cuts through west Africa also; it was one reason for the recent collapse of Mali, as it fostered corruption and funded Islamic militants, in a clear case study of how this war on drugs backfires on development.

Uruguay is becoming the first country to legalise and regulate the production, sale and taxation of marijuana. As its courageous President, Jose Mujica, says, this measure targets the traffickers. "It's not a law supporting addiction," he told The Daily Telegraph. "It's a way of battling the black-market economy." Once, this would have provoked a furious response from Nixon's successors in the White House. But last year, the Organisation of American States issued a landmark report exploring the path from prohibition, reflecting concerns of leaders fed up with chaos and carnage in their countries.

The tide has even begun turning in the US, with two states legalising cannabis and two more set to follow after referendums later this year. California is expected to have a ballot in 2016 that, if successful, could spark the end of prohibition in bordering Mexico. As President Barack Obama says, it is wrong to have a law that is widely broken when only a select few get punished. "Middle-class kids don't get locked up for smoking pot, and poor kids do," he told The New Yorker in January.

The influential blogger Andrew Sullivan noted last year how the successful referendum campaigns in Colorado and Washington rebranded reform as a conservative measure. These campaigns were powered not by hippies seeking the right to smoke spliffs, but by parents concerned about children's safety. Advocates include such unlikely figures as Pat Robertson, the right-wing Christian evangelical, who said: "This war on drugs just hasn't succeeded."

These cannabis ballots are just the start. Mujica and other Latin leaders are now floating the idea of wider drug reform, while in the US the polls are shifting fast. A majority support legalisation of marijuana, a threefold increase in just 25 years. More significantly, two-thirds of Americans – including a majority of Republicans – favour greater emphasis on treatment rather than punishment for any drug use, with just a quarter wanting the focus on prosecuting users.

Drug dealers have also embraced the digital age, creating synthetic drugs sold online across borders. If the law steps in, chemists simply tweak composition to evade the ban – and there are thought to be some 250 of these new narcotics on the market. The Association of Chief Police Officers has pointed out the futility of constantly adding new drugs to the list of banned substances, given the speed with which the market provides replacements. New Zealand found a far better solution – clinical trials for toxicity, followed by strictly regulated sales from licensed vendors.

Although drug use is falling in Britain, this country still has the highest rates of drug use in Europe, with one in 12 adults and one in six older teenagers admitting having taken an illegal drug last year. All these people are putting their lives in the hands of dealers who use murder and mayhem to promote their illegal business. The tragic results are seen too often, such as with the spate of deaths of youngsters who thought they were taking ecstasy but were sold the far stronger para-Methoxyamphetamine (PMA).

Legalisation would replace this ultra-free market that exists to the benefit of the world's most vicious criminal groups with a system in which supply was controlled, products regulated and profits taxed. This is far safer for children, since parents will have more control than at present; it is safer for users, since the drugs can be tested for strength and purity; and it is safer for society, since it cuts off funding for the gangs that scar our cities and the cartels that carve up the world.

Current policies are staggeringly wasteful of taxpayers' cash, something that should always concern conservatives. One report found that more than £65bn is spent globally each year on enforcement – yet the booming illicit trade is the same size as the Danish economy, the 32nd biggest in the world. In Britain, annual public expenditure on treatment, policing and criminal justice in relation to drugs is £4.5bn, but the cost of cocaine has plummeted in recent years.

Drug reform should appeal to a Conservative Party seeking ways to connect with young and ethnic minority voters, who bear the brunt of street-enforcement strategies by police. These two groups are crucial to the party's long-term survival. Instead of resorting to misanthropic messaging and failed core-vote strategies aimed at frightened older generations, here is an issue offering something bold, conservative and modern that the party could take a lead on.

It makes sense on economic, political, social and moral grounds. It is also popular because, just as in the US, pressure for reform is growing in Britain. A poll by the campaign group Transform found that a majority favour permitting cannabis use, while four in 10 Britons favour total decriminalisation and more than two-thirds favour a comprehensive review of all drug policies. Support cuts across political divisions and embraces readers of all papers; some of the most fervent supporters are female readers of mid-market tabloids who see the damage done to families and communities.

Given the voices coming out in favour of reform, it is hardly even controversial these days. Ken Clarke MP, a relic from the jazz age, says that Britain is losing the war on drugs. In a chapter on drugs (which was later deleted) in his 1995 book Saturn's Children, Alan Duncan argued that the number of users would not increase following legalisation, while crime would fall quickly, as we saw following decriminalisation in Portugal. It is worth listening also to Labour's Bob Ainsworth, whose experiences as a Home Office minister turned him into an unlikely drugs campaigner; as he told me, the public are in a far more progressive place than politicians on this issue.

Prohibition is on its way out; one day, people will look back on it with as much bemusement as to the days when alcohol was banned in America. The Conservative Party should lead reform rather than continue to adopt a Canute-style stance against the tide of history. Already the Liberal Democrats are looking to set the pace, while Labour's shadow cabinet has discussed its position and the Ukip leader, Nigel Farage, backs reform. The Tories, whose leader showed courage and realism before taking office with calls for "fresh thinking" on this subject, should seize the opportunity to outflank them by proposing a total overhaul of drug laws instead of continuing to fight Nixon's futile war.

After all, what could be more conservative than a policy that is tough on crime, cuts public spending, protects children, safeguards families and aids global security?

Ian Birrell is a former speechwriter to David Cameron and co-founder of the music project Africa Express

This is an edited extract from 'The Modernisers' Manifesto', published by Bright Blue tomorrow; brightblue.org.uk