Four decades ago, on 17 July 1971, President Richard Nixon declared what has come to be called the "war on drugs". Nixon told Congress that drug addiction had "assumed the dimensions of a national emergency", and asked Capitol Hill for an initial $84m (£52m) for "emergency measures".

Drug abuse, said the president, was "public enemy number one".

But as reported the following morning in our sister newspaper, the Guardian, the president's initiative appears to have been primarily motivated not by considerations of the ghettoes or Woodstock festival, but by addiction among soldiers fighting in Vietnam: the first and immediate measure in the "war on drugs", implemented 40 years ago this weekend, was the institution of urine testing for all US troops in Indochina. The Guardian's "sidebar" story to the news bulletin was not from Chicago or Los Angeles but the Mekong Delta, with soldiers laughing: "You can go anywhere, ask anyone, they'll get it for you. It won't take but a few seconds."

Nixon signed his war on drugs into law on 28 January 1972, Adam Raphael quoting him in this newspaper as saying: "I am convinced that the only way to fight this menace is by attacking it on many fronts." The catchphrase "war on drugs" mimicked that of Nixon's predecessor Lyndon B Johnson, who had declared a "war on poverty" during his state of the union address in 1964.

Four decades on, in a world (and an America) accursed by poverty and drugs, there is almost universal agreement that the war on drugs has failed as thoroughly as that on poverty. In the US and Europe, the war has been fought on the streets, in the courts and through the jail system, to no apparent avail. In the world that has "developed" since 1971, it has been fought in the barrios; it has defoliated land and driven peasants into even worse poverty. The war in the so-called "producing" countries has ravaged Colombia, is currently tearing Mexico apart, and again threatens Afghanistan, Central America, Bolivia, Peru and Venezuela. In places such as west Africa, the war is creating "narco states" that have become effective puppets of the mafia cartels the war has spawned.

The drugs themselves have wrought misery and havoc across the planet, and continue to do so. According to the United Nations, in an exhaustive report by a global commission on drugs published this summer, worldwide opiate consumption increased by 34.5% between in the two decades to 2009, and that of cocaine by 25%. The UN estimates the drug business to be the third biggest in the world after oil and arms, worth £198bn a year. The former head of its office on drugs and crime, Antonio Maria Costa, posits that the laundered profits of the narco-trafficking underworld by the "legitimate" financial sector is what kept the banks afloat for years before they finally crashed in 2008.

But while Costa (and I, for what it's worth, after three years covering the Mexican drug war) advocates going after the money as the most urgent priority, most of the lexicon in the now burning debate about what to do in the wake of the drug war's manifest failure concerns decriminalisation, or even legalisation.

There has been a campaign for the legalisation of drugs in the US ever since the first state ban on marijuana in 1915.

Now President Barack Obama's drug tsar, Gil Kerlikowske, carefully describes America's own war on drugs as "unhelpful". Last month, former president Jimmy Carter wrote in the New York Times that "excessive punishment" has "destroyed the lives of millions of young people and their families"; drug policy, he said, should be "more humane and more effective".

Obama has entirely changed the language of the US's relationship with Mexico, conceding "co-responsibility" for the dual catastrophe of violence south of the border and addiction north of it. Experts such as David Shirk, of the Trans-Border Institute in San Diego, say that "the legalisation of marijuana in the US within 10 years is an inevitablity".

The UK has traditionally been a hardline participant in the war on drugs, but in opposition David Cameron said: "Drugs policy has been failing for decades." Professor David Nutt of the Independent Scientific Committee on Drugs, who has become synonymous with the anti-prohibition movement, says that "the obscenity of hunting down low-level cannabis users to protect them is beyond absurd".

In Europe, the Netherlands famously refuses to criminalise cannabis users, while Portugal became the first European country, in 2001, to abolish criminal penalties for personal possession of all drugs, sending addicts for counselling instead. Italy has decriminalised possession of less than half a gram of most illegal substances.

Prohibitionism, however, remains strongly supported by most law enforcement agencies in the US and UK. But prohibitionism has its creative and radical exponents too, not least Costa, who argues: "Why open the floodgates to addiction by increasing access to drugs?" He wrote for this newspaper: "Maybe western governments could absorb the health costs of increased drug use, if that's how taxpayers want their money spent. But what about the developing world? Why unleash an epidemic of addiction in parts of the world that already face misery and do not have the health and social systems to cope with a drug tsunami?"

Costa proposed that "drugs should be regarded as a health issue"; he wanted to "reduce vulnerability to drugs in regions of the world where governance is weak", and – calling the bluff of rich countries that host the big banks – "get serious about organised crime".

The most compelling and sophisticated initiatives in the debate 40 years on from Nixon's declaration come from the region that has arguably suffered more than anywhere from the ravages of drug production, trafficking and now addiction, too – Latin America.

Last August, Argentina's supreme court ruled it unconstitutional to punish people for using marijuana for personal use. Even Mexico, which has since 2005 been the theatre for a singularly vicious drugs war, has elected to legalise limited amounts of all drugs for personal use, for example: 0.5g of cocaine, 40mg of methamphetamine and 50mg of heroin. Felipe Calderón, the president, has called for "a fundamental debate on the legalisation of drugs", even though he opposes such a policy himself.

A landmark proposal was made by three former presidents: Ernesto Zedillo of Mexico, César Gaviria of Colombia and Fernando Henrique Cardoso of Brazil. It opened with the salvo: "Prohibitionist policies based on eradication, interdiction and criminalisation of consumption simply haven't worked … The revision of US-inspired drug policies is urgent in the light of the rising levels of violence and corruption associated with narcotics."

Of the three, Colombia – which has just emerged from a ferocious drug war – is the most impatiently adventurous in its call for a complete global overhaul of drugs policy. Colombia's ambassador to London, Mauricio Rodríguez, says that this should be "based not on anyone's political or ideological agenda, or any media agenda, but on the economic and social data and information we do not yet have, and must acquire"

"Our president has said very clearly that this is the time for a deep analysis of what has happened over the past 40 years, and to learn the lessons of the mistakes that have been made," Rodríguez said in an interview with the Observer last week. "And we have to evaluate every alternative, without excluding any possibility – from complete legalisation to a second, different, war on drugs."

The discourse, he says, "can no longer be that of this country or that, but a serious global discussion". It cannot even, he says, make the conventional distinctions between "producing" and "consuming" nations, "since we now have widespread consumption in the so-called 'producing' nations and, with synthetic drugs, significant production in the 'consuming' nations".

The global horizon is necessary, he says, "because we want to be sure that what is good for Colombia is good for the rest of the world, and vice versa. We are not saying that we have a solution; we are saying that we do have some moral authority because we have lived 24 years in hell – been through a painful experience which has cost us thousands of lives lost, hundreds of millions of dollars lost, the reputation of the country lost. We are now re-establishing that reputation, and stabilising the country, but the problem has moved elsewhere – to Central America, Mexico and West Africa.

"And so we realise," Rodríguez continues, "that we need to start afresh with a global and entirely different approach, different analyses which are social, economic – and different solutions. I've been studying this for 25 years, and have heard too many superficial analyses: that this is to do with leisure, or this is a problem of mafia. We need a more complex analysis of the deep roots of this problem – we cannot fool ourselves about the root causes of the tragedy of drugs in both the developed and developing world, they are to do with poverty and deprivation, young people without access to employment, with no access to education. They are about alienation, fear and anger."

Among the strategies he wants to see pursued are, first, "detailed economic work on these causes and the ways of combating them". Second: "Let's be serious about where the big money is. If you look at the trail of cocaine, you'll find that 5% of the profits remain in the producing countries; 95% is in the distribution networks and laundered. The big money is in the big banks in the big countries; the big money is in the US, Europe and Asia." And third: "Let's talk about the human rights issues."

Rodríguez discusses a programme his embassy has launched in collaboration with the police in Scotland called "Sniffing the rainforest". "We tell the young people that one gram of cocaine involves the destruction of four square metres of rainforest. That it costs hundreds of lives – and although they are taken to drugs by economic alienation, the response to this kind of language has been positive." We also tell them, he adds, "what else is involved in sniffing cocaine: gasoline, cement, sulphuric acid – and they listen to that".

It was while working along the US-Mexican border, in an inferno of violence and addiction, that I came to see the wisdom of the proposed Colombian strategy. In the rehab clinics and wretched outskirts of Ciudad Juárez, where drugs are legal for personal use and easier to obtain than soft drinks, I developed a problem with the scope of the lexicon in the UK and, to a lesser extent, the US, which too often presumes that people take drugs for reasons of "recreation", rather than out of desperation and despair.

This view of drugs as stimulating entertainment may hold true of Camden Lock in London and the capital's West End clubs, but not of São Paulo or even the valleys of south Wales, let alone the US-Mexican border. What may work for Notting Hill might not work in Rhondda, let alone Tijuana.

In Tijuana in 2009, addicts could not believe their luck – those arriving at a Narcóticos Anónimos session were amazed that possession of up to four lines of cocaine or 50mg of heroin was now legal. Juan Morales Magana, 17, a windscreen-washer and registered methamphetamine and heroin addict, was working out how many hits the legal limit of 40mg of meth would get him, though his counsellor, an evangelical pastor, was ambivalent: "I wouldn't want anyone to think that just because it is legal, one should live like this for fun. Drugs are the scourge of our society. All this can do would limit killing between small-time cholos [gangsters] for street-corner turf. The danger is that kingpins will accelerate the domestic market if possession is legal and smuggling into the US more difficult."

"Personally, I sometimes wish drugs would be made legal so that the gringos can get high and we can live in peace," said Tijuana police officer Elisio Montes, whose two best friends, his former boss and assistant, were murdered by executioners for the cartels. "Then I say to myself: no – these drugs are addictive after one single hit. They're terrifying, they destroy lives, they destroy our young people. If they're legal, they'll buy more."

Writing in the Times, Antonia Senior insists that: "Drugs policy must start from the premise that teenagers like taking drugs, because drugs make them feel good." I would rather it started from the premise that life in most places is so awful that it leads to catastrophic addiction such as that in the barrios of Honduras or, indeed, the back streets of Naples or Swansea.