Standing on the podium at the United Nations in New York in June 1998, Kofi Annan declared: “It is time for all nations to say ‘yes’ to the challenge of working towards a drug-free world!” The leaders assembled at that meeting agreed: illegal drugs were to be eradicated from the face of the planet. They even set a deadline: 10 years to rid the globe of this scourge. A drug-free world by 2008.

We all know that drug use can cause great damage to individuals, their families and communities. Drug addiction can be miserable, debilitating, and often fatal. And a profitable illegal market only strengthens the organised crime groups that we fight so hard to contain.We want to get to grips with these problems. We expect our politicians to work together to reduce and eliminate suffering wherever possible.But with hindsight, the naivety displayed by world leaders in 1998 and at many previous and subsequent meetings is breathtaking. There is something profoundly misguided about the idea that human beings, who for thousands of years have taken an array of chemicals and plants for recreational and ritual purposes, would, in the space of 10 years, suddenly sober up at the request of the UN.

Far from shrinking, the global trade in illicit drugs has ballooned into a $330bn business with an estimated 270m “customers” and a foothold in every corner of the world, from the jungles of Latin America to the streets of European villages. In the face of so much suffering, we have no choice but to be honest and to call for an evidence-based and open-minded debate. The “war on drugs” has failed. The black market is booming. Criminal gangs are raking in huge profits and leaving a trail of violence and misery in their wake. More and more lives are being ruined and more and more families are being torn apart. It is time we stopped repeating the mistakes of the past.

The UN Office on Drugs and Crime paints a stark picture: “The illicit drug business is worth billions of dollars a year, part of which is used to corrupt government officials and to poison economies. Drug cartels are spreading violence in Central America, Mexico and the Caribbean. West Africa is under attack from narco-trafficking. Collusion between insurgents and criminal groups threatens the stability of west Asia, the Andes and parts of Africa, fuelling the trade in smuggled weapons, the plunder of natural resources and piracy.”

In Mexico alone, an estimated 100,000 people have been killed in drug-related violence since 2006. Much of the heroin, cocaine and cannabis produced in or passing through these countries is bound for our own shores. The demand for drugs from our own citizens means Europe is the biggest drugs market in the world after North America. While we haven’t suffered to anything like the degree that producer and transit countries have, millions of European citizens’ lives are affected by drugs and by public policy responses to drugs. In fact, the “war” very often seems to have been visited on the weakest members of our societies: people with drug problems who can’t get treatment, the victims of property crime committed by people trying to finance their drug use, vulnerable people pressured into becoming drug couriers, young lives and careers blighted by criminal records.

This is the desperate context to the decision, in 2012, by a group of heads of state from Mexico, Colombia and Guatemala, to call on the UN general assembly to convene a special session in April 2016 to look for options. That request was granted and governments across the world are gearing up for the meeting, which takes place in New York in April.

Those who aren’t familiar with the UN may be surprised to learn how much influence it has over the global system of drug control. There are three international drugs treaties, signed in 1961, 1971 and 1988. They commit the signatories to prohibit the use of recreational drugs and also regulate the supply of some of those drugs as medicines. These treaties are overseen by a set of UN institutions, based in Vienna. If we want a better system, one that truly works to protect human health and welfare, then the UN is the right place for that debate.

Some countries have decided to take bold unilateral steps. In the US, a number of states have voted to establish regulated recreational cannabis markets and the federal government has started to take steps to reduce the prison population. Uruguay has become the first nation state to do so and the new Canadian government has said it will follow suit. The Czech Republic and Portugal have decriminalised possession of small amounts of drugs so the state can focus resources on prevention and treatment rather than overcrowded prisons. Switzerland has pioneered supervised heroin-injecting clinics, a model Denmark, France and Ireland are following. The UK has led the way on harm-reduction measures such as needle exchange and methadone treatment.

In spite of these developments, the blunt truth is that Europe isn’t pulling its weight on the international stage. Preparations for the April summit are trundling along in the diplomatic undergrowth. It’s time for politicians to step in, joining forces with the presidents of Mexico and Colombia to push for an ambitious deal in New York.

What does a good deal look like? The best scenario is one in which the agreement published at the end of the summit eschews talk of the total eradication of drugs or punishing users and instead calls for measures that promote public health, human rights, harm reduction and support and treatment for people who use drugs, while creating space for countries seeking alternative approaches to really explore them.

Above all, governments need to base their policies on the best available evidence, rather than political posturing. A good example is the Czech Republic, where a new law was passed in 1998 criminalising the possession of small amounts of drugs. The then government set up mechanisms to evaluate the impact of the change and concluded that the more prohibitionist approach wasn’t working. A new law was introduced in 2010. While drugs remain illegal, most people caught in possession of drugs for their own use receive an administrative fine rather than a criminal record.

In 2014, the UK coalition government conducted a major review of the international evidence from other countries with different approaches, both harsher and more liberal. The review found no evidence that criminalisation of possession led to fewer people taking drugs. The evidence supports an approach that decriminalises the user, but retains criminal penalties for those who profit from the drug market.

Six weeks ago, we saw what diplomacy can do at the Paris climate talks. The UN general assembly special session on drug policy may be the summit that no one has heard of, but the opportunity to mend the mistakes of the past and change the direction of the future puts it in the same category of importance. Kofi Annan strikes a very different tone than he did 18 years ago. Witnessing the effects of drug markets and misguided policies on his own region of West Africa, he recently observed: “Drugs may have destroyed many people, but wrong governmental policies have destroyed many more. Let us not repeat this mistake.”

• Bohuslav Sobotka is the prime minister of the Czech Republic. Nick Clegg is a Liberal Democrat politician and former UK deputy prime minister.