The past two months have been an incredibly busy and productive time for global drug policy reform and scientific research into psychedelic drugs. Never in my 15 years as director of the Beckley Foundation have I seen such rapid progress in these complementary fields.

Earlier this week, Imperial College and the foundation hosted a forum on psychedelic drugs research, which discussed the breakthroughs that have been made by our organisations' research programme and heard presentations from scientists from around the world. And in April, the three-day Psychedelic Science Conference took place in San Francisco, also co-hosted by the Beckley Foundation.

At last the potential value of these compounds is beginning to be explored. Not only are they gateways to understanding consciousness itself, they also – alone or as aids to psychotherapy – open up valuable new avenues of treatment for many human ills, including depression, cluster headaches, anxiety, addiction and post-traumatic stress disorder. After many years in the darkness, we are finally starting to uncover the potential of these compounds, which inter-relate with human neurochemistry so intimately that they bring about radical changes in consciousness that – with careful handling – can be channelled into treating disease and transforming awareness.

Research into the potential of these drugs has been blocked for the past 40 years by their being scheduled as "controlled substances". This has surrounded them with taboos and regulations that still make research extremely difficult and expensive, deterring all but the most determined explorers. I am fortunate to have been collaborating for the last few years with leading researchers including David Nutt of Imperial College, Valerie Curran of UCL and Roland Griffiths of Johns Hopkins University, Maryland.

After decades hidden away, the Cinderella of science is putting on her slippers and setting off for the ball.

The day before the London conference I returned from Guatemala, where I had been invited by President Otto Pérez Molina to attend the General Assembly of the Organization of American States in the beautiful ancient capital of La Antigua. The assembly discussed a report on The Drug Problem in the Americas, commissioned by the OAS last year and launched at a ceremony at the presidential palace in Bogotá that I attended in May.

The report and the meeting mark a paradigm shift by initiating intergovernmental dialogue that recognises the need for greater flexibility, pragmatism and respect for diverse opinions. As the OAS general secretary put it, "We are knocking on sensitive doors, but we need to examine drug policy."

Even the US representative, secretary of state John Kerry, said that efforts must focus on the treatment of users and on education rather than incarceration. At last, a major international body is recognising that the dominant prohibitionist approach to drug control laid down by the UN drug conventions of 1961, 1971 and 1988 must be reformed. There is near-unanimous agreement in Latin America that the "war on drugs" has failed, with leaders becoming increasingly vocal in their determination to push the reset button.

Now former and present presidents of the world, and other leading global figures, publicly acknowledge the need for reform – as expressed by the Global Commission on Drug Policy and by the signatories of the Beckley Foundation Public Letter. President Santos of Colombia has called on the world to find new ways to tackle the scourge of the illegal drugs trade, including the possibility of legalisation with regulation.

President Pérez Molina, who last year invited the Beckley Foundation to advise him on drug policy reform, is the leading international advocate of the urgent need for a new approach. He is accompanied by President Santos and other Latin American leaders such as Presidents Mujica of Uruguay and Morales of Bolivia, whose countries have suffered so grievously as a result of the prohibitionist approach of the past 50 years, and who now look for new solutions to improve health, security, development and education.

We are delighted to advise President Molina and his government, as in my view the misguided approach to the control of illicit drugs has caused much more damage than the drugs themselves. The Beckley Foundation has always advocated an evidence-based, health-oriented, harm-reducing, cost-effective approach that respects human rights. The mark of a successful drug policy should not be the amount of drugs intercepted or the number of people in jail, but rather the security and stability of the state's institutions, and the health and wellbeing of its citizens.

We need to remember that, according to the UN's estimates, only 5% of the world's population use drugs and fewer than 12% of those become problem users. Drug misuse needs to be treated as a health problem. People who use drugs without causing harm to others should no more be criminalised than moderate users of alcohol.

Even within the citadel of the USA, the architecture of prohibition is beginning to crumble: last November, the states of Colorado and Washington voted to legalise the possession of limited amounts of cannabis – even for recreational use. Cannabis policy is progressing towards decriminalisation and regulation. Couldn't all drug-related harms be more effectively reduced within a strictly regulated market than they can under the current criminalised regime, which is completely unregulated?

After 50 years of prohibition, drugs are cheaper and more available than ever before. The collateral damage – particularly to countries that produce the drugs and those through which they pass – is devastating. Surely the governments of the world can do a better job of limiting harms than the cartels, whose only motivation is profit, and who are the principal beneficiaries of the present approach?