There is no evidence that tough enforcement of the drug laws on personal possession leads to lower levels of drug use, according to the UK government’s first evidence-based study.

Examining international drug laws, the groundbreaking Home Office document brings to an end 40 years of almost unbroken official political rhetoric that only harsher penalties can tackle the problem caused by the likes of heroin, cocaine or cannabis.

It is signed off by the Conservative home secretary, Theresa May, and the Liberal Democrat minister Norman Baker, and will be published alongside an official expert report calling for a general ban on the sale and trade in legal highs.

Baker said the international comparisons demonstrated that “banging people up and increasing sentences does not stop drug use”. He said the last 40 years had seen a drugs debate in Britain based on the “lazy assumption in the rightwing press that if you have harsher penalties it will reduce drug use, but there is no evidence for that at all”.

Baker added: “If anything the evidence is to the contrary.”

The minister added that wider societal factors, such as a more risk-averse generation of young people, who suffered fewer alcohol problems and were healthier, contributed to the general downward trend in drug use.

It documents in detail the successes of the health-led approach in Portugal combining decriminalisation with other policies, and shows reductions in all types of drug use alongside falls in drug-related HIV and Aids cases.

The Home Office international research paper on the use of illegal drugs, which redeems a Liberal Democrat 2010 election pledge for a royal commission to examine the alternatives to the current drug laws, also leaves the door open on the legalisation experiments in the American states of Washington and Colorado, and in Uruguay.

Medicine Man Denver is the single largest legal medical and recreational marijuana dispensary in Denver, Colorado. Photograph: Alamy

It says that “it is too early to know how they will play out but we will monitor the impacts of these new policies in the years to come”.

Regarding legal highs, Baker said the government would look at the feasibility of a blanket ban on new compounds of psychoactive drugs that focused on dealers and the “head shops” that sell tobacco paraphernalia rather than users.

“The head shops could be left with nothing to sell but Rizla papers,” Baker said. “The approach of a general ban had a dramatic effect on their availability when it was introduced in Ireland, but we must ensure that it will work here.”

A ban would apply to head shops and websites. Legal highs are currently banned on a temporary 12-month basis as each new substance arrives on the market. Legislation is possible before the election but not certain.

The new blanket or “generic” ban would not be accompanied by a ban on the possession or use of the new psychoactive substances, which often mimic the effects of traditional drugs. This would remain legal.

It is expected the expert report on legal highs will recommend a threshold for substances to be banned so that those with minimal psychoactive effects such as alcohol, tobacco, tea and coffee would not be caught by the proposed new ban.

The report firmly rejects a New Zealand style-approach of regulating head shops and other sales outlets for legal highs.

Publication of both reports has been held up for months as interminable negotiations between the two coalition parties have gone on over every detailed issue.

Baker has repeatedly warned of the dangers of legal highs, citing evidence that some cannabinoids synthesised in chemical labs are 100 times more powerful than traditional strains of cannabis.

The expert report says there were 60 deaths related to new psychoactive substances in 2013 – up from 52 the year before.

It also considers basing future controls of the effect on the brain rather than the current test of their chemical structure.

Frontline health staff are also urged to receive strengthened training to deal with their effects.

Danny Kushlik, of the Transform drugs charity – which campaigns for drug legalisation, said the international report represented a landmark in British drugs policy since the introduction of the 1971 Misuse of Drugs Act that is still in force today.

“This is a historic moment in the development of UK drug policy. For the first time in over 40 years the Home Office has admitted that enforcing tough drug laws doesn’t necessarily reduce levels of drug use,” said Kushlik.

“It has also acknowledged that decriminalising the possession of drugs doesn’t increase levels of use. Even more, the department in charge of drugs prohibition says it will take account of the experiments in the legal regulation of cannabis in Washington, Colorado and Uruguay.

“Pragmatic reform will only happen if there is crossparty support for change and we can assume now that the Labour party can engage constructively on this previously toxic issue.”

A Home Office spokesperson, responding to the evidence of the international report, said: “This government has absolutely no intention of decriminalising drugs. Our drugs strategy is working and there is a long-term downward trend in drug misuse in the UK.

“It is right that we look at drugs policies in other countries and today’s report summarises a number of these international approaches.”

Earlier this year the deputy prime minister, Nick Clegg, pledged to abolish prison sentences for the possession of drugs for personal use – including class-A substances such as heroin and cocaine. He urged David Cameron to look at issues such as decriminalisation or legalisation of drugs.