Thirty years ago this November, a DJ called Danny Rampling co-founded a club night in Southwark, London, that he called Shoom. Essentially, he was trying to capture what he had experienced that summer on the Balearic island of Ibiza: a celebratory subculture built around so-called house music, and the drug methylenedioxymethamphetamine (or MDMA), otherwise known as ecstasy, whose euphoric rush inspired Shoom’s name.

So began the revolutionising of British popular culture, and the decisive passage of an array of illicit drugs from the cultural fringes to the mainstream. The saga is long and complicated, but it reached an unlikely kind of denouement this week, when that well-known raver the Duke Of Cambridge paid a visit to a treatment centre near London’s Liverpool Street station, and asked a group of recovering addicts the kind of “massive question” guaranteed to instantly crash-land in the headlines. “There’s obviously a lot of pressure growing in areas about legalising drugs and things like that,” he said. “What are your individual opinions on that?”

Back in 1987, Rampling and his friends were in the vanguard of what was quickly known as acid house, which initially developed away from the attentions of the media.

But by the autumn of 1988, ad hoc raves around the M25 and two ecstasy-related deaths (one of which was caused by ingesting 18 tablets) had sent an alliance of MPs, senior police officers and journalists into one of their customary panics. Six years later, the Conservative government passed the Criminal Justice Act, which stands as the only item of legislation to specifically outlaw gatherings of people fond of a particular musical genre, one characterised “as a succession of repetitive beats”. The authorities had clearly decided that what acid house had spawned was nothing but trouble; the clear intention was to kill it.

Inevitably, the opposite happened. We are now as far from the initial stirrings of acid house as its pioneers were from the arrival of rock’n’roll in the 1950s, but its legacy is immovable. Its implicit offer of a more peaceable, open-minded alternative to the standard Saturday-night staples of drinking and fighting was built into the culture for keeps. The music’s mixture of incessant rhythms and gurgling electronics is now an ingrained part of no end of successful pop. And ecstasy use sits at the heart of hundreds of thousands of people’s weekends.

Paul Oakenfold, Lisa Lashes and friends at Shoom, London, 1988. Photograph: Swindells/PYMCA/Rex/Shutterstock

Moreover, partly because acid house also found room for other substances (the joint to ease the ecstasy comedown, the cocaine to give the experience that extra edge, the LSD that would give nights out a different kind of trippiness), these too have become part of everyday life, to the point that talking about a self-contained “drug culture” seems absurd. In both sociological and geographical terms, illegal intoxicants are everywhere, and the contexts in which people take them are as varied as Britain itself.

As much as consumerism, or the resentments that flared up around Brexit, this is part of who we are now. Walk around the nightlife district of any British city and you will soon see – and hear – the telltale teeth grinding and self-fixated chatter of recreational cocaine users. At any number of summer festivals, the mood in the small hours and beyond is usually defined by great crowds of saucer-eyed millennials, high on ecstasy and frantically looking for their next source of excitement. More mundanely, marijuana smoke routinely wafts through the British air, sometimes in the most unlikely places: I got my last faceful earlier this week, while waiting for my local branch of Argos to open on a drizzly Tuesday morning.

And then comes the more malign end of it all. In any sizeable British settlement, if you know what to look for, it is not hard to pick out the local unfortunates craving their next hit of heroin or crack (or both), these days sometimes joined by the pitiful devotees of such synthetic highs as spice, apparently beloved of people with a taste for oblivion and nothing left to lose.

According to figures released last month, 2016 saw the highest-ever recorded number of deaths in England and Wales from what the Office for National Statistics calls “drug poisoning”, which includes everything from paracetamol to amphetamines: 3,744, up from 2,600 20 years ago. Heroin and morphine were involved in 1,209 deaths, which amounted to about a third of the total; cocaine was cited in relation to 371 fatalities. And ecstasy was named on the death certificates of 63 people, up from 57 in 2015, 43 in 2014, 31 in 2013 – and, back in 1993, a mere 12.

Each drug-related statistical category has its own cliches: the hardened heroin addict OD-ing in some grimy hovel, the lifelong cocaine user succumbing to a coronary arrest in her luxury apartment. Within hours of the ONS figures’ release, the fact they suggested people aged 40-49 were now at the greatest risk of drug-related death had spawned another handy trope: that of the “Trainspotting generation”, the fortysomethings who began taking drugs in the slipstream of acid house and never really recovered.

But in the case of deaths caused by ecstasy (or, rather, whatever version of it is sold to the people concerned by unscrupulous dealers), the prevailing stereotype is altogether younger – and in its own way, more heartbreaking.

Go through the archives, and the human stories behind such numbers become painfully clear. An inordinate number of ecstasy deaths involve teenagers and twentysomethings, whose life stories are often illustrated with a portrait taken from social media of some gap-year escapade or college trip. Such people usually seem to have been as innocent as I was when I took my first E. But they are the victims of a grim historical coincidence: the fact that ecstasy use has reached a modern peak (in 2015, its use among young adults was reckoned to have risen by 84% in just two years), just as it has become much stronger. Some modern pills contain as much as 300mg of MDMA, as against the 50mg to 80mg that was the norm when acid house first arrived; in some cases they are also cut with PMA and PMMA, substances that bring on some of the same effects as ecstasy, but are way more dangerous.

Which takes us to the question posed by our presumptive next king-but-one, and the stupid tangle of legal and cultural conventions that get in the way of recognising what is happening, and doing something about it. Just as hardened heroin addicts are often killed by dealers who play fast and loose with their supply, so it is with scores of young ecstasy users. In other words, for as long we allow our young people to ingest chemicals cooked up in bathtubs by career criminals, with no means of checking what on earth they are about to swallow, tragedies will happen.

I am as unsure about the sweeping legalisation of drugs as Prince William appeared to be. God knows how you liberalise the supply of crack; the idea of powerful hallucinogens available from your local off-licence seems problematic to say the least. But the idea of decriminalising at least the possession of most drugs seems increasingly unanswerable – and in time, it is not hard to envisage the liberalisation of cannabis pioneered by a handful of US states extending to Britain, as well as ecstasy being legalised, made subject to official standards, and freely bought and sold.

In the meantime, given that pills and powders can be quickly tested for their constituent ingredients so as to allow people to either moderate their intake or not bother at all (a practice pioneered in Switzerland and the Netherlands, and trialled at a handful of British events this summer), such arrangements ought to be an in-built part of clubs and festivals, if not the average high street. The idea that we can somehow convince young people to chant “just say no” and stick to the questionable pleasures of booze is laughable. No one, surely, is going to be able to roll back a social transformation that dates back to the era of Margaret Thatcher. The point needs to be made as repetitively as all those beats: if acid house changed everything, it is surely time for the official attitude to drugs and the people who take them to be pushed towards their own belated revolution.

• John Harris is a Guardian columnist