The Observer’s groundbreaking survey on recreational drug use shows it spans all social classes and is as much a part of modern life as food banks and farmers’ markets

Back in November 2008 when the Observer conducted its last survey on drug use in Britain, the country and the world around had just entered a new phase that no one quite understood. Lehman Brothers had recently collapsed and suddenly the c-word that was the second-most associated with the City was crash, not cocaine. Yet not even those in the grip of a truly paranoia-inducing marijuana high could fully conceive of the years of depression and austerity that were to follow.

So in 2014, with all the economic and geopolitical problems that beset us, it might seem trivial or somehow decadent to look too closely at drug consumption and attitudes. But that would be an erroneous conclusion for two reasons.

First, if we’re to use death as a metric for significance, in recent years 10,000 violent deaths a year can be attributed to the drugs trade in Mexico alone.

Second, it’s also a matter of life. For, like it or not, illicit and/or recreational drugs are part of how we live now. That doesn’t mean that everyone takes them, although almost a third of us – about 15 million – have done so (31%, up from 27% six years ago). But it does mean that everyone is affected by them.

Given the percentages, it follows that everyone knows someone who has taken drugs. And the chances are that most of us know someone who acknowledges that they’ve had problems with drugs. In addition, it’s not unlikely that we’ve fallen victim to a crime committed by an addict funding his or her habit.

The point is that drugs cannot be safely categorised as a niche activity or a passing fad. They are as much part of modern life as food banks or farmers’ markets. They span all social classes and, although disproportionately concentrated among the under-35s, all ages.

Facebook Twitter Pinterest Deb Greene, the first customer at the Cannabis City marijuana shop in Seattle, Washington in July, after the state legalised marijuana for recreational use. Photograph: Pool/Elaine Thompson/AP

That said, most drug taking is conducted over a relatively short period of time. It’s rare you find life-long drug users in the way, for instance, that you find life-long drinkers and smokers. The profile of an average drug user is someone who starts at 19, and the average stopping age is 26. For most of these twentysomething slackers, their drug experimentation will be limited to smoking marijuana. And the majority of it is conducted in the home.

However, that’s only part of the story. The exceptions to the rule are so varied and widespread they can hardly be described as exceptional. The use of drugs such as cocaine and ecstasy remains high: each has been tried by around a third of drug users. Now a new generation of drugs is increasingly available, especially in London. The so-called “legal highs”, or, as one expert called them, “novel psychoactive substances”, haven’t made a big impression on the survey yet, but all indications are that we will see more synthetic drugs, which seek to mimic the effects of established drugs while circumventing the law.

And there are signs that these drugs suffer little of the stigma that dog, say, crack cocaine and heroin. According to Dr Owen Bowden-Jones, a consultant psychiatrist working in the drug field, many who experience problems with club drugs are reluctant to consult conventional drug clinics because they associate them with crack and heroin addicts. Of current drug-users, 40% come from social grades AB – ie the wealthiest bands.

In this respect the most revealing drug story of the last year was that of the Rev Paul Flowers, the 64-year-old Methodist minister, former Labour councillor and one-time chairman of the Co-operative Bank. It would be hard to invent someone with a more established position in society than a man who is prominent in the church, politics and banking. And, to be sure, it didn’t look good when the so-called Crystal Methodist was exposed as a regular user of cocaine, methamphetamine and ketamine. Especially considering that the bank discovered a £1.5bn hole in its finances during Flowers’s chairmanship.

Paul Flowers, who was chairman of Co-operative Bank. Photograph: Photograph: Lynne Cameron/PA

Not even the most gung-ho advocate of drugs would dare argue that a cocktail of cocaine and methamphetamine would be likely to sharpen one’s grasp of complex financial transactions and investments. But, leaving aside whether or not the drugs affected Flowers’s professional judgment, the striking detail was that it wasn’t that shocking that he was found to have extensive experience of class A and B drugs. Newsworthy, yes, but not beyond contemplation.

Similarly, the reluctant admission last year by Toronto mayor Rob Ford that he had smoked crack cocaine also laid dramatic claim to the headlines. But again, it was a media splash, not a moral earthquake. Both men claimed they acted under pressure, which may be true, but it’s also part of the narrative of redemption and recovery. Drug use in this way is a symptom of deeper social and psychological problems. Again, that is often the case. But not always. Drug use is sometimes about not a great deal more than using drugs. Whatever their causes and motivations, these two tales confirmed that drug use or abuse is not the preserve of the young or the powerless. By their own estimation, most people “use” rather than “abuse” drugs. The survey shows, in a representative analysis of everyone who has taken drugs, that only 13% thought that they had ever had a drug problem. In terms of actual numbers, that amounts to a sizeable figure – somewhere around 2 million Britons consider that at some point in their lives they have had a dysfunctional relationship with drugs. And about half of them still do. What constitutes a problem will, of course, vary from individual to individual, perhaps from outright dependency down to minor health concerns.

The government determines what is problematic in terms of economic impact. Home Office research suggests that between a third and a half of all acquisitive crime is committed by offenders who use heroin, cocaine or crack cocaine. The last time the cost of that crime was added up was 10 years ago and the figure the government arrived at was £13.9bn. The total economic and social cost was estimated to be a staggering £15.4bn. Although there are no exact figures, over the last 15 to 20 years drug-associated crime has been decreasing, and there have been many theories put forward to explain the drop. For those who argue that the correct means of dealing with illegal drugs is the criminal justice system, then the decline in class A drug-related crime indicates that prohibition and prison work.

As journalist Ross Clark put it, writing in the Times earlier this year: “We can only guess how many would have experimented with heroin and crack had they been legal: 1%? 10%? But to judge by the upwards jolt in crime that accompanied an epidemic [of heroin and crack in the 1990s] involving 0.5% of the population, Britain would have been a bloodbath.”

Elsewhere – particularly in America, where a similar drop has occurred – it’s been powerfully argued that the decrease in heroin and crack use has less to do with prison than with a community reaction against the devastation these drugs can wreak. Advocates of decriminalisation, such as film-maker Eugene Jarecki, maintain that it’s health departments, not crime departments, that should be dealing with drug problems. The so-called “war on drugs” has cost America $1trn since 1971 and has resulted in 45 million arrests, and a 12-fold increase in the number of offenders imprisoned on drug charges. It’s a much more draconian response to a much larger problem than exists in Britain (where the prison population doubled between 1993 and 2012).

As a consequence, the drugs debate is more urgent in America. One aspect of this is that the general drama of drugs plays a more pronounced role in American culture. It’s no coincidence that two of the most impressive American TV series of recent years have focused on crack cocaine (The Wire) and crystal methamphetamine (Breaking Bad). In both cases it’s worthy of note that the previously irredeemable archetype of the street dealer emerges (several in The Wire, Jesse in Breaking Bad) as a sympathetic figure.

Bryan Cranston and Aaron Paul playing drug producers Walter White and Jesse Pinkman in the TV series Breaking Bad Photograph: Photograph: Frank Ockenfels/AMC

Another, more concrete, development is that, in the non-fiction world, two US states, Washington and Colorado, have decriminalised marijuana. Our survey shows that Britons are divided over the idea of instituting a similar scheme in this country, with just over half supporting it. But the divisions remain wide: 27% of the population thought our drug laws are too liberal, and exactly the same number thought that they’re not liberal enough – whereas 46% think they’re just right.

The underlying trend is towards greater liberalisation (only 18% felt the laws needed to be more liberal in 2008), but as things stand there’s a balance, with more younger people wanting reform and more older people wanting stricter controls, and the half in the middle content with what we’ve got. Our survey shows 33% of those aged between 16 and 24 said drug laws were not liberal enough, and 40% of those aged 65 and over said drug laws were too liberal.

To some extent, drug policy in this country seeks to cater to all those groups – on the one hand talking tough; on the other often turning a blind eye, and the main body of policing targeting “problem” drug users. It’s not a picture that suggests radical change is afoot.

Yet it’s striking that as many as 84% of Britons don’t believe that the “war on drugs” can ever be won. If that’s the case, you might think, then why is there not more support for policies that reject this failed strategy? The answer is probably that “war” is a misleading metaphor for combating drugs. As cultural critic and former prison doctor Theodore Dalrymple has put it: “Saying the war against drugs is unwinnable is like saying the war against burglary is unwinnable and we should open our doors. Absurd. War is the wrong word.”

As with so many other areas of life, the UK has one foot in America and one in Europe. We want both approaches – the harsh and the lenient – simultaneously. It’s made for a strategy that is neither clear nor long-term, but appears to accept the idea that there is not a great deal else to be done.

Certainly there seems little political appetite for the sort of measures brought in by Portugal in 2001. Partly as a response to the country’s high rate of HIV among intravenous drug users, the authorities decriminalised drug possession, as long as the user had no more than 10 days’ supply. The results have been mixed, with some reduction in HIV and crime, but also increases in some hard drug use.

Ultimately, until the major drug-consuming nation (the US) and the major drug distributing and producing countries (Colombia, Mexico, Afghanistan) undergo a major decriminalisation process, then violence and illegality will continue to underpin the drug trade. In other words, no matter what happens in Amsterdam or Lisbon, it’s not going to affect the terrifying number of killings associated with the drug business in Latin America.

Which brings us back to where we started. This survey, when matched against previous surveys, appears to suggest that we are moving as a society towards greater tolerance and understanding of drug use. But it’s a gradual shift and there is no guarantee as to where it will end.

As things stand, the pleasure derived from drugs cannot easily be divorced from the misery they inflict on more than one in 10 users, nor from the savage violence that continues to characterise their production and distribution. One man’s high currently seems to necessitate another’s low.

It would be nice to think that before the next Observer survey on drugs, there will be a coordinated international effort to look at a more rational, healthier and less murderous means of organising the drugs trade. But for the time being that remains what opium smokers used to refer to as a pipe dream.