On the face of it, it is a familiar enough story. “By its end, much of the optimism of the 20th century had faded. Towns and cities … that used to produce steel, glass, furniture or shoes, and that are fondly remembered by people in their 70s as having been great places to grow up, had been gutted, their factories closed and shops boarded up.”

The words are taken from a remarkable new book by the academics Anne Case and Angus Deaton. On the face of it, this passage perhaps suggests one of those cliched accounts of what sits behind the growth of populism. But rather than framing the condition we now know as “left behind” as a matter of political preferences, the cheerily titled Deaths of Despair and the Future of Capitalism sees it in terms of stark matters such as illness, addiction and rising mortality rates among white Americans aged between 45 and 54, which the book links to “suicides, drug overdoses and alcoholic liver disease”. Between 2014 and 2017, these factors contributed to the first decline in average American life expectancy since records began in 1933.

The Institute For Fiscal Studies has found that deaths of despair have been rising among middle-aged people

Obviously, this is not to deny the existence of deeper and more long-term inequalities. As the authors plainly put it: “Whites without a college degree are not the poorest group in the US; they are much less likely to be poor than African Americans.” It remains the case that African Americans tend to die younger than white Americans. Self-evidently, “deaths of despair” are a problem for people of colour too (Hispanic people, interestingly, are much poorer on average than non-Hispanic whites, but also have lower mortality rates).

But the crucial shift of the last quarter century is plain. In whiter parts of the demographic map, there has been a grim process of catchup – as deindustrialisation, stagnating wages and family breakdowns have done their work in communities that once had at least a measure of security and stability. In an economy that is increasingly divided between winners and losers, if the key determinant of success or failure is whether you are a graduate or not, millions will inevitably lose out. “It’s not black culture, it’s not white working-class culture,” Case said recently. “We really think if you treat people shabby [sic] enough, for long enough, bad things happen to them.”

As Case and Deaton see it, large swaths of their story are uniquely American, and bound up with the endless shortcomings of private medicine and the impact of the opioid epidemic. In a recent UK interview, Case acknowledged that something is also happening in Britain, albeit on the level of a “more muted crisis”. But as I read their book and instinctively projected its findings to parts of the UK I have reported from over the past 10 years, this did not quite ring true. There is a similar story to be told about this country, with its own set of specifically British features: it is partly about the kind of deaths of despair that have risen so dramatically in the US, but also about the mess of things that swirl around them.

Last year, as it launched a five-year review of British inequality chaired by Deaton (who holds both British and US citizenship), the Institute for Fiscal Studies acknowledged that in England, stereotypical deaths of despair have been rising among middle-aged people. Rates of alcohol-specific deaths in the UK have increased by 13% since 2001. People who live in more deprived areas are up to six times more likely to die from alcohol-related liver disease than those in wealthier places. In Scotland there is an ongoing and overlooked crisis surrounding drug-related deaths, which have followed an upward trend for two decades, and now mean the country has the highest per-capita rate of drug fatalities in Europe. There has been a huge rise in the use of non-prescribed benzodiazepines, “street valium”. At the last count, two-thirds of the country’s drug deaths were of people aged between 35 and 54.

I do not like the “left behind” tag, or the picture that it tends to paint of helpless places full of nothing but poverty and anomie. But there are undoubtedly large areas of the country where infirmity and illness – both physical and mental – have become ingrained in the culture, and a decade of cuts and continuing neglect has made things even worse. It is a story I have heard time and again, a self-fulfilling prophecy: having lost their jobs, steelworkers, dockers and miners were routinely put on incapacity benefit and, over time, what was colloquially known as being “on the sick” became not just a signifier for how people had been removed from the job market but something that ran through the ensuing decades.

Now, while the most blighted parts of our cities continue to be bywords for poverty and inequality, less urban places are full of mobility scooters, long queues at the local chemist and the idea that the comfort of chemical sedation may only be a phone call away.

The relevant statistics only underline the point. Ten days ago the Institute of Health Equity at University College London triggered a slew of news stories when it published the latest research by Prof Michael Marmot. As if to underline parallels between the US and UK, Marmot’s report pointed out that growth in average life expectancy stalled in 2010, and that among women in the poorest areas of England it actually fell between 2010 and 2018. Marmot’s work highlights the clear links between deprivation and early death, and the fact that the time people spend in poor health has gone up across England since 2010.

Number of people in poverty in working families hits record high Read more

On the political right, the answers supposedly lie in Boris Johnson’s ideas about “levelling up”, which so far seem to represent a crass underestimate of the difficulties so many places face, not to mention the role Conservative governments have played in their current predicament. Among elements of the liberal left, meanwhile, things often feel conflicted. Grinding social problems are too often understood only in the abstract, as matters of statistical gaps, or our old friend the north-south divide. In England, as with the US, there is another problem: the sense that, in some people’s eyes, the association of whole chunks of the country with support for the populist right has put them beyond the pale.

In the midst of our polarised politics, it is worth remembering that the most representative face of these people and places is not some verbose racist, discovered by a team of television researchers and presented to the viewing public as an authentic spokesperson. It is likely to be someone much quieter, who is well aware of shrinking horizons and vanishing prospects, and looking with uncertainty into a future of dependency, ill health and an early death. This, it seems to me, is the social crisis of our times.

• John Harris is a Guardian columnist