Jane Ballantyne was, at one time, a true believer. The British-born doctor, who trained as an anaesthetist on the NHS before her appointment to head the pain department at Harvard and its associated hospital, drank up the promise of opioid painkillers – drugs such as morphine and methadone – in the late 1990s. Ballantyne listened to the evangelists among her colleagues who painted the drugs as magic bullets against the scourge of chronic pain blighting millions of American lives. Doctors such as Russell Portenoy at the Memorial Sloan Kettering Cancer Center in New York saw how effective morphine was in easing the pain of dying cancer patients thanks to the hospice movement that came out of the UK in the 1970s.

Why, the new thinking went, could the same opioids not be made to work for people grappling with the physical and mental toll of debilitating pain from arthritis, wrecked knees and bodies worn out by physically demanding jobs? As Portenoy saw it, opiates were effective painkillers through most of recorded history and it was only outdated fears about addiction that prevented the drugs still playing that role.

Opioids were languishing from the legacy of an earlier epidemic that prompted President Theodore Roosevelt to appoint the US’s first opium commissioner, Dr Hamilton Wright, in 1908. Portenoy wanted to liberate them from this taint. Wright described Americans as “the greatest drug fiends in the world”, and opium and morphine as a “national curse”. After that the medical profession treated opioid pain relief with what Portenoy and his colleagues regarded as unwarranted fear, stigmatising a valuable medicine.

These new evangelists painted a picture of a nation awash in chronic pain that could be relieved if only the medical profession would overcome its prejudices. They constructed a web of claims they said were rooted in science to back their case, including an assertion that the risk of addiction from narcotic painkillers was “less than 1%” and that dosages could be increased without limit until the pain was overcome. But the evidence was, at best, thin and in time would not stand up to detailed scrutiny. One theory, promoted by Dr David Haddox, was that patients genuinely experiencing pain could not become addicted to opioids because the pain neutralised the euphoria caused by the narcotic. He said that what looked to prescribing doctors like a patient hooked on the drug was “pseudo-addiction”.

Portenoy toured the country, describing opioids as a gift from nature and promoting access to narcotics as a moral argument. Being pain-free was a human right, he said. In 1993, he told the New York Times of a “growing literature showing that these drugs can be used for a long time, with few side-effects, and that addiction and abuse are not a problem”.

Long after the epidemic took hold, and the death toll rose into the hundreds of thousands in the US, Portenoy admitted that there was little basis for this claim and that he had been more interested in changing attitudes to opioids among doctors than in scientific rigour.

“In essence, this was education to destigmatise and because the primary goal was to destigmatise, we often left evidence behind,” he admitted years later as the scale of the epidemic unfolded.

Likewise, Haddox’s theory of pseudo-addiction was based on the study of a single cancer patient. At the time, though, the new thinking was a liberation for primary care doctors frustrated at the limited help they could offer patients begging to get a few hours’ sleep. Ballantyne was as enthusiastic as anyone and began teaching the gospel of pain relief at Harvard, and embracing opioids to treat her patients.

“Our message was a message of hope,” she said. “We were teaching that we shouldn’t withhold opiates from people suffering from chronic pain and that the risks of addiction were pretty low because that was the teaching we’d received.”

But then Ballantyne began to see signs in her patients that experience wasn’t matching theory. Doctors were told they could repeatedly ratchet up the dosage of narcotics and switch to a new and powerful drug, OxyContin, without endangering the patient, because the pain, in effect, cancelled out the risk of addiction. To her dismay, Ballantyne saw that many of her patients were not better off when taking the drugs and were showing signs of dependence.

Among those patients on high doses over months and years, Ballantyne heard from one after another that the more drugs they took, the worse their pain became. But if they tried to stop or cut back on the pills, their pain also worsened. They were trapped.

“You had never seen people in such agony as these people on high doses of opiates,” she told me. “And we thought it’s not just because of the underlying pain; it’s to do with the medication.”

As Ballantyne listened to relatives of her patients talk about how much the drugs had changed their loved ones, her misgivings grew. Husbands spoke of wives as if a part of them were lost. Mothers complained that children had become sullen and distant, their judgment gone, their personality warped, their character altered. None of this should have been happening. Pain relief was supposed to free the patients, not imprison them. It was all very far from the promise of a magic bullet.

As the evidence that opioids were not delivering as promised piled up, the Harvard specialist began to record her findings. By then, though, there were other powerful forces with a big financial stake in the wider prescribing of painkilling drugs. Pharmaceutical companies are not slow to spot an opportunity and the push for wider prescribing of opioids had not gone unnoticed by the drug-makers, including the manufacturer of OxyContin, Purdue Pharma, which rapidly came to play a central role in the epidemic.

As the influence of the opioid evangelists grew, and restraints on prescribing loosened, the pharmaceutical industry moved to the fore with a push to make opioids the default treatment for pain, and to take advantage of the huge profits to be made from mass prescribing of a drug that was cheap to produce.

Bottles of painkiller OxyContin, made by Purdue Pharma. Photograph: Reuters

The American Pain Society, a body partially funded by pharmaceutical companies, was pushing the concept of pain as the “fifth vital sign”, alongside other measures of health such as heart rate and blood pressure. “Vital signs are taken seriously,” said its president, James Campbell, in a 1996 speech to the society. “If pain were assessed with the same zeal as other vital signs are, it would have a much better chance of being treated properly. We need to train doctors and nurses to treat pain as a vital sign.”

The APS wanted the practice of checking pain as a vital sign as a matter of routine adopted in American hospitals. The key was to win over the Joint Commission for Accreditation of Healthcare Organizations, which certifies about 20,000 hospitals and clinics in the US. Its stamp of approval is the gateway for medical facilities to tap into the huge pot of federal money paying for healthcare for older, disabled and poor people. Hospitals are careful not to get on the wrong side of the joint commission’s “best practices” or to fail its regular performance reviews.

In response to what it called “the national outcry about the widespread problem of under-treatment” – an outcry in good part generated by drug manufacturers – the commission issued new standards for pain care in 2001. Hospital administrators picked over the document to ensure they understood exactly what was required.

Every patient was to be asked about their pain levels, no matter what the reason they were seeing a doctor. Hospitals adopted a system of colour-coded smiley faces, to represent a rising scale of pain from 0-10. The commission ruled that anybody identifying as a five – a yellow neutral face described as “very distressing” – or above was to be was to be referred for a pain consultation.

The commission told hospitals they would be expected to meet the new standards for pain management at their next accreditation survey. Purdue Pharma was ready. The company offered to distribute materials to educate doctors in pain management for free. This amounted to exclusive rights to indoctrinate medical staff. A training video asserted that there is “no evidence that addiction is a significant issue when persons are given opioids for pain control”, and claimed that some clinicians had “inaccurate and exaggerated concerns about addiction, tolerance and risk of death”. Neither claim was true.

Some doctors questioned the value of patient self-assessment, but the commission’s regulations soon came to be viewed as a rigid standard. In time, pain as the fifth vital sign worked its way into hospital culture. New generations of nurses, steeped in the opioid orthodoxy, sometimes came to see pain as more important than other health indicators.

Dr Roger Chou, a pain specialist at Oregon Health and Science University who has made long-term studies of the effectiveness of opioid painkillers and helped shape the Centers for Disease Control and Prevention’s policy on the epidemic, said the focus on pain caused patients to give it greater weight than made sense.

“When you start asking people: ‘How much pain are you having?’ every time they come into the hospital, then people start thinking: ‘Well, maybe I shouldn’t be having this little ache I’ve been having. Maybe there’s something wrong.’ You’re medicalising what’s a normal part of life,” he said.

One consequence was that people with relatively minor pain were increasingly directed toward medicinal treatment while consideration of safer or more effective alternatives, such as physiotherapy, were marginalised. Another, said Chou, was the increased expectation that pain can be eliminated. Chasing the lowest score on the pain chart often came at the expense of quality of life as opioid doses increased. “It’s better to have a little bit of pain and be functional than to have no pain and be completely unfunctional,” said Chou.

Health insurance companies piled yet more pressure on doctors to follow the path of least resistance. This meant cutting consultation times and payments for more costly forms of pain treatment in favour of the direct approach: drugs.

The joint commission needed a way to judge whether its 2001 edict on pain was being adhered to and latched on to patient satisfaction surveys. It took a determined doctor to resist the pressure to prescribe. Physicians could spend half an hour pressing a person to take more responsibility for their own health – eat better, exercise more, drink less, find ways to deal with stress – only to watch an unhappy patient make their views known on the satisfaction survey and face a dressing down from hospital management. Or they could quickly do what the patient came in for: give them a pill and get full marks.

In Detroit, Dr Charles Lucas’s three decades of experience as a surgeon told him it was possible to do what was easy and sign the prescription, or to do what was hard. Lucas grew up in the city and had been instrumental in establishing Detroit’s publicly owned hospital as the highest-level trauma centre in Michigan and one of the first top-tier centres in the country.

Activists in New York, during a protest denouncing the city’s ‘inadequate and wrongheaded response’ to the opioid overdose crisis. Photograph: Getty

Emergency departments became beacons for the opioid dependent, who quickly learned to game the system to get drugs on top of their prescriptions. They turned up feigning pain, knowing harassed medical staff under pressure of time and the commission’s standards were likely to prescribe narcotics and move on without too many questions.

“Some of the old-time nurses, they have that jaundiced look in their eye and say ‘So-and-so’s complaining of pain’. You can tell by the look in their eye that they don’t think it’s justified that they get any more medicine,” said Lucas. “The younger nurses, they say we have to treat this pain – because they’ve been indoctrinated – they’ve got to get rid of the pain. God forbid you don’t get rid of the pain. That would be like a mortal sin.”

But there was a price for resisting the pressure to prescribe ever higher doses of pain relief.

Lucas was knocked back in surprise, and then infuriated, to be summoned to appear before his hospital’s ethics committee after a nurse reported him for failing to provide adequate pain treatment.

The surgeon’s longstanding patients included Gail Purton, the wife of a well-known Michigan radio personality. Lucas operated on Purton a few times, and she was back for surgery after her ovarian cancer spread. “It was a big operation. Cut off all sorts of cancer.” The next day, a nurse asked Purton if she was in pain. Purton said she was. The nurse reported Lucas for failing to properly address a patient’s pain. “I got reported because I wasn’t giving her enough pain medicine. She had a big cut from here to here,” Lucas said, running his finger across the front of his shirt and scoffing at the idea that she could be pain-free after an operation like that.

The surgeon responded with a five-page letter to the ethics committee chairman, whom he happened to have trained, challenging the questioning of his professional judgment. Purton wrote her own letter, praising Lucas’s care and saying that she never expected not to have pain after a major operation.

The case was dropped, but it was not an isolated incident. Lucas has worked closely with another surgeon, Anna Ledgerwood, since 1972. She too was hauled before the ethics committee on more than one occasion, on the same charge. It cleared Ledgerwood, but Lucas said more junior surgeons buckled to the pressure to administer opioids just to stay out of trouble.

Lucas regarded the new pain orthodoxy as a growing tyranny. He also thought it was killing patients. He began to collect his own data.

As the joint commission was pushing out its new standards for pain treatment in the early 2000s, the industry was driving a parallel effort to influence the prescribing habits of doctors in small clinics and private practices across the country. Many were still hesitant to prescribe narcotics, in part because of fear of legal liability for overdose or addiction.

The American Pain Society and Haddox, who was by then working for Purdue Pharma, were instrumental in writing a policy document reassuring doctors they would not face disciplinary action for prescribing narcotics, even in large quantities. The industry latched on to the Federation of State Medical Boards because of its influence over the health policy of individual US states which regulate how doctors practise medicine.

In 2001, Purdue Pharma funded the distribution of new pain treatment guidelines drawn up by the FSMB that sounded many of the same themes as the standards written by the joint commission.

The document picked up on Haddox’s pseudo-addiction theory. “Physicians should recognise that tolerance and physical dependence are normal consequences of sustained use of opioid analgesics and are not synonymous with addiction,” it said.

The FSMB pressed state medical boards to adopt the guidelines and to reassure doctors that adhering to them would diminish the likelihood of disciplinary action.

Over the following decade, the FSMB took close to $2m (£1.52m) from the drug industry, which mostly went to promote the guidelines and to finance a book, Responsible Opioid Prescribing, written with the oversight and advice of a clutch of doctors who were strong advocates of wider use of prescription narcotics. The book was sold to state medical boards and health departments for distribution to physicians, clinics and hospitals. The drug industry paid for the publication but the FSMB kept the $270,000 profits from sales.

Within a few years, the model guidelines were adopted in full or in part by 35 states, and the floodgates were open to mass prescribing of what Drug Enforcement Administration agents came to call “heroin in a pill”. Opioids were soon the default treatment even for relatively minor pain. Dentists gave them to teenagers after pulling their wisdom teeth. Not just one or two days’ worth of pills, but a fortnight or a month’s worth, which, if they did not draw the intended recipient in, frequently sat in the medicine cabinet waiting to be discovered by someone else in the family. The lack of caution in prescribing left an impression among the users that the drugs were harmless, and some people shared them with others as easily as they might an aspirin. Prescribing escalated year on year. So did profits. OxyContin sales passed $1bn a year in 2000. Three years later they were twice that. Other opioid makers were pulling in huge profits too.

By the time the FSMB guidelines were landing in doctors’ inboxes in the early 2000s, Ballantyne had reached her own conclusions about the impact of escalating opioid prescribing. In 2003, she co-authored an article in the New England Journal of Medicine highlighting the dearth of comprehensive trials and saying that two important questions remained unanswered even as mass prescribing of opioids took off. Do they work long term? Are higher doses safe to take year after year? The drug industry and opioid evangelists said yes, but where was the evidence for it?

Ballantyne wrote that there was evidence that putting some patients on serial prescriptions of strong opioids has the opposite of the intended effect. High doses not only build up a tolerance to the drug, but cause increased sensitivity to pain. The drugs were defeating themselves.

Her assessment seemed to warn that if there was an epidemic of pain, it was partly driven by the cure. On top of that, there was evidence that the drugs were toxic. Then came the conclusion that stuck a dagger into the heart of the campaign for wider opioid prescribing. “Whereas it was previously thought that unlimited dose escalation was at least safe, evidence now suggests that prolonged, high-dose opioid therapy may be neither safe nor effective,” she wrote.

Ballantyne was also increasingly aware that the claim that pain neutralised the risk of addiction was false. Quantifying addiction, and who may be vulnerable, is notoriously difficult. Ballantyne, like a lot of doctors, estimated that between 10 and 15% of the population is vulnerable, but that it depends on the substance and circumstances. What she was certain of was that Purdue’s high-strength pill, OxyContin, had been a game changer. “The long-acting opiates suddenly put much higher doses into people’s hands and much more of it, and taking it around the clock made them dependent on it.”

From her research, Ballantyne concluded that OxyContin supercharged what was already widespread dependence on weaker opioid pills by drawing a new group of people into the category at risk of addiction and death. The danger was compounded by OxyContin’s failure to live up to its promise of holding pain at bay for 12 hours. For some patients, it wore off after eight, causing them to take three pills a day instead of two, greatly increasing their overall dose of narcotic and with it the risk of addiction.

Ballantyne thought the article would at least cause her profession and the drug industry to take stock of the impact of mass prescribing. By the time the article appeared, the documented death toll from prescription opioids was running at around 8,000 a year.

“When the 2003 New England journal article came out, I thought it was going to make the medical community sit up and say: ‘Wow. These drugs that we’ve been thinking are helping people are not. We have a real problem.’ But the medical community didn’t at all say: ‘Wow,’” Ballantyne said with half a laugh, 15 years later.

“People in my field who had been, like me, taught we have to do this – people who’d been lobbying to try and increase opiate use, like the palliative care physicians – said: ‘What are you doing? We worked so hard to get to this point, and now you’re going to turn it all around. They become so rattled when you suggest you shouldn’t give the opiates – it’s partly people in the pain field and especially people in pharma – because it’s big business.”

Lucas and Ledgerwood had their own study on the impact of opioids in the works. They came to believe the tyranny of the colour-coded smiley faces was costing lives. Years of surgery have given Lucas a healthy respect for pain as a tool for recovery. To suppress it was dangerous. But as large doses of opioids became the norm, the surgeon noted an increasing number of incidents of patients struggling to breathe after routine operations and being moved to intensive care.

Lucas and Ledgerwood visited trauma centres to collect data on deaths before and after the joint commission standards on pain treatment. In 2007, the two doctors published their findings. Before the commission’s dictum, 0.7% of trauma centre patients died from “excess administration of pain medicines”. The death toll rose to 3.6% after the commission’s policies kicked in.

“In each case, administration of sedation led to a change in vital signs or a deterioration in the respiratory status requiring some type of intervention which, in turn, led to a cascade of events resulting in death,” the paper said. Those were only the deaths in which there was little doubt opioids were responsible, and the real toll was almost certainly higher. “Overmedication with sedatives/narcotics … clearly contributed to deaths,” the study concluded.

A memorial in Washington DC, consisting of 22,000 engraved white pills representing the face of someone lost to a prescription opioid overdose in 2015. Photograph: Mark Wilson/Getty

“I’m convinced that because of the pressures brought to bear by the joint commission, we are killing people,” Lucas told me. The study said the medical staff lived in fear of the joint commission standards which created “great psychological pressure on caregivers” to use narcotics.

In a damning critique, the paper said that the commission’s reliance on pain scales to guide treatment had created an “excessive emphasis on undermedication at the same time ignoring overmedication”. The obsession with ensuring people were not in pain came at the expense of ignoring the dangers of giving large amounts of opioids to people recovering from surgery or serious injury. The drugs may kill the pain but they also risked killing the patient.

The two doctors made no secret of who they blamed for “this preventable cause of death and disability”. “It’s about money. Money has influence, and it influenced the joint commission,” said Lucas.

The surgeon presented the paper to a meeting of the Central Surgical Association and saw it published by the Journal of the American College of Surgeons under the headline “Kindness Kills: The Negative Impact of Pain as the Fifth Vital Sign.”

Afterwards, Lucas got a stream of letters and emails from doctors who recognised the problem. But, unlike Ballantyne, he wasn’t surprised when the policy remained the same. “Did I expect a change? No. It is too ingrained into the medical profession. It’s become financial just like the drug industry is financial. It’s nothing to do with right or wrong. It’s about how the money flows,” he said. “When you write a paper you want there to be unemotional data out there. You want that unemotional data to be analysed and interpreted in one way or the other, but you don’t expect the Renaissance.”

In 2012, nine years after Ballantyne’s cautioning against the mass prescribing of opioids as a quick fix for pain was published in the New England Journal of Medicine, a renowned British pain specialist, Cathy Stannard, called the doctor’s paper “a distant warning bell”, challenging the opening of the floodgates to strong opioids.

Ballantyne continued to collect data and publish ever more detailed insights into the impact of painkillers. A less rapacious drug industry might have paused in its headlong charge to sell opioids, and less blinkered and compliant regulators might have determined that this was the moment to weigh the claims made in favour of permitting such widespread prescribing.

Instead the pharmaceutical companies took the warnings as a challenge to their business interests. Through the 2000s, industry poured money into a political strategy to keep the drugs flowing. It funded front groups and studies to claim that there was indeed an epidemic – but it was of untreated pain. The millions coping with chronic pain were the real victims, the industry said, not the “abusers” hooked on opioids they often bought on the black market or obtained from crooked doctors. That one frequently became the other was conveniently overlooked.

Pharma’s lobbyists worked to persuade Congress and the regulators that to curb opioid prescribing would be to punish the real victims because of the sins of the “abusers”, and it worked. As a result, the devastation ran unchecked for another decade and more. By 2010, doctors in the US were writing more than 200m opioid prescriptions a year. As the prescribing rose, so did the death toll. Last year, more than 72,000 Americans died of drug overdoses, the vast majority from opioids, nearly 10 times the number at the time Ballantyne published her warning.

The head of the FDA at the time OxyContin was approved for distribution two decades ago, Dr David Kessler, later described the opioid crisis as an “epidemic we failed to foresee”. “It has proved to be one of the biggest mistakes in modern medicine,” he said.

Kessler was wrong. It wasn’t a mistake. It was a betrayal.

This is an edited extract from American Overdose by Chris McGreal, published by Guardian Faber on 15 November (£12.99), and published by PublicAffairs Books in the US on 13 November. To order a copy for £9.99, go to guardianbookshop.com or call 0330 333 6846. Free UK P&P over £10, online orders only. Phone orders min P&P of £1.99.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.