Stop me if you’ve heard this one before. On 19 December 1984, The New York Times ran a story about parents who feared the risks of routine vaccinations. The parent quoted in the article was a lawyer who blamed vaccines for the death of his daughter. The story was framed as a conflict between parents such as him and medical experts, who pointed out that serious side-effects of vaccines were extremely rare, and that the diseases vaccines prevented were far worse. We have to be careful, the scientists said, or these fears could result in fewer people getting vaccinated and more people getting sick.

On 27 April 1999, The New York Times ran a story about parents who feared the risks of routine vaccinations. The parent quoted in the article was college-educated, an author and professional activist, who blamed vaccines for her son’s brain damage. The story was framed as a conflict between parents such as her and medical experts, who worried that the internet was spreading false information and unwarranted fears. Parents today just haven’t seen the devastation vaccine-preventable diseases can cause, the scientists said.

On 21 March 2008, The New York Times ran a story about parents who feared the risks of routine vaccinations. The article noted that parents who refused vaccines for their children were often ‘well-educated and financially stable’. The story was framed as a conflict between those parents and medical experts, who worried that geographical pockets of vaccine refusal could help spread preventable diseases, such as measles. Parents today just haven’t seen the devastation vaccine-preventable diseases can cause, the scientists said.

For more than 30 years now, we journalists have been telling the same story, with the same actors, playing the same roles, and speaking the same lines. The authors change, but the news doesn’t. It barely even counts as ‘new’.

There are two groups of people you can blame for this pattern of repetitive storytelling. Maybe it’s them: maybe the problem is parents whose anti-science proclivities have carried them so far away from the facts that journalists have no choice but to repeat ourselves ad nauseum. The story doesn’t change because the story hasn’t changed.

That could be true. But there’s also another option. Maybe it’s us: maybe journalists aren’t listening. The story never changes because we stopped looking for the other stories we could tell.

If that’s true, it’s a big deal. And not just for journalists. Vaccination is a deeply important part of public health. Whether to vaccinate or not isn’t simply a decision you make for yourself or your family, independent of the choices of everyone else. Vaccines work in two ways. They decrease your personal risk of contracting a disease, and they reduce the number of potential hosts and carriers in the population. That means the more vaccinated people there are, the harder it is for a disease to spread. Vaccines can stop an outbreak before it happens. This so-called ‘herd immunity’ protects children who are too young to get a vaccine, people who are too sick to get one, and anybody whose vaccination isn’t working as well as it should.

That’s why medical experts really care about vaccination and why they’re worried about vaccine rejection, even though, nationwide in the United States, children are vaccinated at rates of 90 per cent or better for most vaccines. In specific places, and for specific vaccines, uptake can be a lot lower, enough to give diseases a foothold. Measles, for instance, is highly contagious. To prevent its spread, you need at least a 96 per cent vaccination rate. In California, where a measles outbreak last year infected nearly 200 people and spread to 23 other states, the measles vaccination rate is about 92 per cent – and scientists have estimated that regions near where the outbreak began could have rates as low as 50 per cent and certainly no higher than 86 per cent.

Public health experts and the public they serve have very good reason to nose about in our neighbours’ business when it comes to vaccines. This is why the US Centers for Disease Control and Prevention (CDC) collects data on vaccination rates. It’s why scientists have done surveys that show the number of parents with concerns about vaccines increased by 163 per cent between 2000 and 2009. But we still seem to be pretty clueless when it comes to why those people fear vaccines and what could be done to change their minds. Case in point, a paper published in 2014 tested different strategies for improving the likelihood that skeptical parents would vaccinate their kids. None of the tested techniques worked. When the researchers tried debunking vaccine misinformation, they succeeded in convincing more parents that vaccines don’t cause autism. But those same parents were actually more likely to reject vaccines afterwards.

Clearly, something is amiss here, and it matters to all of us.

I’m a journalist. I write, mostly, about science. My two daughters, the first aged two, the baby aged six months, are both getting all their shots according to the schedule recommended by the CDC. I have written about vaccine rejection – stories that follow basically the same narrative as The New York Times pieces I mentioned earlier. I have also written stories bemoaning Americans’ insistence on rejecting scientific fact.

What I hadn’t done, at least until recently, was question whether those stories were accurate. I don’t mean in the sense that I’d published incorrect information. Instead, I’ve come to believe that I haven’t been asking the right people the right questions, and that’s leading me to write stories that are factually correct, but don’t accurately reflect what’s really happening. What if scientists are wrong about the reasons parents don’t vaccinate? What if, as a journalist, I’ve been steering the national conversation in the wrong direction by not questioning the reasons put forth? Herd immunity matters a lot, and there’s good reason to think we aren’t listening well enough to what the herd has to say.

In 2014, I audited a class at Harvard’s Kennedy School of Government. Called ‘Science, Power, and Politics’, it addressed the way humans and our biases affect how science gets done and is used to shape society. The professor, Sheila Jasanoff, assigned a final paper and I decided to take the opportunity to see how those issues came up in journalistic coverage of vaccine controversies.

it was the exact same story – a tale where scientists and experts face off against well-educated, well-off parents

I decided to focus on The New York Times because that publication generally represents what journalists think of as high-quality reporting. (Fair notice: I’ve written for the Times and had a column, Eureka, with the Times Magazine for two years.) I used their archives to find stories about vaccine controversies and to weed out stories that weren’t strictly news, such as opinion pieces and blogs. I ended up with 23 articles, stretching from 1981 to 2014.

I expected to see coverage of vaccine controversies change over time. In particular, there’s a common trope that journalists spent a good chunk of the 1990s and early 2000s giving too much credence to the anecdotes of frightened parents – anecdotes that fuelled public panic and contributed to the growth of vaccine rejection. But I didn’t find that in the Times at all. Instead of tracing why the story changed, I found myself having to figure out why it didn’t.

All told, of the 23 articles I read, 11 told the exact same story – a tale where scientists and experts face off against well-educated, well-off parents. The parents fear vaccines. The scientists say those fears are unwarranted. Wash, rinse, repeat.

What makes the parents afraid, though? This is where things get interesting. In the 1999 story I mentioned before, the parent activist Barbara Loe Fisher, president of the National Vaccine Information Center (NVIC), a non-profit charity based in Virginia, said she wasn’t against vaccination, just mandatory vaccination. She had a problem with the aspect of vaccine law that’s used to increase and maintain herd immunity. That’s because vaccines, like all medicines, do have risks.

For instance, the National Academies of Sciences, Engineering and Medicine in Washington, DC concluded that there is convincing evidence to support the idea that it could be dangerous for people with compromised immune systems to get the vaccine for measles, mumps and rubella (MMR). Those people could end up with encephalitis, an inflammation of the brain that can lead to paralysis, memory problems, even death. There are also a number of potential risks that the National Academies don’t consider proven or disproven yet: there’s just not enough evidence either way. Included in that category are things such as inflammation of the optic nerve, which can lead to blindness.

Such risks are extremely rare. And catching measles comes with risks of its own, including, again, encephalitis. So you don’t avoid risk by avoiding vaccination. You just change what the risk comes from. And, from a public-health standpoint, vaccinating helps reduce the risks to other people, including those who have immune problems and can’t get the vaccine themselves.

It’s this last bit that Fisher had a problem with. Sure, the risks of vaccination were low and the benefits to society were high. But if your kid is the rare person who experiences a severe side effect, the greater good no longer matters. In the story, she is quoted as saying: ‘Parents are saying: “Hey, wait a minute, my child is important, too!”’

Contrast that with how scientists characterised the problem in the same article. ‘We are in danger of becoming a victim of our own success,’ said Louis W Sullivan, the former US Secretary of Health and Human Services. He, and other experts, believed that the problem was that parents were ignorant of the risks of vaccine-preventable diseases.

That is a significant disconnect. If the scientists are right, then this is a question of facts. Parents need to be shown data. They need to see evidence comparing the risks of disease with the risks of vaccination.

‘I refuse to sacrifice my children for the greater good … [but] I cannot deny that my child can put someone else at risk’

But if Fisher is right, then that approach wouldn’t do a damn bit of good. The argument, to her, isn’t about numbers. It’s about philosophy. Vaccines, when used to prevent the spread of disease through herd immunity, are in essence presenting a trade-off. You accept the risk of vaccination, small though it might be, and, in exchange, society as a whole is safer. Kids with cancer are safer. Little babies are safer. Elderly people are safer. But those getting the vaccine are taking a risk.

Fisher didn’t think that was a fair trade. And she’s not the only one. In that 2008 article in The New York Times, Sybil Carlson, the parent of an unvaccinated child who attended a school that had recently had a measles outbreak, said: ‘I refuse to sacrifice my children for the greater good … [but] I cannot deny that my child can put someone else at risk.’

Meanwhile, in the same story, Mark Sawyer, a pediatrician and infectious disease specialist at Rady Children’s Hospital in San Diego, told the paper that: ‘Most of these parents have never seen measles, and don’t realise it could be a bad disease so they turn their concerns to unfounded risks. They do not perceive the risk of the disease but perceive risk of the vaccine.’

Carlson had, actually, seen measles. And she did know she was placing others at risk. But Sawyer’s perspective on what people like her are thinking is also treated as fact in a way that Carlson’s own statements about herself are not. Four paragraphs after Carlson is quoted and just before quoting Sawyer, the article’s author Jennifer Steinhauer states, in the voice of an objective narrator: ‘it is the absence, or close to it, of some illnesses in the United States that keep some parents from opting for the shots’.

Scientists keep insisting that this is a question of parents rejecting science and ignoring facts. But that’s not what the parents are saying. The debate they are broaching is a much more complex one, and one that can’t be solved with the recitation of objective facts. Here is what they are asking: if the preservation of herd immunity is a trade-off, who has the right to make you accept the trade? Why should the government have that power?

Scientists think that they know why people reject vaccines. Journalists assume the scientists are right. And when we accept that single story as the only one that can be told, we end up backing ourselves into a corner – forced to repeatedly engage in a fight that isn’t about what we think it is about. It’s possible that this is why the communication techniques which scientists try to use to reduce vaccine hesitancy don’t work – they’re communicating the wrong information.

If two groups don’t agree on what the argument is about, then public debates become basically futile, no matter how many times you have them, says Maya Goldenberg. She’s an associate professor of philosophy at the University of Guelph in Ontario, where she studies the process of evidence-based medicine and sociopolitical context of scientific research. She has recently begun to study vaccine resistance and efforts to combat it. In a forthcoming paper that will be published in the journal Perspectives on Science, Goldenberg reviewed existing social-science research on vaccine hesitancy and came to the conclusion that the central argument is really a philosophic one.

The public might not know much science, and might be bad at estimating relative risks, Goldenberg writes, but those facts don’t explain vaccine hesitancy. Her research found that many parents’ concerns about vaccines are similar to those of Fisher and Carlson: they can’t quite mesh the goals of widespread, population-scale public health with their personal goals for their individual children. She cites focus groups where experts reassured frightened parents that the MMR vaccine was safe for the general public. Meanwhile, the parents kept asking: ‘Is the MMR vaccine safe for my child?’

In multiple surveys, Goldenberg found parents who were perfectly happy to believe that the MMR vaccine was generally safe, even as they decided they wouldn’t be vaccinating their kid. They worried about their own specific child’s sleep patterns, their child’s family medical history, their child’s recent illnesses – and they found, in those things, reasons why they didn’t believe the MMR vaccine was safe for their child.

‘The conflict isn’t about science,’ Goldenberg says. ‘It’s a conflict about much deeper values, and the science serves as placeholders for arguing out value disputes.’

The same is true of climate change, Goldenberg says, which has become a battlefield on which Americans hash out their differences over how market economies ought to function, how much regulatory power the federal government should have, and what does and does not constitute undue government intrusion in daily life. That debate is not about climate data. It looks that way only if you aren’t paying attention.

Here’s another example, one that isn’t so emotionally fraught to Americans. In June 2013, the anthropologist Andrew Mathews at the University of California, Santa Cruz published a paper in the journal Social Studies of Science that focused on a series of scandals in Mexico related to government tree-planting programmes.

Starting with the election of President Felipe Calderón in 2006, the government of Mexico embarked on a tree-planting palooza, aimed at sequestering carbon and reducing the erosion linked to deforestation. By 2008, they’d already exceeded their goals. Instead of 5 million new trees, the armies of volunteers, corporate sponsors and government workers had planted more than 8 million. President Calderón had been a big figurehead for this work. He planted trees at his residence, he travelled to four different states to plant trees with locals, and he and his son had planted a little sweet-gum tree in one of Mexico City’s most historic public parks.

That last tree ended up at the centre of a political scandal. Just a year after its 2007 planting, journalists from the newspaper Reforma found that the tree had devolved from a lush, bushy plant to a single, mostly barren stick capped with a sad toupee of leaves. The pictures turned a PR dream into a nightmare, leading to widespread criticism of the tree-planting programme, a high-profile debunking of its success, even undermining President Calderón’s political reputation.

Scientists try to fight with facts, assuming the public just doesn’t know what’s going on, while the public gets angrier because it’s not being listened to

All of that seems a little crazy. The sweet-gum tree wasn’t even dead, as Calderón’s government was quick to point out. It just wasn’t growing as well as it maybe could have. They tried to disprove the tree’s demise. They tried to explain the science behind tree growth and why, appearances to the contrary, the sweet-gum tree might be perfectly healthy and go on to lead a long and happy life. None of it worked. That’s because, Mathews discovered, the scandal wasn’t about the tree’s health (or lack thereof). It wasn’t really even about the specific tree-planting programme, though a Greenpeace report later exposed a lot of waste and failure there. The whole thing was really about a long-standing lack of trust in the government, both in general and on the issue of tree-planting in particular. All the way back to the 1930s, Mathews wrote, the Mexican government had used tree-planting as part of authoritarian media performances, often in conjunction with actions against the indigenous people it blamed for the deforestation. Pulling out data to show that this one sweet-gum tree in the park was alive didn’t fix that. In fact, it kind of made things worse.

Goldenberg thinks that this is the same kind of thing we’re seeing with vaccine rejection today. Scientists try to fight with facts, assuming the public just doesn’t know what’s going on. When that doesn’t work, vaccine rejection gets attributed to people being ‘anti-science’. Meanwhile, the public gets angrier because it’s not being listened to. Vaccine hesitancy continues to rise because the public sees experts inundating them with numbers and refusing to answer the real questions.

There are historical reasons to think that vaccine hesitancy might have a lot in common with this tree-planting issue. In 1977, the US launched the Childhood Immunization Initiative. It was an enormous programme aimed at dramatically increasing immunisation rates. This programme created the first laws linking school eligibility to vaccination – essentially making vaccination mandatory. It also created mechanisms for enforcement of those mandates. By 1981, all 50 states had such laws. Prior to the initiative, the trade-off – acceptance of individual vaccine risk in exchange for herd immunity and vast public health benefits – was, at least to some extent, a voluntary one. Afterwards, it was not. This is hardly ever mentioned – by either side – in public discussion about vaccine controversies. But it supports the idea that the backlash against vaccines is being driven by political and philosophical concerns.

As I read through coverage of vaccine controversies, a final story, from The New York Times Magazine on 20 April 2011, stood out. Written by Susan Dominus, it is a profile of Andrew Wakefield, a British former doctor famous for publishing the study that first linked the MMR vaccine to autism in the public consciousness. His paper, published in 1998 in the prestigious journal The Lancet, purported to discover a connection between MMR vaccination and gastrointestinal disorders in autistic children – disorders that Wakefield claimed, in a press conference, might be responsible for autism itself. He advocated breaking up the combined vaccine into three parts, and maybe delaying or skipping it altogether.

This paper was eventually discredited. Not only were Wakefield’s research methods flawed, there was also evidence that he had manipulated data. And he turned out to have an unreported conflict of interest – he had been paid to do the study by trial lawyers who wanted to prove that MMR caused autism as part of a lawsuit. Wakefield lost his medical licence, but he went on to find fame working with parents who believed their children had been damaged by vaccines. As Dominus documented in her story, his success in that arena seems to be due, at least partly, to his willingness to listen to parents and spend large amounts of time talking and thinking about the problems they notice in their children and what they think might be causing those problems.

That is a big part of what made this article so interesting. Dominus explored how tensions between doctors and patients – over bedside manner, over basic respect and, significantly, over who should be making healthcare decisions for patients’ children – could have been the cause of vaccine skepticism, rather than just a consequence of it. Parents trusted Wakefield in a way that they didn’t trust legitimate, factually correct doctors because he was willing to acknowledge their fears and their philosophical concerns.

I don’t think experts intend to ignore what the debate over vaccines is really about. They care deeply about the public health implications of vaccine refusal. They’re worried about the health of their individual patients. But they personally think the trade-off between the small risks of side effects and the big benefit of herd immunity is a fair one. They decided this long ago, and that belief is built into every aspect of their work. For a lot of them – a lot of us, if I’m honest – it’s easy to forget that our perspective on the trade-off is a belief, and not a provable fact. We are uncomfortable with the idea that opinions on scientific topics could be influenced by philosophy, politics and other things that aren’t easily quantifiable.

That’s not surprising. But it is something that has to be acknowledged. If we don’t do that, we can’t ever resolve the conflict.