My son had his first vaccination within hours of entering the world. In the London borough where he was born eight years ago, tuberculosis rates ran high. When a nurse appeared at my hospital bed bearing the BCG vaccine, I offered my red and roaring baby without a moment’s hesitation.

Since then, he and his sister have had every inoculation going. I proffer them up, willingly and gratefully, to each needle and nasal spray, unwavering in my belief in the scientific consensus that they are safe, effective and essential public health tools.

The vast majority of parents in the UK do the same, vaccinating their children in accordance with the NHS’s schedule, which recommends routine immunisations at each stage of a child’s development. So it is tempting to ignore the deniers and doubters, to write them off as extremists peddling conspiracies and disproved theories. Except this strategy appears to be failing – with potentially catastrophic consequences.

Virulent opposition to vaccines is as old as vaccines themselves. In the 18th century, Edward Jenner’s smallpox vaccine attracted wrath from religious groups, which fired up frightened parents. Andrew Wakefield’s discredited 1998 paper, spuriously linking the measles, mumps and rubella (MMR) vaccine to autism, provoked widespread, global panic, then in 2016, his film Vaxxed: From Cover-up to Catastrophe raked it up again. “Anti-vaxxers” are now back on the scene, amplified by the wild west of social media.

Yet it isn’t this group that is causing most concern. Among the top 10 threats to global health outlined this year by the World Health Organization (WHO) is “vaccine hesitancy”. This group – the undecided and the questioning – is far harder to pinpoint. They are not shrill on social media. In fact, many keep their concerns private, for fear of censure. But they have researchers and medical professionals very worried indeed.

Last month, the biggest global study conducted into attitudes about immunisation identified a crisis of confidence across Europe; the Wellcome Global Monitor survey, looking at 2018 data, found that, in France, one in three people believe that vaccines are not safe. In Ukraine, just half of those surveyed trusted in their efficacy. Both countries have experienced recent measles outbreaks.

In the UK, meanwhile, the proportion of children receiving both doses of the MMR jab by their fifth birthday has fallen for four years in a row to 87.2%, significantly below the 95% that the WHO says is necessary for population-wide protection. Last June, Public Health England (PHE) reported an almost fourfold rise in measles cases compared with the previous year.

What has provoked people to question a public health intervention described by PHE as second only to clean water in saving lives? Can listening to the growing number of “hestitant” parents shed light on how to counter the trend?

***

“I understand that there may be an increase in measles in the world. But weigh that up against somebody who’s telling you their child was deeply affected by vaccinations,” says Haru, a Londoner who works in the music industry. His one-year-old daughter would typically have had vaccinations at eight weeks, 12 weeks, 16 weeks and around her first birthday. These would have guarded against a list of diseases longer than her little arm: diphtheria, tetanus, pertussis (whooping cough), polio, haemophilus influenzae type B, hepatitis B, meningococcal group B, rotavirus gastroenteritis, 13 pneumococcal infections, meningitis, measles, mumps and rubella.

However, Haru and his partner have decided against giving their daughter any vaccinations so far. “I’m getting a lot of pressure from my parents and it’s quite difficult,” he says. “But one thing that really clinched it for me was a friend’s brother who said: ‘My child went through all the vaccinations and has never been the same since – just far less alert.’ You only need one first-hand experience like that and you’re going to disregard everything else you’ve heard. Because it’s someone who has nothing to gain either way.”

Haru is far from alone in being swayed by anecdotes from his peer group. In 2017, the WHO published a guide aimed at public health spokespeople facing the daunting prospect of engaging “vocal vaccine deniers” in debate. “The more narratives about vaccine side-effects people read, the higher is their perception of the risk of side-effects, even if they know the statistical base rate,” it says.

Meanwhile, the success of immunisation programmes means that most parents in the developed world now hear few first-hand stories about the horrors of the diseases they protect against, leaving vaccination in a catch-22. “I’m sure if people started dying in big numbers, I’d be a lot more concerned,” admits Haru. “I’m far more worried about pollution.”

When he voiced his doubts to peers and family members, he says he was made to feel like an extremist. “One of the things I’m worried about is my child being ostracised by other parents.” But, tellingly, he found endless material online to feed his fears: videos of people claiming to be doctors and nurses who weren’t vaccinating their children. On social media, of course, the loudest voices do not belong to public health bodies, or to the majority of parents who vaccinate quietly and contentedly. A 2010 study, published in the peer-reviewed academic Journal of Health Psychology, suggested that people need only view anti-vaccination websites for five to 10 minutes in order for their perception of the risks to be heightened, and their perception of the risks attached to avoiding vaccination to diminish.

You’ll think I’m mad, but I did convince myself, after my eldest had the MMR, that I saw changes in behaviour

Dr Stuart Blume, an anthropology professor at the University of Amsterdam and the author of Immunization: How Vaccines Became Controversial, argues that it’s too convenient to blame vaccine hesitancy “on people’s stupidity and anti-vaccination campaigners. Because it avoids any need on the part of the public health system to ask: are we doing something wrong? It avoids the need for any self-reflection.”

There was a time, Blume says, when people trusted what they were told and doctors could simply direct people to take vaccines. “Since then, a whole lot of things have happened to undermine people’s trust; some of these are in the vaccination system and some are broader and deeper.”

He points to the concurrent rise in populist politics. “We’re seeing the same discontents that lie behind Trump, Salvini and even Brexit. It’s systemic dissatisfaction, loss of faith and trust.” Vaccines, manufactured by big pharma and regulated by governments, are a prime target for those who mistrust people in power.

***

Annabel has two children, aged eight and four. Their artwork hangs on the walls of her sitting room, along with a certificate the eldest was recently awarded at school. He has had some of the routine vaccinations. His sister, however, has had fewer – and not the MMR.

Annabel is highly educated and works in public health. She explains that, “There’s a bit of me that wants to trust the public health authority and to play a role in the public health agenda by supporting the vaccination agenda.” Yet, she adds, “MMR didn’t feel 100% right.”

Here were her concerns: in the 50s, a licensed polio vaccine proved to be defective and resulted in the deaths of several children in the US. “Even though the methodology for making them has totally changed, that historical event, combined with knowing quite a bit about the harmful side-effects of a lot of medicines, have all fed in to my reluctance,” she says. “Combine all that with a fairly heavy distrust of the data, the MMR scare [in the late 90s] and the relatively low immediate disease-exposure risk and you have a very reluctant person.”

Annabel remembers the panic that spread in the aftermath of Wakefield’s report in 1998. “I was at an age when it really affected me,” she said and, despite numerous subsequent studies robustly discrediting any link between the vaccination and autism, her suspicion remained. “You’ll think I’m mad, but I did convince myself, after my eldest had the MMR, that I saw changes in behaviour.”

He is now a healthy and lovely boy – precociously funny, clever and sporty. Yet, when it came to her second child, first-hand experiences with health professionals tipped the balance. “A very austere nurse completely dismissed my concerns, refused to have any dialogue and frowned.” Then a private paediatrician “didn’t have the time or inclination for a discussion about risks, and spoke to me as though I was silly to consider any other option”.

She stalled for a while, then “tried to get her MMR done, but the nurse couldn’t say whether the dose would be appropriate because my child was, by this stage, older than the recommended age. I had no idea what to do.” In the end, her daughter remained unvaccinated.

***

“Doctors and health practitioners aren’t having an easy time of this, either,” says Dr Heidi Larson at the London School of Hygiene and Tropical Medicine. “Population and economic pressures mean they have less time and more people. I’ve had some mothers say to me: ‘I feel like we’re just a number.’ I think we need to work on that, to build dialogues and conversations.”

Larson and I meet in her office where, for the past 10 years, she has been running the Vaccine Confidence Project. Here, she and her team have developed an index measuring people’s attitudes towards vaccines’ importance, safety, effectiveness and compatibility with religious beliefs, painting a picture of how opinions have shifted.

Dr Heidi Larson: ‘Doctors have less time and more people. I’ve had mothers say they feel they’re just a number.’ Photograph: Patrick Lewis Dowse/The Guardian

Across the world, Larson explains, the spectrum between those who say they “strongly agree” with vaccines and those who “strongly disagree” is vast, including “70% to 80% who are questioning more and more”.

The breadth of this spectrum varies tremendously by country and region. The recent Wellcome report, which used Larson’s index, made it clear that Europe is the world’s most vaccine-sceptical region, with France its most sceptical country. The reasons differ on a local level. Sometimes, Larson explains, it stems from distrust in the system. Sometimes it has roots in a suspicion that one’s liberty or dignity is not being respected. Sometimes it is about religious beliefs; nearly three-quarters of New York’s recent measles cases have occurred among orthodox Jewish communities, for example.

“Then there’s the group that didn’t start as an anti-vaccine group, but is now part of that portfolio, intent on raising their children in the most natural, organic manner,” says Larson, linking reluctance to vaccinate to segments of the “natural parenting” community. This movement encourages parents to question some forms of medical intervention even from pregnancy, with an emphasis on drug-free births.

Larson stresses that the vast majority of these people go on to vaccinate, “but there are some who start with very genuine and responsible questions about safety. Then, when they’re not getting answers, or they feel like they’re being dismissed, they move to feeling alienated, that something’s being hidden, and are much more open to some of the negative views.”

Addressing so many disparate concerns sounds daunting. But, says Larson, “I don’t think it’s unreasonable that in this environment we should make time to try to help parents and carers navigate this pretty confusing space.”

Parents often come into the GP surgery with questions about things they have seen online, she says: “Questions that are hard to anticipate and fall outside the scope of standard training.” In January, a Royal Society for Public Health survey found that three-quarters of parents valued the advice of health visitors and midwives regarding vaccinations. Yet, it noted: “A report by PHE indicated that, over time, significantly fewer parents have reported that health visitors are discussing immunisations with them.”

The following month, a review published in the Journal of Advanced Nursing examined practice nurses’ influence on the uptake of the MMR. It found that training and annual updates were essential to keep them “abreast of the evidence base underpinning national immunisation programmes”, and pointed out that, historically, nurses had found it challenging to keep up to date. That may, however, be changing.

PHE and the Royal College of Nursing publish training standards and a curriculum for all health practitioners involved in immunisation programmes in the UK. The latest version, updated in February 2018, acknowledges that doctors and nurses are crucial to parents’ decisions: “A parental attitudinal-tracking survey conducted in England in 2016 found that 71% of parents had a discussion with a healthcare professional before their child was due to be immunised. For those who were undecided... 60% said that they felt more confident about immunising their child following their discussion.”

The alternative, of course, is to make vaccinations mandatory. In May, the health secretary, Matt Hancock, told the BBC’s Today programme: “I don’t want to have to reach the point of compulsory vaccination, but I will rule nothing out.” He added: “Failure to vaccinate when there isn’t a good reason is wrong. Those people who campaign against vaccination are campaigning against science.” Last month, the British Medical Journal published a blog post arguing that the UK should consider “refusing to allow children to go to nursery or school unless and until all their vaccinations are up to date”.

It may seem a neat answer to those without doubts or, more compellingly, for parents of children whose compromised immunity means they cannot be vaccinated and thus rely on the so-called “herd immunity” provided when a critical mass of others are immunised. But, as Jo, the mother of a seven-month-old, tells me: “To those predisposed to conspiracy theories, mandating vaccines doesn’t smell good.”

As we talk, Jo bounces her baby on her lap. She has a specific concern with vaccinations: “I had really, really bad eczema all through my childhood. Every shower or bath I had, I would be screaming in pain because there were open wounds on my arms and the backs of my legs. I had a lovely childhood, but that pain is one of the lasting memories from it. My mum is absolutely adamant that it was because of the polio vaccine, that it happened in the direct aftermath of me receiving it.”

Jo, who has a questioning and critical mind, says: “It’s a thought I grew up with, so it’s quite difficult to dislodge. That’s the fear for me, when it comes to my son.”

All trial data – good and bad – goes to regulators. It is them, not big pharma, that decide if drugs will be licensed

She decided against giving him the BCG and rotavirus jabs, believing them non-essential. Giving him the routine vaccinations suggested for babies at two, three and four months was “an immensely distressing and difficult decision”, she says. “I was in floods of tears for days. I would have done anything to avoid making a decision that I felt might result in him having debilitating eczema for the rest of his life.” He had the jabs and remains a healthy, happy baby, but Jo is still concerned that vaccines could overstimulate his immune system and result in allergies. “I’ve got five months till he’s due for the MMR and I’m still on the fence,” she says. “I’m considering doing just the measles vaccination and paying for it, although I’d need to do further research.” Some private clinics import single measles vaccines into the UK. The NHS strongly advocates against it, describing them as significantly less safe. Jo, however, wonders whether it might lessen the impact on her son’s immune system.

Yet, even though Jo has a first-class maths degree, doing that research has proved difficult. “There’s lots of hyperbole about the dangers, without any reference to supporting evidence. But then, on the pro-vaccination side, or the NHS websites, there was no real mention at all of any potential long-term risks. They mention the short-time side-effects: your baby might have an immediate fever. But I just don’t believe there can’t be any chance of long-term side-effects at all.”

***

The Vaccine Knowledge Project’s website is designed to counter this mistrust of the data. Run by the University of Oxford and approved by the World Health Organization’s Vaccine Safety Net as an independent, authoritative and reliable source of information, it answers frequently asked questions from parents and doctors, and provides detailed lists of the ingredients in vaccines and their potential side-effects. “Nothing in life, including vaccination, can be completely risk-free,” it says, “but, so far, all the evidence tells us that vaccinating is safer than not vaccinating.” There is no evidence that immunisation weakens the immune system or triggers allergies, it adds.

In the case of the MMR: “There are now a large number of studies that show no evidence at all of any link between the MMR vaccine and autism.” It lists the side-effects, including that up to one in 10 people will get a raised temperature, loss of appetite and a measles-like rash for two to three days. Temporary joint inflammation is rare in children, but more common in adult women, while about one in 50 children will experience mumps-like symptoms.

One in 1,000 people at each dose may experience fits, while one in every 24,000 doses will prompt “a skin rash of small, bruise-like spots”. As with any vaccine, medicine or food, the website points out, there is a very small chance of anaphylaxis (a severe allergic reaction): “In the UK between 1997 and 2003, there were a total of 130 reports of anaphylaxis following all immunisations. Around 117m doses of vaccines were given in the UK during this period. This means that the overall rate of anaphylaxis is around one in 900,000.”

Dr Manish Sadarangani, director of the Vaccine Evaluation Center at the University of British Columbia in Canada, tells me that there “are no long-term side-effects consistently linked with any of the routinely used vaccines.” (His organisation’s website is careful to point out that it is “separate from government and vaccine manufacturers and provides independent, expert vaccine evaluations”.)

Indeed, the long-term risks attached to the vaccines used today are, he says, “as close to zero as you can say anything is” because of the rigour of the trials and monitoring systems to which they are subjected. By the time vaccines have made it through lab trials, animal testing and three phases of increasingly large trials in order to be licensed for public use, “they’ve been given to thousands, if not tens of thousands of people”, usually in several countries.

All the trial data – good and bad – has to be submitted to regulators, such as the Food and Drug Administration in the US, and the Medicines and Healthcare Products Regulatory Agency or the European Medicines Agency. It is these, Sadarangani explains, that decide whether the drug will be licensed to use, not the big pharma companies that so many vaccine-hesitant parents distrust.

And in the case of routine childhood vaccinations in the UK, there is another gatekeeper: the Joint Committee on Vaccines and Immunisation, which re-examines the data on the safety and efficacy of each vaccine when it is introduced to the schedule, not only in isolation but when put in the context of other inoculations. Even once children begin to use them, “there is what’s called phase-four surveillance. Any adverse events that happen after a vaccine has been licensed are still reportable… That information is collected and collated,” explains Sadarangani.

I find myself breathing a sigh of relief. Worry is contagious, especially when it comes to children. Although I have not had my rational position on the issue shaken, after a week talking to vaccine-hesitant parents, I have found my nerves jangled nonetheless.

How could the wealth of information be better communicated? “Hesitancy is a spectrum,” Sadarangani says. “There’s a lot of focus on the very small number of people, the hardcore anti-vaccine people who are never going to change their mind, no matter what. But I would say most of the rest just have questions that are reasonable and want answers. Those are the people that we need to try to speak to.”

Shortly after my conversation with Sadarangani, an email arrives. It is from Annabel. She has just read “a story about the regret and devastation of parents who did not vaccinate and whose kids then got sick. Made me intent on getting them done. The GP surgery are calling me back today.”

A few hours later, she writes again: “This time, the nurse told me there were guidelines for children who need to catch up. She didn’t give me a hard time, was very kind and made it easy. We’re going in on Monday.”

• Some names have been changed. The WHO’s Vaccine Safety Net is a global network of websites that provides reliable information on vaccinations.

Comments on this piece are premoderated to ensure the discussion remains on the topics raised by the article. Please be aware that there may be a short delay in comments appearing on the site. If you would like a comment to be considered for inclusion on Weekend magazine’s letters page in print, please email weekend@theguardian.com, including your name and address (not for publication).