Vaccines are one of the great success stories of modern medicine. Because of them we are no longer vulnerable to smallpox or polio or measles. The flu vaccine, however, is a different story. Its effectiveness varies from patient to patient, from population to population, and from year to year. It needs to be updated each season, and even in a good year is usually no more than 50% effective. We may rely on it to avoid catching the flu, but its story demonstrates how far we still are from a reliable vaccine.

Vaccination, the process of infecting a healthy person with a microbe to prevent disease, dates back at least a thousand years. But the start of vaccination as we think of it today is generally credited to the work of Edward Jenner, a British physician born in 1749. Jenner was a keen observer with a deep interest in the natural world, and found time for both serious study and artistic play. He investigated everything from hydrogen balloons to the life cycle of the cuckoo, wrote poetry and played the violin, but smallpox – or rather, the eradication of it – is his legacy. Because of Jenner, this virus is not on our list of worries today.

Smallpox was a vicious disease that killed more than 30% of those who contracted it. In the 1700s, there was one demographic, however, that seemed to be immune: milkmaids. It had been observed that in the course of their job milking cows, women came into contact with the milder bovine version of the smallpox virus, called cowpox.

These women then became immune to the deadlier human smallpox virus. There was something in the cowpox that protected against smallpox, and in 1796 Edward Jenner famously took material from the fresh pustules on a milkmaid’s hand and inserted it under the skin of a young boy named James Phipps. After a brief and mild illness, Phipps recovered completely. Jenner then infected him with scrapings from a smallpox lesion, again and again, but the boy never got sick. Jenner named this process “vaccination” after variolae vaccinae, a Latin term for cowpox. His technique quickly spread through 19th-century England and beyond, saving countless people, inspiring modifications to the technique and changing the course of history.

Jenner’s smallpox vaccine was improved and modified over the next several decades, and was soon joined by others. Louis Pasteur developed vaccines for animal diseases such as chicken cholera and anthrax, but of these he is best remembered for his rabies vaccine. Rabies was a common and uniformly fatal disease in the 19th century. Once a victim is bitten by a rabid animal, the virus multiplies slowly and infects the brain and nervous system. Pasteur did not know of the viral cause, but this didn’t really matter. He dissected and dried out the spinal cords of infected animals and then injected the remains into test animals, which then showed immunity to rabies. What Pasteur was doing was, in fact, weakening the virus, making it a Goldilocks version. It was not strong enough to kill, and it was not weak enough to be ignored by our immune system.

One hundred years ago, during the 1918 flu pandemic, there were no flu vaccines. We didn’t know precisely what was causing the disease, so we couldn’t manufacture a vaccine to protect us. But this didn’t stop scientists and doctors from doing something, anything, to combat the outbreak. In 1919 Edward Rosenow from the Mayo Clinic in Rochester, Minnesota, isolated several bacteria from the sputum and lungs of flu patients in Rochester, formulated a vaccine that contained five different kinds of bacteria, and doled it out to 100,000 people. At Tufts College Medical School in Boston, Timothy Leary (whose nephew and namesake would also become a doctor and experiment with psychedelics) produced his own blended vaccine using strains from the Chelsea Naval hospital, a nurse’s nose at Carney hospital and the infected wards of Camp Devens. Leary mixed these samples together, grew them on plates of agar and then sterilised the mixture. His vaccine was sent to San Francisco, where at least 18,000 people were inoculated with it.

These and other efforts gave hope to a ravaged nation. One health official at the time wrote that the greatest value of a flu vaccine was that it reduced “fluphobia”. Worry and fear were as quick to spread as the disease itself, and any vaccine that provided at least mental relief was welcome. There was no evidence, of course, that any of these vaccines actually worked. Today, physicians go to great lengths to be sure that vaccine trials adhere to stringent standards, but a century ago these did not exist. Many of the vaccine trials were conducted on survivors of the flu, after the initial epidemic had passed, meaning that the pool was tainted with a degree of immunity.

In 1933 the flu virus was identified, and scientists could then confront the culprit rather than the mess in its wake. Russians led the field, weakening the virus by transplanting it between chicken eggs. Around one billion people in the USSR were vaccinated using the live but weakened flu virus, and it was still in use at the end of the 20th century. Although it appeared to be successful, the live-flu vaccine was never tested in a rigorous way, and it remained a constant danger. Since it used a live virus, it could cross with other strains and morph into a more virulent version.

Vaccine researchers therefore turned their attention to creating a vaccine containing what they called “inactive” strains. The virus was still grown in chick embryos, but this time it was rendered inactive by dunking it in a bath of formalin disinfectant. Although a higher dose of the inactive vaccine was needed to produce an immune response, there was no concern about the virus replicating.

For the first several years the influenza vaccine contained only one strain, the influenza A virus, because, as far as anyone knew, that was the only kind of influenza out there. In 1940 influenza B was identified, which kicked off the perpetual task of calibrating vaccines to deal with multiple evolving strains. By the 1950s we had a vaccine that was effective against both A and B, but the virus, as always, was outpacing us. By the late 1970s we had to make a vaccine to hit three strains. For the 2016-7 flu season, most of the vaccine doses manufactured in the US targeted four different strains. The past 100 years have been a ceaseless arms race against an enemy with whom we cannot negotiate.

The key to a good flu vaccine is matching it to the strains that are in circulation during a given season. The challenge is that it takes about six months to produce the vaccine, and so the manufacturers have to base their ingredients on some clever detective work led by the World Health Organization.

There are about 110 WHO flu centres in 80 countries that receive swabs from the noses and throats of patients with influenza-like illnesses. These centres identify the flu strains that are circulating, and occasionally they will find a new one. When this happens, they send it to one of five collaborating centres, in London, Atlanta, Melbourne, Tokyo and Beijing, for a more detailed molecular analysis. Twice a year (in February for the northern hemisphere, and September for the southern) the WHO convenes a meeting to collate all the information and recommend a vaccine recipe for the upcoming season. In the US, the Centers for Disease Control and Prevention (CDC) in Atlanta provide additional domestic data, and the Food and Drug Administration makes a final decision on what goes into the vaccine. The manufacturers then have about six months to get the recommended flu vaccine to market.

The US recommends universal flu vaccination, but the UK does not.

Photograph: David Cheskin/PA

Because the influenza virus can mutate so quickly, nailing the exact recipe is challenging. In some seasons the match is close to perfect, but this is not always the case. If the virus drifts after the February meeting of the WHO, there will be a mismatch between vaccine and virus. The greater the mismatch, the less effective the vaccine. In a good year, we might expect the vaccine to be 50-60% effective. In the 2004-5 flu season, that figure was only 10%, meaning that the vaccine was a big misfire. We also botched the 2014-5 season, when new strains hadn’t been included in the vaccine. That season the vaccine was a measly 19% effective, compared to over 50% in the previous year. Early on in the 2017-8 influenza season, there were near-record numbers of hospitalisations, and though the latest vaccine’s performance improved throughout the season, overall effectiveness is estimated to have been 40%.

Even if the vaccine hits the bullseye, different demographics have different reactions to it. Children have a very good response to the vaccine. The situation is more complicated with elderly patients, who have weaker immune systems overall but also have a lifetime buildup of natural immunity. After withstanding many flu seasons, their immune systems are wiser, you might say, than those of the young.

The US and most other developed countries strongly recommend that older people receive a flu vaccine. One study compared 18 different groups over 10 influenza seasons and found that the vaccine reduced the overall winter mortality rate in older people by an astonishing 50%. But CDC epidemiologists have shown that the influenza mortality rate among older people rose alongside the vaccination rate, which raises questions about the urgency to vaccinate them. The bottom line is this: even if elderly people are vaccinated, they are still the population most likely to die from influenza.

One way to better protect older people is to vaccinate an entirely different demographic: schoolchildren. This notion was elegantly demonstrated in a natural experiment in Japan. From 1962 to 1987 most Japanese schoolchildren were vaccinated against influenza; at one point the vaccine was mandatory for a solid decade. The vaccination rate grew to around 85%, but the mandatory vaccination programme was discontinued in 1994. Over the next several years, there was an increase in the number of deaths in elderly people during the flu seasons. In the US, where there had been no change in the vaccination policy, deaths of elderly people over the same flu seasons remained unchanged. Vaccinating one part of the population, in other words, benefits another.

Data can be interpreted in many ways, and each nation has crafted its own policy accordingly. The CDC has recommended the flu vaccine for all healthy children in the US since 2008. In 2013 the UK phased in a child flu vaccination policy, in contrast to the majority of European countries. Germany provides free vaccines only to older people, leaving parents to pay for their children. Across Europe, the childhood vaccination rate is 15%, compared to almost 60% in the US.

If flu vaccines are indeed mankind’s greatest weapon against the flu, why are they used in wildly different capacities?

When my colleagues gave one another the influenza vaccine at the George Washington University hospital, we were following the advice of the CDC. When patients with influenza started to roll into the emergency room a few months later, I would ask if they had received a flu shot. Many had, and yet here they were in the hospital. I knew how they felt. My only visit to the emergency department as a patient – the year I got a nasty case of the flu – was after I had been given the flu vaccine.

Despite the regular failure of the vaccine, Americans are bombarded every year with reminders and opportunities to get a flu shot. By the end of August, pharmacies are posting signs and doctors’ offices are gearing up. The vaccine is offered at many workplaces and houses of worship, and hospitals require all their health care providers to be vaccinated.

Behind this effort is the CDC, which recommends the flu vaccine for everyone over the age of six months. One CDC poster that caught my attention asked: “Who needs a flu vaccine? a) You, b) You, c) You, d) All of the above”. (In case you were wondering, the correct answer is d.) The poster reminded us that “even healthy people can get the flu, and it can be serious”. The message then got more explicit: “Everyone 6 months and older should get a flu vaccine. This means you.”

Recommendations about the use of vaccines in the US are made by the Advisory Committee on Immunization Practices (Acip), a group of more than a dozen experts with backgrounds in vaccination research, public health and health policy. It meets three times a year to review any new evidence and provide advice and guidance to the director of the CDC about the use of vaccines. As recently as 2006, the committee recommended the flu vaccine only for those at high-risk for complications of influenza and adults over the age of 50. But a couple of years later it expanded its recommendations to include everyone over the age of six months. And that recommendation has remained in place ever since.

The public health campaign by the CDC to vaccinate everyone is not shared by other countries. Europe and Australia recommend the vaccine only for the very young, the elderly and those with underlying illnesses. Healthy adults are simply not targeted. It’s very difficult to compare death rates from influenza across different countries because the definition of an influenza case varies, as does the way in wahich a country collects its own statistics. Often, deaths from viral influenza and bacterial pneumonia are listed together. It is challenging, therefore, to compare the data we have from the US and the UK. However, in the UK, the death rate from influenza in 2014 was 0.2 per 100,000 people. And in the US, it was 1.4 per 100,000. That is seven times higher than in the UK, a country that vaccinates far less of its population. These numbers must be interpreted with caution, but they do at least suggest that the approach in the UK is reasonable.

How can we properly determine whether a “vaccination for all” programme, such as the one in the US, saves more lives and protects more people than the English “vaccination for some”? Perhaps for one flu season we could encourage everyone to get vaccinated, and for the next we would encourage only those at increased risk. We could compare the influenza death rates between the two groups and get our answer. Of course, it’s more complicated than that. Because the mortality rate from influenza is so low, we would need to enroll hundreds of thousands of patients in order to see if the vaccine made a difference. However, an experiment like this might be undermined by the strains of influenza that were circulating in each year. If one year’s strain was more contagious or deadlier than the next, our experiment would tell us nothing.

Influenza patients at Fort Riley, Kansas, in 1918. Photograph: AP

We could instead collect evidence from small trials and look for trends. This method was used by the Cochrane collaboration, an international group of 37,000 medical contributors, in 2014, when they reviewed all the studies that evaluated the effects of the flu vaccine in healthy US adults. It was a large undertaking; there were 90 studies that compared giving the vaccine to withholding it, and a total of 8 million patients were involved.

The Cochrane review found that the effect of the flu vaccine in healthy adults was “small”. About 2.5% of those not vaccinated became ill, versus 1.1% of those who were. That’s very small. Put another way, you would need to vaccinate 71 people to prevent one case of actual influenza. The vaccine did not reduce the number of working days lost, or the number of hospitalisations. So yes, the vaccine does prevent influenza in young, healthy adults, but in a very modest way. So why does the US still recommend universal vaccination, while the UK does not?

It comes down to language. The CDC describes the flu like this in a poster intended for use in doctors’ offices: “The flu may make people cough and have a sore throat and fever. They may also have a runny or stuffy nose, feel tired, have body aches, or show other signs they are not well. The flu happens every year and is more common in the fall and winter in the US. People of all ages can get the flu, from babies and young adults, to the elderly.”

Not so bad. But then this is on the homepage of the CDC’s flu site: “It can cause mild to severe illness. Serious outcomes of flu infection can result in hospitalization or death. Some people, such as older people, young children, and people with certain health conditions, are at high risk of serious flu complications. The best way to prevent the flu is by getting vaccinated each year.”

The CDC’s approach to flu is that it is a potentially deadly disease that can be prevented with a vaccine. The British take another approach. Here is the advice about the flu from the NHS: “Flu is a common infectious viral illness spread by coughs and sneezes. It can be very unpleasant, but you’ll usually begin to feel better within about a week … [If you are an otherwise healthy adult] there’s usually no need to see a doctor if you have flu-like symptoms. The best remedy is to rest at home, keep warm and drink plenty of water to avoid dehydration.”

At most, according to the British, influenza can be a bit of a nuisance: “Most people will make a full recovery and won’t experience any further problems, but elderly people and people with certain long-term medical conditions are more likely to have a bad case of flu or develop a serious complication, such as a chest infection.” There’s no mention of death as a complication. It’s all very “keep calm and carry on” – just like it was during the 1918 pandemic.

Is the flu a killer or an irritant? We know with certainty that each year it kills many people in both the US and Britain. And we know with equal certainty that for almost all healthy people the flu is nothing but a minor annoyance. Both are correct. That’s the nature of influenza. It’s tricky and mysterious, causing discomfort in some of its victims and death in others. It’s just that the US and the UK quantify them in different ways.

The British version of the CDC’s vaccine advisory committee is called the Joint Committee on Vaccination and Immunisation (JCVI). It meets three times a year, reviews the scientific evidence and makes recommen-dations to the secretary of state for health if there is a need to change the vaccination policy. Andrew Pollard, the head of the JCVI, trained as a paediatrician and is now professor of paediatric infection and immunity at the University of Oxford. Pollard is extremely cognisant of the numerous effects of the flu, but for the JCVI, the biggest measure is cost-effectiveness.

It may seem cold or callous to fixate on costs when lives are at stake, but money and resources are limited, and reckless or misdirected spending can lead to poor medical practices or greater harm. For example, spending $1m on medications for those who have had a heart attack might save 1,000 lives each year. That same $1m might have been spent instead on screenings for cervical cancer, which would save the lives of 60,000 women each year. What’s more important: saving 1,000 lives or 60,000 lives? It often comes down to who is asking (and which disease it is that you have).

Andrew Pollard and his team at the JCVI looked at studies that measured the cost-effectiveness of the influenza vaccine. They concluded that given the vanishingly small number of young, healthy adults who become severely ill or die from the flu, it is not cost-effective to vaccinate this section of the population.

Pollard’s committee measures the cost to the health system itself: how much the vaccine costs and how much it decreases the number of days that patients spend in hospital or in intensive care. They also estimate the vaccine’s effect on the number of flu-related visits to doctors’ offices. What they don’t measure is the wider cost to society, which includes lost labour, lost wages or the time a parent must spend taking care of a child. These, too, are burdens on society, but they do not enter into the deliberations of the JCVI. The vaccine is cost-effective to the healthcare system when it is given to children, older people, those with medical conditions and pregnant women. It is not cost-effective when given to young, healthy adults.

In the US, the cost-effectiveness of the vaccine is less important. What is more important is whether or not it works. This approach has resulted in another difference in vaccination policy between the US and the UK, this time over the vaccine for chickenpox. The varicella vaccine can prevent both chickenpox and shingles, a later complication of the disease. In the US, the varicella vaccine is recommended for all children; the first dose is given at 12 months of age, and a booster shot four years later. In the UK, varicella is not on the list of vaccines for children. In the US, if a vaccine has been shown to work safely, the CDC generally recommends it.

At the beginning of the 1976 flu outbreak, President Gerald Ford had to choose between two perfectly sound recommendations. One was to quickly vaccinate as many people as possible, while the other was to stockpile the vaccine and wait to see whether things got worse. Ford rejected the wait-and-see approach. “We cannot afford to take a chance with the health of our nation,” he said. “Better to err on the side of overreaction than underreaction.”

This is the overriding approach to healthcare in the US. They are always ready to do more, to try the latest medications or surgical procedures, because, well, why take a chance? Compared with other western countries, the US does more invasive studies of the heart for patients with chest pain, without actually improving their outcomes. We in the US put more of our patients into the intensive care unit, even though they are, on average, less sick than their counterparts abroad. We give more chemotherapy to cancer patients near the ends of their disease, even though it improves neither the quality nor the length of their lives. We do these things because we can, because to do otherwise would be considered giving up – even if doing less would be an extremely sensible and kind decision.

Influenza is not cancer, and it is not heart disease. But the US approach to it is emblematic of the way it treats most diseases: doing more is better. If there is an unexhausted option, exhaust it. And because many vaccines have had spectacular success in preventing and eradicating some ghastly infectious diseases, the expectation is that the influenza vaccine will do the same. It’s another hi-tech solution. To most people, the word “vaccine” is tantamount to a guarantee that a disease will leave you alone.

It’s hard to make a catchy public service announcement that reflects the subtleties therein. “Vaccinate everyone over six months old” is the current message. It is easy to understand and easy to remember. A more accurate message is much clumsier: “Vaccinate school-aged children and pregnant women and probably elderly people (but the evidence is mixed) and those with chronic conditions, but no need to vaccinate young, healthy adults.” That wouldn’t really fit on a billboard. In this case, nuance may invite danger.

The quest for a better influenza vaccine continues. The holy grail would be a vaccine that covers all possible strains of influenza (so there would be no problem of mismatched vaccination) and that needs to be given only once, not every year as is now the case. Dozens of research labs across the world have worked to create this so-called universal vaccine, but so far without success. The influenza virus is just too adept at changing its disguise, remaining one step ahead of our efforts to neutralise it with a one-shot-fits-all vaccine. Although influenza is a common illness, finding an effective vaccine to prevent it remains an exceptionally challenging endeavour.

Adapted from Influenza: The Quest to Cure the Deadliest Disease in History by Jeremy Brown, published by Text on 31 January at £12.99. To order a copy for £10.99, go to guardianbookshop.com or call 0330 333 6846

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.