The common cold has the twin distinction of being both the world’s most widespread infectious disease and one of the most elusive. The name is a problem, for starters. In almost every Indo-European language, one of the words for the disease relates to low temperature, yet experiments have shown that low temperature neither increases the likelihood of catching a cold, nor the severity of symptoms. Then there is the “common” part, which seems to imply that there is a single, indiscriminate pathogen at large. In reality, more than 200 viruses provoke cold-like illness, each one deploying its own peculiar chemical and genetic strategy to evade the body’s defences.

It is hard to think of another disease that inspires the same level of collective resignation. The common cold slinks through homes and schools, towns and cities, making people miserable for a few days without warranting much afterthought. Adults suffer an average of between two and four colds each year, and children up to 10, and we have come to accept this as an inevitable part of life.

Public understanding remains a jumble of folklore and false assumption. In 1984, researchers at the University of Wisconsin-Madison decided to investigate one of the best-known ways of catching a cold. They infected volunteers with a cold virus and instructed them to kiss healthy test subjects on the mouth for at least one minute. (The instruction for participants was to use whichever technique was “most natural”.) Sixteen healthy volunteers were kissed by people with colds. The result: just one confirmed infection.

The most common beliefs about how to treat the disease have turned out to be false. Dubious efficacy has done little to deter humankind from formulating remedies. The Ebers Papyrus, a medical document from ancient Egypt dated to 1550BC, advises a cold sufferer to recite an incantation, “in association with the administration of milk of one who has borne a male child, and fragrant gum”. In 1924, US President Calvin Coolidge sat down in an airtight chlorine chamber and inhaled the pungent, noxious gas for almost an hour on the advice of his physicians, who were certain that his cold would be cured quickly. (It wasn’t.)

Today, “winter remedy” sales in the UK reach £300m each year, though most over-the-counter products have not actually been proven to work. Some contain paracetamol, an effective analgesic, but the dosage is often sub-optimal. Taking vitamin C in regular doses does little to ward off disease. Hot toddies, medicated tissues and immune system “boosts” of echinacea or ginger are ineffective. Antibiotics do nothing for colds. The only failsafe means of avoiding a cold is to live in complete isolation from the rest of humanity.

Although modern science has changed the way medicine is practised in almost every field, it has so far failed to produce any radically new treatments for colds. The difficulty is that while all colds feel much the same, from a biological perspective the only common feature of the various viruses that cause colds is that they have adapted to enter and damage the cells that line the respiratory tract. Otherwise, they belong to quite different categories of organisms, each with a distinct way of infecting our cells. This makes a catch-all treatment extremely tricky to formulate.

Scientists today identify seven virus families that cause the majority of colds: rhinovirus, coronavirus, influenza and parainfluenza virus, adenovirus, respiratory syncytial virus (RSV) and, finally, metapneumovirus, which was first isolated in 2001. Each has a branch of sub-viruses, known as serotypes, of which there are about 200. Rhinovirus, the smallest cold pathogen by size, is by far the most prevalent, causing up to three-quarters of colds in adults. To vanquish the cold we will need to tackle all of these different families of virus at some stage. But, for now, rhinovirus is the biggest player.

Scientists first attempted to make a rhinovirus vaccine in the 1950s. They used a reliable method, pioneered by French biologist Louis Pasteur in the 1880s, in which a small amount of virus is introduced to a host in order to provoke a defensive immunological reaction that then protects the body from subsequent infection. Even so, those who had been vaccinated caught colds just as easily as those who had not.

Over the next decade, as the techniques for isolating cold viruses were refined, it became clear that there were many more rhinoviruses than first predicted. Researchers realised it would not be possible to make a vaccine in the traditional way. Producing dozens of single-serotype vaccines, each one targeting a different strain, would be impractical. The consensus that a rhinovirus vaccine was not possible deepened. The last human clinical trial took place in 1975.

Then, in January last year, an editorial appeared in the Expert Review of Vaccines that once again raised the prospect of a vaccine. The article was co-authored by a group of the world’s leading respiratory disease specialists based at Imperial College London. It was worded cautiously, yet the claim it made was striking. “Perhaps the quest for an RV [rhinovirus] vaccine has been dismissed as too difficult or even impossible,” it said, “but new developments suggest that it may be feasible to generate a significant breadth of immune protection.” The scientists were claiming to be on the way to solving a riddle that has stumped virologists for decades. One virologist told me it was as if a door that had been closed for many, many years had been re-opened.

Part of the Imperial scientists’ motivation was the notion that since we now have vaccines for many of the most dangerous viruses (measles, polio, yellow fever, cholera, influenza, and so on), it is time to tackle the disease that afflicts us most often. “Rhinovirus is by far the most common cause of illness,” says Sebastian Johnston, a professor at Imperial and one of the authors of the editorial. “Look at what people spend on ineffective over-the-counter medications. If you had a safe and effective treatment, you’d take it.”

I asked Johnston if he was optimistic. He pointed out that because their studies so far have only been in mice, they are not sure that the vaccine will work in humans. “The data is limited,” he says. “But it’s encouraging.” It was not the resounding triumphalism that I was expecting, but then cold scientists learned long ago to be careful about making grand proclamations. Theirs is an undertaking that, more than anything, has been defined by consistent disappointment.

The first scientist to try and fail to make a rhinovirus vaccine was also the first scientist to distinguish it from the jumble of other cold viruses. In 1953, an epidemiologist called Winston Price was working at Johns Hopkins University in Baltimore when a group of nurses in his department came down with a mild fever, a cough, sore throat and runny nose – symptoms that suggested the flu. Price took nasal washings from the nurses and grew their virus in a cell culture. What he found was too small to be influenza virus. In a 1957 paper, “The isolation of a new virus associated with respiratory clinical disease in humans”, Price initially named his discovery “JH virus”, after his employer.

Price decided to try to develop a vaccine using a bit of dead rhinovirus. When the immune system encounters an invading virus – even a dead or weakened virus – it sets out to expel it. One defence is the production of antibodies, small proteins that hang around in the blood system long after the virus is gone. If the virus is encountered a second time, the antibodies will swiftly recognise it and raise the alarm, giving the immune system the upper hand.

At first, Price was encouraged. In a trial that involved several hundred people, those vaccinated with JH virus had eight times fewer colds than the unvaccinated. Newspapers across the US wanted to know: had the common cold been cured? “The telephone by my bed kept ringing until 3 o’clock in the morning,” Price told the New York Times in November 1957. The celebration would be short-lived. Though Price’s vaccine was effective against his particular “JH” rhinovirus strain, in subsequent experiments it did nothing. This indicated that more than one rhinovirus was out there.

By the late 1960s, dozens of rhinoviruses had been discovered. Even in the alien menagerie of respiratory disease, this level of variation in one species was unusual; there are just three or four influenza viruses circulating at any one time. Scientists at the University of Virginia decided to try a different tactic. Instead of inoculating patients with a single strain of rhinovirus, they combined 10 different serotypes in one injection. But after this, too, failed to shield participants from infection, they were out of ideas.

As hope for a vaccine receded, scientists began investigating other ways to combat colds. From 1946 until it closed in 1990, most research into respiratory viruses in the UK was undertaken at the Common Cold Unit (CCU), a facility backed by the Medical Research Council that occupied a former wartime military hospital in the countryside near Salisbury. In its four decades of operation, some 20,000 volunteers passed through the doors of the CCU, many to be willingly infected with cold virus in the name of scientific progress.

An early experiment at the CCU involved a group of volunteers being made to take a bath and then to stand dripping wet and shivering in a corridor for 30 minutes. After they were allowed to get dressed, they had to wear wet socks for several hours. Despite a drop in body temperature, the group did not get any more colds than a control group of volunteers who had been kept cosy.

Illustration: Nathalie Lees

The CCU began focusing on cold treatments in the 1960s and 70s, when research into a substance produced by the human body called interferon was gaining momentum. Interferons are proteins that are secreted by cells when they are attacked by a virus. They act as messengers, alerting nearby cells to the invader. These cells in turn produce an antiviral protein that inhibits, or interferes with, the virus’s ability to spread, hence the name.

In 1972, researchers at the CCU decided to investigate whether interferon could be used as a treatment for colds. They infected 32 volunteers with rhinovirus and then sprayed either interferon or placebo up their noses. Of the 16 given a placebo, 13 came down with colds. But of the 16 given interferon, only three got ill. The findings, published in The Lancet, made the front page of the New York Times (below a story on Watergate). A rush of interferon research got underway. But, once again, the excitement was premature. A review by the CCU in the 1980s uncovered a fatal flaw: interferon only worked when it was given to the patient at the same time as the virus. But in real life – that is, outside the lab – a rhinovirus enters the nose between eight and 48 hours before the onset of cold symptoms. By the time you feel a cold coming on, it is already too late.

As the 20th century drew to a close, attempts to find a cure grew more desperate. At the CCU, molecules that were found in traditional Chinese medicine, Japanese tea and oranges were all seriously interrogated. In 1990, the CCU closed. The centre had done much to advance our understanding of the virology of the cold, yet it had also exposed the enormity of the task of defeating it.

In the 1990s, as many virologists focused on HIV and Aids, research into the cold tailed off. “Common acute respiratory infections were seen as less important compared with this threat of a worldwide, lethal plague,” writes David Tyrrell, the former director of the CCU, in his 2002 book Cold Wars. A cure seemed more remote than ever.

Sebastian Johnston’s lab is on the third floor of the School of Medicine, part of Imperial College’s St Mary’s Hospital campus in Paddington, west London. Opened in 1851, the original hospital building is red-brick, with high ceilings, arched colonnades and turrets, but numerous extensions, each progressively more box-like, now hem it in. A round blue plaque on the facade states that Sir Alexander Fleming (1881-1955) discovered penicillin in a second-storey room. Entry to a recreation of Fleming’s lab is £4.

Johnston, a professor of respiratory medicine and an asthma specialist, is 58 and bespectacled, with a mop of grey curls that form a peak on his forehead. As a PhD student in 1989, he was dispatched to the CCU, not long before it closed down, to study virus detection methods. “I spent six months there,” Johnston said. “It was a strange place, basically a bunch of nissen huts connected by wooden runways, with lots of rabbits.”

For his PhD on asthma, Johnston developed a technique called polymerase chain reaction, which magnifies DNA so that viruses can be identified more precisely. To his amazement, Johnston discovered that viruses were behind 85% of asthma attacks in children; about half of those were rhinoviruses. Previously, most studies had detected viruses in fewer than 20% of asthma attacks. Johnston went on to find that rhinovirus also exacerbates symptoms in 95% of cases of smoker’s cough (formally known as chronic obstructive pulmonary disease, or COPD).

It wasn’t until the 1990s that scientists fighting rhinovirus properly understood what they were up against. By that time, electron microscopy had advanced and it was possible to see the organism up close. For a pathogen so spectacularly good at infecting our nasal passages – the “rhin” of the name is from the Greek for “nose” – rhinoviruses are astonishingly simple, being little more than strands of ribonucleic acid (RNA) surrounded by a shell: “a piece of bad news wrapped in a protein coat”, as the Nobel Prize-winning biologist Peter Medawar once observed. Under an electron microscope, they are spherical with a shaggy surface like the bobble on a knitted hat.

Though all the rhinoviruses are pretty much the same internally, a subtle alteration to the pattern of proteins on their outer shell means that, to the immune system, they all look different. It’s a cloak-and-dagger strategy, and the reason why early vaccines such as Winston Price’s failed. Antibodies produced for one rhinovirus serotype do not detect the rest. Until recently, it was believed that there were around 100 different strains, and these were grouped into the “A” and “B” families. Then, in 2007, a new cache of viruses was discovered, the “C” group, making the total more like 160.

A computer-generated image of the human rhinovirus. Photograph: Alamy

In 2003, Johnston, who was then working at Imperial, contacted Jeffrey Almond, a former professor of virology at Reading University who had been recently appointed as head of vaccine development at the pharmaceutical giant Sanofi. The company was already manufacturing a jab for influenza and was interested in tackling the common cold. Having bumped into Johnston at academic conferences, Almond felt that their ambitions were aligned. “I said: ‘Let’s think about whether we can do something dramatic,’” Almond told me. “Let’s think about how we can make a vaccine against rhino.”

For doctors, vaccines are preferable to drugs because they shield the host from invasive organisms before they cause any damage. For pharmaceutical companies, vaccines are significantly less attractive. Not only do they take years and hundreds of millions of dollars to develop, even if that process is successful – which it often isn’t – it can still be hard to make much money. Vaccines are usually injections administered on a single occasion, while drugs are taken for prolonged periods. And people don’t want to pay much for vaccines. “Everybody wants vaccines for pennies rather than pounds because you get them when you’re healthy,” Almond said. “Nobody wants to pay anything when they’re healthy. It’s like car insurance, right? But when you’re sick you will empty your wallet, whatever it takes.”

Still, Almond thought there might be a commercial case for a rhinovirus vaccine. Totting up the days off school and work, plus the secondary infections such as sinusitis that require supplementary treatment and even hospitalisation, rhinovirus places a huge burden on health systems. Last year, in the UK, coughs and colds accounted for almost a quarter of the total number of days lost to sickness, about 34m. In the US, a survey carried out in 2002 calculated that each cold experienced by an adult causes an average loss of 8.7 working hours, while a further 1.2 hours are lost attending to cold-ridden children, making the total cost of lost productivity almost $25bn (£19bn) each year. Almond convinced his bosses that, if it were possible to make one, a rhinovirus vaccination would be financially viable. “Our back-of-the-envelope calculations on what we could charge, and what the numbers of sales could be, mean that it’s likely to be quite profitable and quite interesting for a company to develop,” Almond says.

Reviewing the approaches taken in the 1960s and 70s, Almond and Johnston dismissed the idea of a mega-vaccine of all the 160 rhinovirus serotypes, believing it would be too heavy, too complex and too expensive to make. They wondered instead if there was a tiny part of the structure of viruses that is identical, or “conserved”, across the entire species that could form the basis of what is called a subunit vaccine, an approach that has had success with hepatitis B and the human papilloma virus, or HPV.

After comparing the genetic sequences of the different rhinovirus serotypes, the researchers honed in on a particular protein on the virus shell that seemed to recur across many of the serotypes. They took a piece of the conserved shell from a single rhinovirus, number 16, and mixed it with an adjuvant – a stimulus that mimics the danger signals that trigger an immune response – and injected it into mice as a vaccine. The hope was that the immune system would be jolted into recognising the shell protein as an invasive pathogen, conferring immunity against the entire rhinovirus family.

In petri dishes, the scientists mixed the immunised mouse blood with three other rhinovirus serotypes, numbers 1, 14 and 29. An immunological response to rhinovirus 1 was likely because its genetic sequence is similar to 16, but serotypes 14 and 29 are unalike. The mice’s white blood cells responded vigorously against all three strains. “Seeing responses against those two [different serotypes] was very encouraging,” Johnston said. This gave hope that the vaccine might protect against the full gamut of rhinoviruses.

The scientists gathered a group of respiratory medicine specialists to review the findings. The reviewers agreed that the results looked promising. But just as the scientists were ready to take the vaccine forward, there was a setback at Sanofi. “There was a change of direction, a change of guys at the top,” Almond said. “I took early retirement for different reasons. My boss retired as well.”

In 2013, the new management decided that the company’s priorities were elsewhere, handing back to Imperial College the patent that protects the vaccine idea from being developed by other groups. Imperial did not have the resources to develop the vaccine without outside investment. For Johnston, it was frustrating – years of research and toil in the lab had seemed to be finally yielding results. But there was little he could do. The vaccine was shelved.

Across the Atlantic, as Imperial began to search for new backers, Martin Moore, a paediatrician at Emory University in Atlanta, was working on a rival approach to the same problem. A specialist in children’s respiratory disease, for the past three years Moore has been working on a solution so straightforward that when he presented the results of his paper, published in Nature Communications last year, his colleagues struggled to accept them. “But if I pushed them, I couldn’t get a good reason for that other than, just: it hadn’t been done before,” he says.

Moore first resolved to do something about the common cold in 2014, while on holiday with his family in Florida. Shortly after they had arrived, his son, then a toddler, came down with a cold. “He wanted me to hold him day and night,” Moore said. The pair hunkered down in the hotel room watching movies while the rest of the family went to the beach. “It was frustrating because, as a virologist, we can go into the lab and slice and dice these viruses. But what are we really doing about them?”

Moore reviewed the papers from the 1960s and 70s that described the early attempts at a vaccine. He saw that the scientists had demonstrated that if they took one rhinovirus, killed it and then injected it, it would protect people against that same strain. “People actually made decent vaccines against rhinovirus in the 1960s,” Moore told me. What scientists did not account for at the time was that there were so many different serotypes. But where the scientists of the past had seen defeat, Moore saw promise. Why not simply make a vaccine made up of all the rhinoviruses? There was nothing to suggest that it would not work. The problem was not with the science, but with logistics. “I thought, the only thing between us and doing this is manufacturing and economics.”

Moore secured funding from the National Institutes of Health (NIH) and applied for samples of the different serotypes from the Centers for Disease Control and the American Type Culture Collection, a biological material repository headquartered in Virginia. He stopped short of calling in all 160 serotypes, reasoning that 50 would be enough to support his hypothesis.

After developing the vaccine, composed of these 50 serotypes, Moore tested it on a number of rhesus macaque monkeys. When their blood was later mixed with viruses in petri dishes, there was a strong antibody response to 49 of the 50 serotypes. It was not possible to see whether the vaccinated monkeys themselves would be protected from colds, since human rhinoviruses do not infect monkeys. But the ability to induce antibodies in monkey blood does correlate with protection in people.

“Maybe I shouldn’t say this, but I never had a doubt that it would produce antibodies,” Moore told me. “Our paper was about showing it can be done.” There is still a long way to go before Moore’s dream becomes reality. For the vaccine to be tested in a clinical trial, it will need to be made under good manufacturing practice (GMP) conditions – regulations that companies must adhere to for licensing. Under these regulations, substances need to be kept separate to avoid cross-contamination – a substantial challenge for a vaccine that potentially encompasses 160 serotypes (currently, the largest number of serotypes in a single vaccine, for pneumonia, is 23).

For a manufacturing model, Moore is looking to the polio vaccine, since polio and rhinovirus are biologically related. The scale of production would be many times greater, but the basic processes would be alike. In May, Moore’s start-up, Meissa Vaccines, received a $225,000 (£170,000) grant from the NIH for work on rhinovirus. He is taking leave from academia to work on the vaccines.

At this point in time, perhaps the biggest barrier to us curing the common cold is commercial. Researchers at universities can only go so far; the most generous grants from bodies such as the UK Medical Research Council are around £2m. It falls to pharmaceutical companies to carry out development beyond the initial proof of concept. “You’re looking at 10-15 years’ work, minimum, with teams of people, and you’re going to spend $1bn (£760m) at least,” Almond told me.

Successes have been rare, and there have been spectacular flops. Last year, shares in US firm Novavax fell by 83% after its vaccine for RSV, one of the virus families responsible for colds, failed in a late-stage clinical trial. While it is less common than rhinovirus, RSV can cause great harm and even death in those with weakened immunity, including infants and the elderly. An effective vaccine presented an estimated $1bn opportunity for Novavax in the US alone. Before the results came through, chief executive Stanley Erck said it could be “the largest-selling vaccine in the history of vaccines”. But in the phase III trial of elderly patients, it did little to protect against infection. In the hours after the news broke, Novavax share prices fell from $8.34 to $1.40.

Episodes such as this have made pharmaceutical companies wary. Today, vaccines constitute less than 5% of the overall pharmaceutical market, and development is consolidated in a handful of companies: Sanofi Pasteur, GlaxoSmithKline, Pfizer, AstraZeneca, Merck and Johnson & Johnson, among a few other smaller players.

After the $1bn or so spent on development, there are also manufacturing and distribution costs to consider. There needs to be a return on the initial investment. “You sure as hell can’t do it if there’s not a market at the end, you’re wasting the company’s money, and if you do that too often, you’ll bankrupt the company,” Almond says. “There isn’t a conspiracy out there that says, ‘Let’s not do vaccines so people can get ill and we charge them a lot’, nothing like that. It genuinely isn’t easy.”

In August, I called Sebastian Johnston to see if there was any news on his vaccine. He told me that he had just received confirmation of further funding from Apollo Therapeutics, a startup backed by AstraZeneca, GSK and Johnson & Johnson. This would allow his lab to test the vaccine on more strains of rhinovirus. Johnston believes that if the vaccine proves to be protective against, say, 20 serotypes, there is a good chance it will protect against all the rhinoviruses. Beginning in October, the research should take about a year and a half. “At that point, I think we’ll be at a stage where we’ll be able to go to major vaccine companies.”

If the vaccine were to make it through the clinical trials, and was approved by regulators, it would first be rolled out to high-risk groups – those with asthma and COPD, and perhaps the elderly, as the flu jab is in the UK – and then to the rest of the population. In time, as the proportion of vaccinated individuals reach a critical mass, the viruses would cease to circulate because the chain of infection will be broken – a phenomenon called herd immunity.

From where we are today, this scenario is still distant: about 80% of drugs that make it into clinical trials because they worked in mice do not go on to work in humans. Still, for the first time in decades there are now major pharmaceutical companies with rhinovirus vaccine programmes, as well as smaller university research groups like Johnston’s which, through different approaches, are all pursuing the same goal of a cure. Once again, Johnston said, “people are starting to believe it may be possible.”

Illustrations by Nathalie Lees

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.

• This article was amended on 10 October 2017. A previous version stated that rhinoviruses were behind 85% of asthma attacks in children; in fact it was viruses in general, half of which were rhinoviruses.