Coronavirus appears to be with us for the long haul. Despite the unprecedented attempts at isolation and containment, with about 50 million people locked down and economic activity virtually halted in China, cases have spiked in recent weeks. Sixty-seven countries are now reporting a total of more than 8,000 infections outside China. The World Health Organization has been wary of using the word pandemic, but last week it advised countries to prepare as if one were imminent.

Coronavirus live updates: first case confirmed at European Union offices Read more

On the same day, in a rare bit of good news, the US National Institute of Allergy and Infectious Diseases (NIAID) announced it had already received the first candidate for a vaccine against the novel virus, now being called Sars-CoV-2 – produced by a company called Moderna – and that trials could begin as early as April. Anthony Fauci, the institute’s director, said that just three months between discovery and trials was a new record, noting that “nothing has ever gone that fast”, before dropping the other shoe: even with the rapid start, completing the trials and scaling up production would take until at least next year, and there was no guarantee it would even get to that stage.

Vaccines remain science’s best – and virtually only – weapon against viruses, and yet the timeline for developing a vaccine, measured in years, seems hopelessly mismatched to a pandemic, which can spread, kill and often disappear in a matter of weeks or months. The modern record doesn’t inspire much confidence. There have been multiple international viral outbreaks over the past two decades – Sars-CoV-1, Mers, Zika, Ebola – that all provoked similar races to produce a vaccine. Yet to date only the efforts on Ebola have been successful, with a vaccine being approved last year.

Why is this? Vaccines for epidemics have to overcome the same scientific and regulatory hurdles as other promising treatments, but they also suffer from a near total lack of interest from the markets that drive the pharmaceutical industry. Only a few massive companies retain the ability to develop and produce a vaccine from start to finish, partly because of the expense and the timescales involved and partly because they’ve consolidated the patents on manufacturing processes – a situation analysts openly call an oligopoly. A success for one of these companies is a treatment for a widespread, persistent disease that they can sell every single year in perpetuity. The last industry blockbuster was Merck’s HPV vaccine Gardasil, in development for nearly 20 years, released in 2006, and still bringing in over £1bn annually. There is no way to easily apply their slow-burn research and profit model to an epidemic. As the leader of the UK’s Ebola response, Adrian Hill, told the Independent in 2014, “Unless there’s a big market it’s not worth the while of a mega-company … There was no business case to make an Ebola vaccine for the people who needed it most.”

Even if research begins during a pandemic, the unpredictable nature of outbreaks means work is often shelved if the crisis dies down, and so progress halts until the next time a similar infection flares up. Sars and Mers were also coronaviruses but after they petered out, work on them essentially stalled. Jason Schwartz, a professor at the Yale School of Public Health, said as much to the Atlantic earlier this month: “Had we not set the Sars vaccine research programme aside, we would have had a lot more of this foundational work that we could apply to this new, closely related virus.” Clinical trials take nearly a year at minimum, but sustaining basic research on viruses known to have epidemic potential means when a novel variant pops up, we’re not starting from zero each time.

The current setup is often the worst of both worlds – too slow to pick up research on new threats because the money isn’t there, and too quick to drop it if it can’t be sure the money will be there in the future. It’s a highly market-dependent system, and the market usually fails us. Peter Piot, head of the London School of Hygiene and Tropical Medicine, previously declared the entire research and development system “not fit for purpose” for epidemics.

The question is, how can we fix it? The quick delivery of the first Sars-CoV-2 vaccine candidate suggests that some recent efforts are paying off. The Moderna vaccine is in part backed by the Coalition for Epidemic Preparedness Innovations (CEPI), an organisation based in Norway and funded by governments and foundations such as the Wellcome Trust. Its goal is to keep research attention on potentially epidemic diseases, even when there isn’t an outbreak, and to develop broader treatments it calls “platforms” that could be tailored to multiple virus variants. For example, the vaccine sent to NIAID is RNA-based, making it far easier to customise and faster to produce than traditional protein-based vaccines. And CEPI was already funding a DNA-based vaccine for another coronavirus, as well as a platform to block viral cell entry, that it hopes to quickly adapt to Sars-CoV-2.

These kinds of programmes can maintain long-term vision and planning, foster international collaboration and supercharge vaccine research – but they still can’t deliver a vaccine at pandemic scale, because the biotech firms and universities they work with are simply too small, working on the order of millions of doses. The French pharma giant Sanofi alone produces over a billion doses per year. Even the US army, after producing a promising Zika vaccine in 2017, had to sign over the rights to Sanofi to get it produced. (The deal later fell through after criticism over profiteering by watchdog organisations and senator Bernie Sanders.)

There have been calls over the years, usually after an outbreak or a terrorist attack, for the state to match its commitment to research – the US government has spent more than £500m on coronavirus research over the past 20 years – with the manufacturing capacity to become truly self-reliant. The US had a facility like that until the 1990s, producing vaccines for the army at pennies per dose. The UK recently announced a new state of the art facility in Oxford that would allow the government to manufacture its own vaccines, an incredible step, but it won’t be operational until 2022.

A panicked world will be watching the vaccines inch through clinical trials over the next year. The markets will be watching too. After Fauci said it was “very difficult and very frustrating” that no large pharma companies had stepped up to produce vaccines, both Sanofi and Johnson & Johnson announced new partnerships with the US government on potential treatments. However, the health and human services secretary, Alex Azar, gave no guarantee it would be affordable. “We need the private sector to invest,” he said, “price controls won’t get us there.”

• Stephen Buranyi is a writer specialising in science and the environment