Walk past the endless rows of vegetables, past the dozens of stalls selling every possible part of a pig and, at the centre of Cao Lanh city's market, a woman is doing a brisk trade in selling rats for food. Two cages swarm with them on a table next to her. Live frogs are available too, and, on the floor near her stall is a box of sluggish snakes. Chickens and ducks cluck and quack nearby. A faint smell of urine thickens air that is already heavy from the previous night's rains.

Rats are a staple source of meat in Vietnam, farmed and sold much like any other livestock. The stallholder butchers the animals to order. Reaching into the cage she will grab an animal by its tail, hit its head across a large stone, chop off its feet and head with a large pair of scissors, skin it, cut it into pieces and place everything into a small yellow plastic bag. Inevitably, the animal's blood ends up on her hands.

Scores of people are selling and butchering live animals, breathing the same air and in constant contact with the animals' blood, urine and faeces. This woman, and many others like her who work in the farms and abattoirs deep in southern Vietnam's Mekong delta, are doing what they have done for generations. And now they are in the front line in a new scientific race to predict the next pandemic.

Rats for sale in the Cao Lanh Market. Photograph: Stephen Baker

Of the roughly 400 emerging infectious diseases that have been identified since 1940, more than 60% are zoonotic ie they came from animals. Throughout history this has been common. HIV originated in monkeys, ebola in bats, influenza in pigs and birds. The rate at which new pathogens are emerging is on the rise, even taking into account the increase in awareness and surveillance. Which pathogens will cross the species barrier next, and which one is the greatest potential public health concern, is a subject of intense interest. A modern outbreak, caused by a previously unknown virus, could travel at jet-speed around the world, spreading across the continents in just a few days, causing illness, panic and death.

Pathogens have transferred from animals to people for as long as we have had contact. The ancient domestication of livestock led to the emergence of measles, and further intensification of farming in recent decades has caused problems such as the brain-wasting Creutzfeldt-Jakob disease, the human form of BSE. Expanding trade routes in the 14th century spread the rat-borne Black Death across Europe and smallpox to the Americas in the 16th century. Today's tightly connected world has seen the spread of swine flu, Sars, West Nile virus and H5N1 bird flu.

The biggest pandemic on record was the 1918 Spanish influenza, which killed 50 million people at a time when the fastest way to travel the globe was by ship. In 2009 swine flu was the most recent pandemic that got public health officials concerned; first detected in April of that year in Mexico, it turned up in London within a week.

Group of H5N1 virus. Photograph: Alamy

One of the most worrying recent outbreaks for scientists was the re-emergence of the H5N1 bird flu virus in 2005. Jeremy Farrar, a professor of tropical medicine and global health at Oxford University and, until recently head of the university's clinical research unit in Vietnam, says he remembers the night a young girl came into the children's hospital in Ho Chi Minh City with a serious lung infection. Initially, he thought that it might have been Sars – a coronavirus that had first been identified in China in late 2002 and had spread rapidly to Canada among other places – making its comeback. That was until he heard the girl's story from a colleague.

"This is years ago and I remember the story as if it was yesterday," he says. "She had been playing with her duck, arguing with her brother. They had buried it when it died and she had dug it up later to re-bury it somewhere she wanted to bury it."

The duck was the crucial part of the evidence in determining that this was a new outbreak and Farrar says that for the next few hours, no one knew how bad it would get. Would the girl's family come in during the night with infections? Would the nurses and doctors be affected?

H5N1 did not become the next Sars and was contained, although 98 people were infected and 43 died in 2005. It has not gone away, says Farrar, and is still circulating in poultry and ducks in almost the whole of Asia, remaining a major concern for human cases, given how virulent it is when people get infected.

A successful zoonotic pathogen manages to jump from an animal to a person, invades their cells, replicates and then finds a way to transmit to other people. Working out which pathogens will make the leap – a process called "spillover" – is not easy. A pathogen from a primate, for example, is more likely to spill over to humans than a pathogen from a rat, which is more likely to do so than something from a bird. Frequency of contact is also important; someone working on a live bird farm is more likely to be exposed to a multitude of animal viruses than someone living in a city who only sees a monkey in a zoo.

"The truth is, we really don't know how much of this happens," says Derek Smith, a professor of infectious disease informatics at the University of Cambridge. "Much more is noticed today than was noticed 50 years ago and was noticed 50 years before that. There are reasons to think this might be because we disrupt habitats and come into contact with animals we haven't been in contact with before. We have different things that we do socially, perhaps, than we did in the past. But we also look harder."

Viruses and other pathogens continually flow between species, often with no effects, sometimes mutating, once in a while causing illness. This mixing is known as "viral chatter" and the more different species come into regular close contact, the higher the chances of a spillover event occurring.

"This is how viruses have always worked, the big change is us," says Mark Woolhouse, a professor of infectious disease epidemiology at the University of Edinburgh. "The big change happened probably several thousands of years ago when we became a crowd species and that gave these viruses new opportunities which they hadn't had before in humans. Ever since then, from time to time a new virus has come along to take advantage of this new, very densely populated, crowded species – humans – that it can now spread between much more easily. That process is still happening; the viruses are still discovering us. We like to think we discover viruses, but it's also the viruses discovering us."

Tracking what is moving between which species is the task of Stephen Baker's team, based at the Oxford University clinical research unit in Ho Chi Minh City. Baker is an infectious disease biologist who co-ordinates the Vizions project and I met him at his lab while I was making a Radio 4 documentary about the scientific hunt for the next big pandemic.

His sampling teams visit farms, markets and abattoirs across Vietnam to take regular blood from people at high risk of being subject to a spillover event. This high-risk cohort, which will eventually number 1,000 people, will be monitored every six months and, if they ever turn up sick at a hospital, Baker's team will get an alert. The sampling teams also take blood and faecal swabs from pigs, chickens, dogs, cats and rats and anything else living nearby.

Dead rats on sale in a Vietnamese market, where stallholders' hands can be covered in blood. Photograph: Nguyen Huy Kham /Reuters

During a trip to a smallholding near the Cao Lanh food market, Baker explains that it is at places like this, where people are in regular and close contact with animals, that scientists will be able to get their first hints of any spillovers that might become a bigger threat. The farm, which is typical of Vietnam and other parts of south-east Asia, has a range of animals – pigs, ducks and free-range chickens. They are in close exposure to each other and any farmworkers, too. The farms next door are only separated by lines of trees or small fences. As well as the farm animals, Baker's team also do their best to sample wild animals in the vicinity, including civets, rats and bats, that can easily transport pathogens across wide distances.

The other part of the Vizions project is to enrol around 10,000 people over the next three years from those who turn up to hospitals with infections of the central nervous system, respiratory system, lower gut or jaundice. By cataloguing the viruses in their blood and other bodily fluids, Baker wants to build up a database of the kinds of things circulating in different parts of the country.

If there is new influenza, or other zoonotic virus outbreak, Baker's samples will allow scientists to go back in time and investigate where it had been circulating before: "That will allow us to document, retrospectively, what animals that was circulating in and how many people were potentially exposed. We're on the front line of trying to understand how frequently these things may occur."

Another animal of interest to Baker, and many other groups around the world, is the bat. It has become clear in the past few decades that they are the source of some of the most feared human infections, including ebola, Marburg and all the rabies viruses. Bats are also the natural reservoirs for the coronaviruses (including Sars and the recent Mers virus) and newer viruses such as nipah and hendra. Sometimes these have transferred directly to people, and other times they have first crossed into domestic animals.

How do bats survive as reservoirs for all these viruses that are so deadly in other species? James Wood, head of the department of veterinary medicine at the University of Cambridge, says there is likely to be a variety of reasons, not least that bats have different or better-developed innate immune systems that allow them to cope with pathogens that kill other species. With colleagues in Ghana, he has been following populations of fruit bats, sometimes numbering in excess of 10 million individuals, that pass through Accra or Kasanka National Park in a remote part of Zambia.

Sampling on a farm on the outskirts of Cao Lanh City. Photograph: Stephen Baker

I"The particular viruses we're looking at in this species include a rabies-like virus and a henipavirus, a family of viruses in Australia and south-east Asia that have passed from bats to humans," says Wood. "The populations we study, we're repeatedly sampling from on a quarterly or two-monthly basis depending on the season the bats are there. We take blood samples and swabs and urine and faecal samples then release them."

Henipaviruses cause brain infections in people and can be deadly – around half of those infected die. These viruses have spread from bats to humans either directly, such as the 2004 outbreak of nipah in Bangladesh. Or it can spread via domestic animals; in 2010, hendra spread via horses in Australia.

Wood and his colleagues have also been looking at what other environmental factors there might in working out why, in some situations, people get infected and in others they do not. "It may be that the local ecosystem services play a key role in determining risk," he says. "It may well be that, in some situations where there's really rich biodiversity, that can act as a sink for these different viruses, which makes them less likely to spread over into human populations. In other ecosystems that are perhaps more degraded, it may well be that there is more chance, because you have just single species living on their own, there's more chance of spillover happening from bats to humans or from bats to other animals."

Efforts around the world to collect and analyse blood from people and animals will give scientists and public health officials plenty of data to help track new infections. In the best case, having sequences of viruses on file, located to particular countries or even to particular regions within countries, will give vital information after a novel virus is spotted in a hospital. As well as medical and travel histories for a patient, clinicians will be able to match the virus to known viruses and will therefore be able to concentrate their efforts in containing it. They cannot, however, use this data to predict spillover events or, more crucially, when a virus might be dangerous enough to cause a pandemic.

"Not every virus that crosses over will make it [to an outbreak]," says Woolhouse. "Understanding the differences between those that do and those that don't is a major research question. That comes back to reading the [virus] genome – the information that you're going to have quickly that you didn't have a few years ago is the genome sequence.

"If you could read that and interpret it and say, "this one does look like it has the potential to infect and spread between humans" then we're much further ahead of the game than we were before."

Engineering viruses

Ron Fouchier's microbiology labs are on the 17th floor of a building on the sprawling building site that is currently the Erasmus University medical centre based in Rotterdam. His work encompasses a wide variety of viruses, everything from influenza to HIV, carried out by PhD students and postdocs. They work on some deadly pathogens, but the safety protocols are well-trained into everyone who walks the halls and the atmosphere is convivial and unworried.

One lab, however, is not among this network of rooms. Fouchier will not say where it is and, in fact, will not even hint at its general direction from his office. Last year, in that biosafety level 3 facility, he carried out his experiments to mutate the virulent H5N1 flu virus from its wild form, which is dangerous when it infects people but cannot transmit between people, into a modified form that can potentially transmit from one person to another.

The air inside the level 3 lab is at a lower pressure than the air outside, to stop anything escaping through the doors. The air itself goes through virus filters and all experiments are carried out in small, sealed boxes where the airflow is carefully controlled. The scientists operating inside always work in pairs and have to wear masks, thick rubber gloves and are all vaccinated against H5N1. Only six people have access to the steel lockers where the mutated flu virus is stored.

The work was not without controversy. The US authorities prevented Fouchier, and a separate team of scientists led by Yoshihiro Kawaoka of the University of Wisconsin-Madison, from publishing their work for many months, fearful that the information might be used by those who want to make biological weapons.

Fouchier said his work addressed a crucial question of basic science: "Scientists didn't really know what makes any virus airborne in mammals so we were really in the dark. We knew of some mutations from previous pandemics, but whether that would apply to other flu viruses nobody knows." The only way to figure that out was to take those mutations and see if they can make the H5N1 virus airborne as well – in other words, make it transmissible between people via a cough or sneeze.

A human blood sample being taken. Photograph: Getty Images

Influenza pandemics of the past century and a half have all been viruses of the H1, H2 and H3 subtypes. So far, no H5 viruses have caused pandemics because of their inability to transmit between people in the wild. "Until we did the experiments, many expert virologists were of the opinion that H5N1 would never become airborne," says Fouchier. "With [our] information we show a very strong message to the field that we should not underestimate the chances of an unknown virus subtype causing the next virus pandemic," says Fouchier.

His experiments found that the natural version of the H5N1 virus, which is currently circulating in flocks of birds around the world, needs only five mutations in its genetic sequence to become potentially transmissible between people.

In a separate paper, Derek Smith looked at how common these mutations were in the wild. "We found that, of the five mutations that were identified, two existed in the wild in quite large numbers and that there really are only three further mutations the virus would need, of the ones Fouchier and Kawaoka identified, in order to be potentially transmissible between humans," he says. "Of those three, two had never been seen in the wild but one had been seen very, very occasionally. When it had been seen, it didn't seem to occur in a particular region and then persist for a little while and go away, it was just seen in two or three viruses sporadically."

Fouchier's work was controversial at the time of publication but he is bullish about its benefits. Better surveillance capability, for a start, since scientists can routinely look for the specific five mutations in any new influenza viruses that emerge in the wild. If a flock of chickens was found to have a virus with three or four of the dangerous mutations, for example, the decision to cull them would be more clear-cut. The genetically modified viruses will also be useful in testing new vaccines and antivirals more accurately.

The detailed knowledge being gathered about influenza is already impressive but any prediction of transmission events will require even more granular data. Such as, how many virus particles are transmitted in a cough? How likely is it that the viruses that are transmitted are the versions that are the most dangerous in terms of being able to cause a pandemic?

And this sort of work needs to happen with other viruses if scientists want a hope of predicting big pandemics. Scientists might be worried about bats, for example, but have precious little knowledge about the physiology of their viruses. "A huge amount of basic biology needs to be done with these viruses to understand their mode of transmission between different species," says Wood. "Understanding at that whole animal level about transmission but also understanding the sub-cellular mechanisms of replication of these viruses could be really valuable in terms of trying to pinpoint what particular virus features are associated with transmission to humans and which ones aren't."

Information flow

As the scientific effort to build a front-line defence against pandemics gathers pace, authorities need protocols to handle and make decisions on the information coming in. The detection of a potential pandemic virus needs scientific boots on the ground for surveillance, but what happens if they spot something they think is dangerous? A decade ago, when Sars was breaking out in China, the country restricted information and some people think this led to the outbreak lasting longer than it should have done.

Things are different now, says Farrar, who took up his new post as the director of the Wellcome Trust in October. "It really has changed out of all recognition in that 10 years and large areas of the response mode is now reasonable, we've made progress. Sars was in Asia and Canada; coming through to H5N1 we had learned a little bit and improved but there were still gaps; coming through to H7N9, which is another new virus emerging which humans do not have any immunity to in China this year, the Chinese response has been exemplary. As soon as it emerged, it was picked up, the information was communicated both privately and publicly to everybody who needed to know about it. They should be applauded, they did do a great job."

This does not mean public health cannot be improved to deal with potential new threats. The World Health Organisation is nominally in charge when a pandemic is looming and Farrar says its greatest strength is that it represents so many states. But that could also be its greatest weakness: "Because it always has to reach a compromise everybody can sign up to. We now have the international health regulations where it's mandatory that countries report new events. My view is that those regulations were, in the end, a compromise that didn't go as far as anybody, including the WHO, would want in terms of what must be reported."

We are in a better position to detect a potential problem than we have ever been, but all the surveillance does not mean scientists will not be caught out by something that is sitting in an animal to which nobody happens to be paying attention. Woolhouse says there is always the potential for something to come out of left-field, something that surprises us.

And how should anyone making policy prioritise preparing for the next pandemic with more urgent concerns? Many public health officials might point out that emerging infectious diseases are a potential future threat but we also need to deal with real, major threats now such as malaria, TB or HIV. Woolhouse says the counter argument is that, although the toll of current diseases is huge and dealing with them is important, public health services have learned to accommodate them. Emerging infections such as influenza or Sars or the next pandemic would create a shock with the potential not only to overburden health systems but to shut down travel networks, close down work.

"The concern is that these things present such a huge shock that the global system is not really able to cope," he says. "That's why, despite the somewhat forward-looking aspect of this, we think they are, and should remain, a priority. The costs of an H1N1 or Sars pandemic is in the billions to hundreds of billions – substantial costs we could do well without."

Persuading members of the public or governments to keep the surveillance networks strong is an ongoing and crucial task, Woolhouse says: "This is one of those investments that, if it's working, no one notices."

With thanks to Andrew Luck-Baker. Listen to The Next Global Killer, the Radio 4 programme on the hunt for the next pandemic, at 8pm on 26 November.