‘So, what’s in the thermos?” I asked.

I was standing in a lift at Washington University in St Louis, with Professor Jeff Gordon and two of his students, one of whom was holding a metal canister.

“Just some faecal pellets in tubes,” she said.

“They’re microbes from healthy children, and also from some who are malnourished. We transplanted them into mice,” explained Gordon, as if this was the most normal thing in the world.

The lift doors opened, and I followed Gordon, his students, and the thermos of frozen pellets into a large room. It was filled with rows of sealed chambers made of transparent plastic. Peering inside one of these chambers, I met the eyes of one of the strangest animals on the planet. It looked like just a mouse, and that is precisely why it was so weird. It was just a mouse, and nothing more.

Almost every other animal on Earth, whether centipede or crocodile, flatworm or flamingo, hippo or human, is a teeming mass of bacteria and other microbes. Each of these miniature communities is known as a microbiome. Every human hosts a microbiome consisting of some 39 trillion microbes, roughly one for each of their own cells. Every ant in a colony is a colony itself. Every resident in a zoo is a zoo in its own right. Even the simplest of animals such as sponges, whose static bodies are never more than a few cells thick, are home to thriving microbiomes.

But not the mice in Gordon’s lab. They spend their entire lives separated from the outside world, and from microbes. Their isolators contain everything they need: drinking water, brown nuggets of chow, straw chips for bedding, and a white styrofoam hutch for mating in privacy. Gordon’s team irradiates all of these items to sterilise them before piling them into loading cylinders. They sterilise the cylinders by steaming them at a high temperature and pressure, before hooking them to portholes in the back of the isolators, using connecting sleeves that they also sterilise.

It is laborious work, but it ensures that the mice are born into a world without microbes, and grow up without microbial contact. The term for this is “gnotobiosis”, from the Greek for “known life”. We know exactly what lives in these animals – which is nothing. Unlike every other mouse on the planet, each of these rodents is a mouse and nothing more. An empty vessel. A silhouette, unfilled. An ecosystem of one.

Each isolator had a pair of black rubber gloves affixed to two portholes, through which the researchers could manipulate what was inside. The gloves were thick. When I stuck my hands in, I quickly started sweating.

I awkwardly picked up one of the mice. It sat snugly on my palm, white-furred and pink-eyed. It was a strange feeling: I was holding this animal but only via two black protrusions into its hermetically sealed world. It was sitting on me and yet completely separated from me. When I had shaken hands with Gordon earlier, we had exchanged microbes. When I stroked this mouse, we exchanged nothing.

The mouse seemed normal, but it was not. Growing up without microbes, its gut had not developed properly – it had less surface area for absorbing nutrients, its walls were leakier, it renewed itself at a slower pace, and the blood vessels that supplied it with nutrients were sparse. The rest of its body hadn’t fared much better. Compared with its normal microbe-laden peers, its bones were weaker, its immune system was compromised, and it probably behaved differently too. It was, as microbiologist Theodor Rosebury once wrote, “a miserable creature, seeming at nearly every point to require an artificial substitute for the germs [it] lacks”.

Most microbes do not make us sick. At worst, they are hitchhikers. At best, they are invaluable parts of our bodies

The woes of the germ-free mouse vividly show just how invaluable the microbiome is. Most of us still see microbes as germs: unwanted bringers of pestilence that we must avoid at all costs. This stereotype is grossly unfair. Most microbes do not make us sick. At worst, they are passengers or hitchhikers. At best, they are invaluable parts of our bodies: not takers of life but its guardians. They help to digest our food, educate our immune systems, protect us from disease, sculpt our organs, guide our behaviour, and maintain our health. This wide-ranging influence explains why the microbiome has, over the last decade, become one of the hottest areas of biology, and why Gordon – arguably the most influential scientist in the field – is so fascinated by it.

By studying our microbial companions, he is trying to unpick exactly how the microbiome is connected to obesity and its polar opposite – malnutrition. He is studying which species of microbes influence these conditions, and how they in turn are influenced by our diets, our immune systems, and other aspects of our lives. Ultimately, he wants to use that knowledge to manipulate the microbial worlds within us to improve our health.

* * *

Jeff Gordon may be one of the most respected scholars of the human microbiome, but he is also one of the hardest to get in touch with. It took me six years of writing about his work to get him to answer my emails, so visiting his lab was a hard-won privilege. I arrived expecting someone gruff and remote. Instead, I found an endearing and affable man with crinkly eyes, a kindly smile, and a whimsical demeanour. As he walked around the lab, he called people “professor” – including his students. His aversion to the media comes not from aloofness, but from a distaste for self-promotion. He even refrains from attending scientific conferences, preferring to stay out of the limelight and in his laboratory.

Ensconced there, Gordon has done more than most to address how microbes affect our health. But whenever I asked Gordon about his influence, he tended to deflect credit on to students and collaborators past and present – a roster that includes many of the field’s biggest stars. Their status testifies to Gordon’s – he’s not just a king, but a king-maker, too. And his figurehead status is all the more remarkable because long before the microbiome crossed his mind, he was already a well-established scientist who had published hundreds of studies on how the gut develops in a growing human body.

Professor Jeff Gordon, one of the world’s leading experts on the human microbiome, talks to students at Washington University in St Louis. Photograph: Mark Katzman

In the 1990s, he started to suspect that bacteria influence this process, but he was also struck by how difficult it would be to test that idea. The gut contains thousands of species of microbes. Gordon aimed to isolate parts of this daunting whole and examine it under controlled conditions. He needed that critical resource that scientists demand but biology withholds: control. In short, he needed germ-free mice – and lots of them – so he bred them himself. He could load these rodents with specific microbes, feed them with pre-defined diets, and do so again and again in controlled and repeatable conditions. He could treat them as living bioreactors, in which he could strip down the baffling complexity of the microbiome into manageable components that he could systematically study.

In 2004, Fredrik Bäckhed, a member of Gordon’s team, used the sterile rodents to run an experiment that would set the entire lab on a focused path – one devoted to understanding the connections between the microbiome, nutrition, and health. They inoculated germ-free mice with microbes harvested from the guts of conventionally raised rodents. Normally, the sterile rodents can eat as much as they like without putting on weight, but this ability disappeared once their guts were colonised. They didn’t start eating any more food – if anything, they ate slightly less – but they converted more of that food into fat and so put on more pounds.

Mouse biology is similar enough to that of human beings for scientists to use them as stand-ins in everything from drug testing to brain research; the same applies to their microbes. Gordon reasoned that if those early results apply to humans, our microbes must surely influence the nutrients that we extract from our food, and thus our body weight. That was a powerful insight. We typically think of weight as a simple balance between the calories we take in through food and those we burn through physical activity. By contrast, the idea that multitudes of organisms in our bodies could influence that balance was outlandish at the time. “People weren’t talking about it,” says Gordon.

And yet, in 2004, team member Ruth Ley found another connection between microbes and weight, when she showed that obese people (and mice) have different communities of microbes in their guts. The most obvious difference lay in the ratio of the two major groups of gut bacteria – the firmicutes and the bacteroidetes. Obese people had more firmicutes and fewer bacteroidetes than their leaner counterparts. This raised an obvious question: does extra body fat cause a relative increase in firmicutes – or, more tantalisingly, does the tilt make individuals fatter? Is the connection, as Gordon likes to put it, causal or casual? The team couldn’t answer that question by relying on simple comparisons. They needed experiments.

The mice that got microbes from lean donors put on 27% more fat, while those with obese donors packed on 47% more fat

That’s where Peter Turnbaugh came in. Then a graduate student in the lab, he harvested microbes from fat and lean mice, and then fed them to germ-free rodents. Those that got microbes from lean donors put on 27% more fat, while those with obese donors packed on 47% more fat. It was a stunning result: Turnbaugh had effectively transferred obesity from one animal to another, simply by moving their microbes across. “It was an ‘Oh my God’ moment,” said Gordon. “We were thrilled and inspired.”

These results showed that the guts of obese individuals contain altered microbiomes that can indeed contribute to obesity, at least in some contexts. The microbes were perhaps harvesting more calories from the rodents’ food, or affecting how they stored fat. Either way, it was clear that microbes don’t just go along for the ride; sometimes, they grab the wheel.

They can also turn it in both directions. While Turnbaugh showed that gut microbes can lead to weight gain, others have found that they can trigger weight loss. Akkermansia muciniphila, one of the more common species of gut bacteria, is over 3,000 times more common in lean mice than in those genetically predisposed to obesity. If obese mice eat it, they lose weight and show fewer signs of type 2 diabetes.

Gut microbes also partly explain the remarkable success of gastric bypass surgery – a radical operation that reduces the stomach to an egg-sized pouch and connects it directly to the small intestine. After this procedure, people tend to lose dozens of kilograms, a fact typically accredited to their shrunken stomachs. But as a side-effect, the operation also restructures the gut microbiome, increasing the numbers of various species, including Akkermansia. And if you transplant these restructured communities into germ-free mice, those rodents will also lose weight.

Experiments on mice using gut microbes could lead to a greater understanding of the causes of obesity. Photograph: Deco Images II/Alamy

The world’s media treated these discoveries as both salvation and absolution for anyone who struggles with their weight. Why bother adhering to strict dietary guidelines when a quick microbial fix is seemingly around the corner? “Fat? Blame the bugs in your guts,” wrote one newspaper. “Overweight? Microbes might be to blame,” echoed another. These headlines are wrong. The microbiome does not replace or contradict other long-understood causes of obesity; it is thoroughly entangled with them.

Another of Gordon’s students, Vanessa Ridaura, demonstrated this in 2013 by using mice to stage battles between the gut microbes of lean and obese people. First, she loaded these human microbial communities into two different groups of germ-free rodents. Next, she housed the mice in the same cages. Mice readily eat each other’s droppings and so constantly fill their guts with their neighbours’ microbes. When this happened, Ridaura saw that the “lean” microbes invaded guts that were already colonised by “obese” communities, and stopped their new hosts from putting on weight. The opposite invasions never worked: the obese communities could never establish themselves in the gut when the lean ones were already there.

It’s not that the lean communities were inherently superior at taking hold in a mouse’s gut. Instead, Ridaura had tipped the battles in their favour by feeding her mice with plant-heavy chow. Plants contain a wide variety of complex fibres, and microbe communities from lean guts contain a wider range of fibre-busting species than those from obese guts. So, when the obese communities colonised lean guts, they found that every morsel of fibre was already being devoured.

By contrast, when the lean communities entered obese guts, they found a glut of uneaten fibre – and flourished. Their success only evaporated when Ridaura fed the mice with fatty, low-fibre chow, designed to represent the worst extremes of the western diet. Without fibre, the lean communities couldn’t establish themselves or stop the mice from putting on weight. They could only infiltrate the guts of mice that ate healthily. The old dietary advice still stands, over-enthusiastic headlines be damned.

An important lesson emerged: microbes matter but so do we, their hosts. Our guts, like all ecosystems, aren’t defined just by the species within them but also by the nutrients that flow through them. A rainforest isn’t just a rainforest because of the birds, insects, monkeys, and plants within it, but also because ample rain and sunlight fall from above, and bountiful nutrients lurk in the soil. If you threw the forest’s inhabitants into a desert, they would fare badly. Ridaura’s experiments emphasised that although the microbiome can help to explain what makes us fat or lean, it offers no simple solutions. And that’s something the team learned a second time, by studying a very different condition, in a very different part of the world.

Malawi has among the highest rates of child mortality in the world, and half of these deaths are due to malnourishment. One form of malnourishment, known as kwashiorkor, is especially severe and hard to treat. From an early age, a child’s fluids leaks from their blood vessels, leading to puffy swollen limbs, distended stomachs, and damaged skin.

Kwashiorkor has long been shrouded in mystery. It is said to be caused by protein-poor diets, but how can that be when children with kwashiorkor often don’t eat any less protein than those with marasmus, another form of severe malnutrition? For that matter, why do these children often fail to get better despite eating protein-rich food delivered by aid organisations? And why is it that one child might get kwashiorkor while their identical twin, who shares all the same genes, lives in the same village, and eats the same food, gets marasmus instead?

Gordon thinks that gut microbes are involved, and might explain the differences in health between children who, on paper, look identical. After his team carried out their groundbreaking obesity experiments, he started to wonder: if bacteria can influence obesity, could they also be involved in its polar opposite – malnutrition? Many of his colleagues thought it unlikely but, undeterred, Gordon launched an ambitious study. His team went to Malawi and collected regular stool samples from infants until the age of three; some had kwashiorkor, while others were healthy.

The team found that babies with kwashiorkor don’t go through the same progression of gut microbes as their healthy counterparts. Typically, these microbial communities change in the first years of life, in dramatic but predictable ways. Just as new islands are first colonised by lichens, then shrubs, then trees, so too is the infant gut colonised by waves of species that arrive in standardised patterns. But in kwashiorkor infants, microbiomes fail to diversify and mature correctly. Their inner ecosystems become stagnant. Their microbiological age soon lags behind their biological age.

When Gordon’s team transplanted these immature communities from children with kwashiorkor into germ-free mice, the rodents lost weight – but only if they also ate chow that mirrored the nutrient-poor Malawian diet. If the mice ate standard rodent chow, they didn’t lose much weight, no matter whose bacteria they were carrying. It was the combination of poor food and the wrong microbes that mattered. The kwashiorkor microbes seemed to interfere with chemical chain reactions that fuel our cells, making it harder for children to harvest energy from their food – food that contains very little energy to begin with.

The standard treatment for malnutrition is an energy-rich, fortified blend of peanut paste, sugar, vegetable oil and milk. But Gordon’s team found that the paste only has a brief effect on the bacteria of children with kwashiorkor (which perhaps explains why it doesn’t always work). As soon as they reverted to their normal Malawian diet, their microbes also boomeranged back to their earlier impoverished state. Why?

All ecosystems have a certain resilience to change, which must be overcome to push them into a different state. That’s true for coral reefs, rainforests, grassland – and a child’s gut. A poor diet could change the microbes within the gut. The dietary deficiencies could also impair the child’s immune system, changing its ability to control the gut microbiome, and opening the door to harmful infections that alter the gut communities even further. These communities could themselves start to harm the gut, stopping it from absorbing nutrients efficiently and leading to even worse malnutrition, more severe immune problems, more distorted microbiomes, and so on.

This is what microbiome scientists call dysbiosis – a state where the entire microbial community shifts into a harmful configuration. None of its members causes disease in its own right; instead, the entire community is at fault. It’s not clear exactly why the microbiomes of malnourished infants stall in their development in the first place. There are many possible reasons including antibiotic exposures, gut diseases, and poor diets, which vary from person to person. What’s clearer is that once microbiomes end up in a dysbiotic state, it can be hard to pull them back.

But Gordon is trying. His student Laura Blanton, the same woman who I met carrying that thermos of mouse droppings in the lift, recently implanted mice with microbes from either healthy infants or underweight ones. She then housed rodents from both groups in the same cages, allowing them to swap their microbiomes. When they did so, the normal communities from the healthy infants invaded and displaced the immature communities from the malnourished ones.

Blanton found that five species of bacteria from the healthy microbiomes were especially good at colonising the immature ones. When she fed this quintet to mice carrying the microbiomes of malnourished children, the rodents put on weight in a normal, healthy way. Rather than breaking down the amino acids in their diet for energy, they instead converted these nutrients into flesh and muscle.

This promising experiment suggests that the team might be able to create a probiotic cocktail of specially chosen bacteria that can turn a dysbiotic gut into a healthy one. But there’s reason to be cautious. Despite the hype that surrounds them, current probiotics – products that contain supposedly beneficial microbes – confer few big health benefits, because they contain small amounts of bacteria and consist of strains that are bad at taking hold in the gut. Gordon knows that if he wants to concoct better products, he must find ways of giving the incoming microbes a competitive advantage in their new homes. Maybe that means pairing the probiotics with foods that will nourish them. Maybe it means treating the human hosts as well as the microbes they carry, or training their immune systems to accept the newcomers.

Gordon is optimistic but cautious. As he sees it, studying the microbiome will ultimately help us to better treat conditions that are still mysterious and often intractable. But as he has said to me on more than one occasion, he’s wary of the intense hype that clouds the microbiome world. “I talk about the importance of sobriety and humility,” he says. “There’s lots of hope and expectation around this transcendent view of ourselves.” But he and other microbiome researchers still need to show that their discoveries can help people.

Bifidobacterium are used as a probiotic to promote good digestion, boost immune function, and increase resistance to infection. Photograph: Phototake/Alamy

* * *

Discoveries by Gordon and others have created the perception that the microbiome is the answer to everything. It has been linked to an absurdly long list of conditions that includes Crohn’s disease, ulcerative colitis, irritable bowel syndrome, colon cancer, type 1 diabetes, type 2 diabetes, coeliac disease, allergies, atherosclerosis, autism, asthma, Alzheimer’s disease, Parkinson’s disease, multiple sclerosis, depression, anxiety, rheumatoid arthritis, stroke, and many more.

Many of these proposed links are just correlations. Researchers often compare people with a particular disorder to healthy volunteers, find microbial differences, and stop. Those differences hint at a relationship but they don’t reveal its nature or its direction. Studies by Gordon and others go one step further. By showing that transplanted microbes can reproduce health problems in germ-free mice, they strongly hint at a causal effect.

Still, they provide more questions than answers. Did the microbes set symptoms in motion or just make a bad situation worse? Was one species responsible, or a group of them? Is it the presence of certain microbes that matters, or the absence of others, or both? And even if experiments show that microbes can cause diseases in mice and other animals, we still don’t know if they actually do so in people. Beyond the controlled settings of laboratories and the atypical bodies of lab rodents, are microbial changes really affecting our everyday health? When you enter the messy, multifaceted world of dysbiosis, the lines of cause and effect become much harder to untangle.

There is still a lot about the microbiome that we do not understand, and some of what we think we know is almost certainly wrong.

Remember how obese people and mice have more firmicutes and fewer bacteroidetes in their guts than their lean counterparts? This famous finding worked its way into the mainstream press and the scientific literature – and it’s a mirage. In 2014, two attempts to re-analyse past studies found that the F/B ratio is not consistently connected to obesity in humans. This doesn’t refute a connection between the microbiome and obesity. You can still fatten germ-free mice by loading them with microbes from an obese mouse (or person). Something about these communities affects body weight; it’s just not the F/B ratio, or at least not consistently so.

It is humbling that, despite a decade of work, scientists are barely any closer to identifying microbes that are clearly linked to obesity, which has received more attention from microbiome researchers than any other. “I think that everybody is coming to the realisation that, unfortunately, a really compelling simple biomarker, like the percentage of a certain microbe, is not going to be enough to explain something as complicated as obesity,” said Katherine Pollard, who led one of the re-analyses.

These conflicting results naturally arise in the early days of a field because of tight budgets and imprecise technology. Researchers run small, exploratory studies comparing handfuls of people or animals in hundreds or thousands of ways. “The problem is that they end up being like the Tarot,” said Rob Knight, another leading microbiome scientist. “You can tell a good story with any arbitrary combination.”

Human geneticists faced the same problem. In the early 21st century, when technology hadn’t quite caught up with ambition, they identified many genetic variants that were linked to diseases, physical traits, and behaviours. But once sequencing technology became cheap and powerful enough to analyse millions of samples, rather than dozens or hundreds, many of these early results turned out to be false positives. The human microbiome field is going through the same teething problems.

It doesn’t help that the microbiome is so variable that the communities in lab mice can differ if they belong to different strains, come from different vendors, were born to different mothers, or were reared in different cages. These variations could account for phantom patterns or inconsistencies between studies. There are also problems with contamination. Microbes are everywhere. They get into everything, including the chemical reagents that scientists use in their experiments. But these problems are now being ironed out. Microbiome researchers are getting increasingly savvy about experimental quirks that bias their results, and they’re setting standards that will shore up the quality of future studies. They are calling for experiments that will show causality, and tell us how changes in the microbiome lead to disease. They are looking at the microbiome in even greater detail, moving towards techniques that can identify the strains within a community, rather than just the species.

They are also setting up longer studies. Rather than capturing a single screenshot of the microbiome, they are trying to watch the entire movie. How do these communities change with time? What makes them resilient or unstable? And does their degree of resilience predict a person’s risk of disease? One team is recruiting a group of 100 volunteers who will collect weekly stool and urine samples for nine months, while eating specific diets or taking antibiotics at fixed times. Others are leading similar projects with pregnant women (to see if microbes contribute to pre-term births) and people at risk of developing type 2 diabetes (to see if microbes affect their progression to full-blown disease).

And Gordon’s group has been charting the normal progression of microbes in healthy developing babies, and how it stalls in kids with kwashiorkor. Using stool samples collected from Bangladeshi and Malawian children over their first two years, the team has created a score that measures the maturity of their gut communities and will hopefully predict if symptomless infants are at risk of developing kwashiorkor. The ultimate goal of all of these projects is to spot the signs of disease as early as possible, before a body turns into the equivalent of an algal reef or a fallow field: a degraded ecosystem that is very hard to repair.

Children wait for water at a borehole near Malawi’s capital Lilongwe. Photograph: Mike Hutchings/Reuters

* * *

“Professor Planer!” said Jeff Gordon. “How are you?” He meant Joe Planer, one of his students, who was standing in front of a standard laboratory bench, complete with pipettes, test tubes and Petri dishes, all of which had been sealed in a transparent, plastic tent. It looked like one of the isolators from the germ-free facility but its purpose was to exclude oxygen rather than microbes. It allowed the team to grow the many gut bacteria that are extremely intolerant of the gas. “If you write the word oxygen on a piece of paper and show it to these bugs, they’ll die,” said Gordon.

Starting off with a stool sample from a Malawian child with kwashiorkor, Planer used the anaerobic chamber to culture as many of the microbes within it as possible. He then picked off single strains from these collections, and grew each one in its own compartment. He effectively turned the chaotic ecosystem within a child’s gut into an orderly library, dividing the teeming masses of microbes into neat rows and columns. “We know the identity of the bacteria in each well,” he said. “We’ll now tell the robot which bacteria to take and combine in a pool.”

He pointed to a machine inside the plastic, a mess of black cubes and steel rods. Planer can programme it to suck up the bacteria from specific wells and mix them into a cocktail. Grab all the Enterobacteriaceae, he might say, or all the Clostridia. He can then transplant these fractions back into germ-free mice to see if they alone can confer the symptoms of kwashiorkor. Is the whole community important? Will the culturable species do? A single family? A single strain? The approach is both reductionist and holistic. They’re breaking down the microbiome, but then recombining it. “We’re trying to work out which actors are responsible,” said Gordon.

A few months after I saw Planer working with the robot, the team had narrowed down the kwashiorkor community to just 11 microbes that replicate many of the disease’s symptoms in mice. None of these were harmful on their own. They only caused a problem when acting together – and even then, only when the mice were starved of nutrients. The team also created culture collections from healthy twins who didn’t develop kwashiorkor, and identified two bacteria that counteract the damage inflicted by the deadly 11. The first is Akkermansia, which is being studied as a way of reducing body weight, but seemingly guards against malnutrition too. The second is Clostridium scindens, which tamps down inflammation by stimulating certain branches of the immune system.

Opposite the tented bench, there was a blender that could take foods representative of different diets and pulverise them into rodent-friendly chow. (On a piece of sticky tape, affixed to the blender, someone had written “Chowbacca”.) Gordon’s lab could now explore the behaviour of Akkermansia and C scindens, either in test tubes or in the gnotobiotic mice, and work out which nutrients the microbes needed. This allowed the team to compare the effects of the same microbes when fed a Malawian diet, or an American one, or on sugars from breast milk that have specifically evolved to feed beneficial microbes. Which of these foods works best? And which genes do the microbes switch on? The team can take any one microbe and create a library of thousands of mutants, each of which contains a broken copy of a single gene. They can put these mutants in a mouse to see which genes are important for surviving in the gut, liaising with other microbes, and both causing or protecting against kwashiorkor.

What Gordon has built is a causality pipeline – a set of tools and techniques that, he hopes, will more conclusively tell us how our microbes affect our health, and take us from guesswork and speculation to actual answers. Kwashiorkor is just the start. The same techniques could work for any disease with a microbial influence.

It is the right time to be doing this work. Our planet has entered the Anthropocene – a new geological epoch when humanity’s influence is causing global climate change, a loss of wild spaces, and a drastic decline in the richness of life. Microbes are not exempt. Whether on coral reefs or human guts, we are disrupting the relationships between microbes and their hosts, often pulling apart species that have been together for millions of years. Gordon is working hard to understand these partnerships to better forestall their untimely end. He is not just a scholar of the microbiome; he is one of its stewards.

Main photograph of faecal bacteria: Science Photo Library

This is an edited extract from I Contain Multitudes, published by Bodley Head

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.