In the 1870s, German physician Robert Koch was trying to curtail an epidemic of anthrax that was sweeping local farm animals. Other scientists had seen a bacterium, Bacillus anthracis, in the victims’ tis­sues. Koch injected this microbe into a mouse – which died. He recovered it from the dead rodent and injected it into another one – which also died. Doggedly, he repeated this grim process for over 20 generations and the same thing happened every time. Koch had unequivocally shown that Bacillus anthracis caused anthrax.

This experiment, and those of contemporaries like Louis Pasteur, confirmed that many diseases are caused by microscopic organisms. Microbes, which had been largely neglected for a couple of centuries, were quickly cast as avatars of death. They were germs, pathogens, bringers of pestilence. Within two decades of Koch’s work on anthrax, he and many others had discovered that bacteria were also associated with leprosy, gonorrhoea, typhoid, tuberculosis, cholera, diphtheria, tetanus, and plague. Microbes became synonymous with squalor and sickness. They became foes for us to annihilate and repel.

Today, we know this view is wrong – as I explain in my new book I Contain Multitudes. Sure, some bacteria can cause disease, but they are in the minority. Most are harmless, and many are even beneficial. We now know that the trillions of microbes that share our bodies – the so-called microbiome – are an essential part of our lives. Far from making us sick, they can protect us from disease; they also help digest our food, train our immune system, and perhaps even influence our behaviour. These discoveries have shifted the narrative. Many people now see microbes as allies to be protected. Magazines regularly warn that antibiotics and sanitisers might be harming our health by destroying our microscopic support system. Slowly, the view that ‘all bacteria must be killed’ is giving ground to ‘bacteria are our friends and want to help us’.

The problem is that the latter view is just as wrong as the former. We cannot simply assume that a particular microbe is ‘good’ just because it lives inside us. There’s really no such thing as a ‘good microbe’ or a ‘bad microbe’. These broad-brush terms belong in children’s stories. They are ill-suited for describing the messy, fractious, contextual relationships of the natural world.

In reality, bacteria exist along a continuum of lifestyles. If they do us harm, we describe them as parasites or pathogens. If they exist neutrally, we call them commensals. If they benefit us, we bill them as mutualists. But these are hardly fixed categories. Some microbes can slide from one end of this parasite-mutualist spectrum to the other, depending on the strain and on the host they find themselves in. For example, the Wolbachia bacteria infect some 40 percent of insects; in some species, these microbes are sexual parasites that kill or manipulate males, whereas in others, they behave as living dietary supplements that provide vitamins missing from the host’s diet.

Other microbes can be pathogen and mutualist at the exact same time. The stomach bacterium Helicobacter pylori is well known as a cause of ulcers and stomach cancer. Less famously, it also protects against oesophageal cancer – and it’s the same strains that account for both these pros and cons. H. pylori is neither a good nor a bad microbe; it’s both.

All of this means that labels like mutualist, commensal, pathogen or parasite don’t work as definitive badges of identity. These terms are more like states of being, like hungry or awake or alive, or behaviours like cooperating or fighting. They are adjectives and verbs rather than nouns. They describe how two partners relate to one another at a given time and place.

Nichole Broderick, assistant professor at the Department of Molecular and Cell Biology, University of Connecticut, found a great example of such shifting relationships when she was studying a soil-dwelling microbe called Bacillus thuringiensis, or Bt. It produces toxins that can kill insects by punching holes in their guts. Farmers have exploited this ability since the 1920s, by spraying Bt onto crops as a living pesticide. Even organic farmers do this. The bacterium’s effectiveness is undeniable, but for decades scientists had the wrong idea about how it kills. They assumed that its toxins inflict so much damage on insects that its victims slowly starve to death. But this couldn’t be the whole story. For instance, research has shown that it takes more than a week for a caterpillar to starve, yet Bt kills in half that time.

Broderick found out what was really going on, and did so almost by accident. She suspected that caterpillars would have gut microbes that protect them from Bt, so she treated them with antibiotics and then exposed them to the pesticide. With the microbes gone, she expected, the caterpillars would die even faster. Instead, they all survived. It turns out that the gut bacteria in caterpillars, rather than protecting their hosts, are the means through which Bt kills. They are harmless if they stay in the gut, but they can pass through the holes created by Bt toxins and invade the bloodstream. When the caterpillar’s immune system senses these displaced gut microbes, it goes berserk. A wave of inflammation spreads through the caterpillar’s body, damaging its organs and interfering with its blood flow. This is sepsis, and it is what kills the insect so quickly.

The same thing probably happens to millions of people every year. We humans are also infected by pathogens that create holes in our guts; and we also get sepsis when our usual gut microbes cross over into our bloodstream. As in the caterpillars, the same microbes can be beneficial in the gut but dangerous in the blood. They are mutualists only by virtue of where they live. The same principles apply to so-called ‘opportunistic bacteria’ that live in our bodies: they are normally harmless, but they can cause life-threatening infections in people whose immune system is weakened. It all comes down to context.

‘They don’t care. It’s not a nice relationship. It’s just biology.’

Even the most long-standing of microbial partners can turn into problems under certain conditions. Mitochondria – the energy-providing structures found in the cells of all animals – are domesticated bacteria that have been incorporated into their host cells for billions of years. This is one of the most effective instances of symbiosis in all of biology, and yet even mitochondria can wreak havoc if they end up in the wrong place. A cut or a bruise can split some of your cells apart and spill fragments of mitochondria into your blood – fragments that still keep some of their ancient bacterial character. When your immune system spots them, it mistakenly assumes that an infection is under way and mounts a strong defence. If the injury is severe, and if enough mitochondria are released, the resulting body-wide inflammation can build into a lethal condition called systemic inflammatory response syndrome (SIRS). SIRS can be worse than the original injury. Absurdly, it’s simply the result of a human body mistakenly overreacting to microbes that have been living beneficially within cells for over two billion years.

Just as a garden flower can be considered a weed if it shows up in the wrong place, our microbes might be invaluable in one organ but dangerous in another, or essential inside our cells but lethal outside them. ‘If you go immunosuppressed for a little bit, they’ll kill you. When you die, they’ll eat you,’ says coral biologist Forest Rohwer of San Diego State University. ‘They don’t care. It’s not a nice relationship. It’s just biology.’ So, the world of symbiosis is one in which our allies can disappoint us and our enemies can rally to our side. It’s a world where mutualisms shatter over a distance of a few millimetres.

Why are these relationships so tenuous? Why do microbes so easily slide between pathogen and mutualist? For a start, these roles are not as contradictory as you might imagine. Think about what a ‘friendly’ gut microbe needs to set up a stable relationship with its host. It must survive in the gut, anchor itself so it doesn’t get swept away, and interact with its host’s cells. These are all things that pathogens must do, too. So both characters – mutualists and pathogens, heroes and villains – often use the same molecules for the same purposes. Some of these molecules got saddled with negative names, like ‘virulence factors’, because they were first discovered in the context of disease, but just like the microbes that use them, they are inherently neutral. They are just tools, like computers, pens and knives: they can be used to do wonderful things and terrible things.

Even helpful microbes, operating in their usual, ostensibly helpful role can indirectly harm us by creating vulnerabilities that other parasites and pathogens can exploit. Their very presence creates openings. An aphid’s microbes, though essential, release airborne molecules that attract the marmalade hoverfly. This black-and-white insect, which looks like a wasp, is death to aphids. Its larvae can eat hundreds of them over a lifetime, and the adults find prey for their offspring by sniffing out Eau de Microbiome – a scent that the aphids can’t avoid giving off.

The natural world is full of these inadvertent lures. You are giving off some of them right now. Certain bacteria turn their owners into magnets for malarial mosquitoes, whilst others put off the little bloodsuckers. Ever wonder why two people can walk through a midge-filled forest and one emerge with dozens of welts while the other just has a smile? Your personal microbes are part of the answer.

Pathogens can also use our microbes to launch their invasions, as is the case with the polio virus. It grabs molecules on the surface of gut bacteria as if they were reins, using them to ride the bacteria towards a host’s cells. The virus gets a better grip on mammalian cells and becomes more stable at our warm body temperatures after touching our gut microbes. These microbes inadvertently turn poliovirus into a more effective attacker.

The important point here is that symbiotic microbes don’t come for free. Even when they help their hosts, they create vulnerabilities. They need to be fed, housed, and transmitted, all of which costs energy. And most important, like every other organism, they have their own interests – which often clash with those of their hosts. Wolbachia is inherited from mothers to daughters, so if it does away with males, it would get more hosts in the short term; in the long term, though, it risks driving those hosts extinct. If my gut microbes suppressed my immune system they would grow more readily, but I would get sick.

Almost every major biological partnership is like this. Cheats are always a problem. Betrayal lurks perpetually on the horizon. Couples might work well together, but if one partner can get the same benefits without spending as much energy or effort, it will do so unless punished or policed. HG Wells wrote about this in 1930 in The Science of Life:

‘Every symbiosis is, in its degree, underlain with hostility, and only by proper regulation and often elaborate adjustment can the state of mutual benefit be maintained. Even in human affairs, the partnerships for mutual benefit are not so easily kept up, in spite of me being endowed with intelligence and so being able to grasp the meaning of such a relation. But in lower organisms, there is no such comprehension to help keep the relationship going. Mutual partnerships are adaptations as blindly entered into and as unconsciously brought about as any others.’

These principles are easy to forget. We like our black-and-white narratives, with clear heroes and villains. The very term symbiosis has been twisted so that its original, neutral meaning – ‘living together’ – has been infused with positive spin, and almost flaky connotations of cooperation and blissful harmony. But evolution doesn’t work that way. It doesn’t necessarily favour cooperation, even if that’s in everyone’s interests. And it saddles even the most harmonious relationships with conflict.

We can see this more clearly if we leave the world of microbes and think a little bigger. Take oxpeckers. These brown birds live in sub-Saharan Africa, clinging to the flanks of giraffes and antelope. They’re classically viewed as cleaners that pick ticks and blood-sucking parasites off their hosts. But they also peck at open wounds – a less helpful habit that stymies the healing process and increases the risk of infection. These birds crave blood, and they satisfy that craving in ways that either profit their hosts, or punish them, depending on context.

A well-functioning partnership could easily be seen as a case of reciprocal exploitation

A similar dynamic goes on in coral reefs, where a small fish called the cleaner wrasse runs a natural health spa. Big fish arrive, and the wrasse picks parasites from their jaws, gills, and other hard-to-reach places. The cleaners get meals, and the clients get healthcare. But the cleaners sometimes cheat by nipping bits of mucus and healthy tissue. The clients punish them by taking their business elsewhere, and the cleaners themselves will castigate any colleagues that annoy potential customers.

Meanwhile, in South America, acacia trees rely on ants to defend them from weeds, pests and grazers. In return, they give their bodyguards sugary snacks to eat and hollow thorns to live in. It looks like an equitable relationship, until you realise that the tree laces its food with an enzyme that stops the ants from digesting other sources of sugar. The ants are not just beneficiaries; they are also indentured servants.

All of these are iconic examples of cooperation, commonly found in textbooks and wildlife documentaries. And each of them is tinged with conflict, manipulation, and deceit in ways that popular presentations rarely acknowledge. ‘We need to separate important from harmonious. The microbiome is incredibly important but it doesn’t mean that it’s harmonious,’ says evolutionary biologist Toby Kiers at Vrije Universiteit in the Netherlands. A well-functioning partnership could easily be seen as a case of reciprocal exploitation. ‘Both partners may benefit but there’s this inherent tension. Symbiosis is conflict – conflict that can never be totally resolved.’

Such a relationship can, however, be managed and stabilised. Our internal biology has no ideology of ‘good’ and ‘bad’ microbes. We have evolved many solutions to the ever-present conflicts that exist with our microbes, and many ways of enforcing our contracts with them. We can restrict them to certain parts of our bodies by creating physical corrals or creating chemical no-go zones. We can go for the carrot, by nourishing the species we want using dedicated foods. We can beat them with the stick, by using our immune system to keep them in place. We have evolved ways of selecting which species live with us, and of controlling their behaviour so they are more likely to be mutualistic than pathogenic.

These means of control reflect the true lesson behind the microbiome: not that nature is inherently harmonious and cooperative, but that all the best relationships take work.

From the forthcoming I CONTAIN MULTITUDES: The Microbes Within Us and a Grander View of Life by Ed Yong. Copyright © 2016 by Ed Yong. To be published on August 9, 2016 by Ecco, an imprint of HarperCollins Publishers. Published in the UK by The Bodley Head.