Historically, microbes have been associated with deadly diseases such as bubonic plague, smallpox and malaria. But they have had a bad press: only a tiny minority are pathogenic (capable of causing disease), and in fact many of the microbes that live on and in our bodies – especially the trillions of bacteria residing in our gut – may be beneficial, or indeed essential, for human health.

Over the past 10 years, the so-called gut microbiome has become one of the hottest areas of biomedical research. In this short time, scientists have discovered that some gut microbes produce chemicals that can influence brain structure, function and development, and the bacteria in your belly have been implicated in everything from anxiety to Parkinson’s and autism to multiple sclerosis.

The work has led experts to suggest what would, just a few years ago, have been thought of as impossible and ridiculous: that dementia and mental illness could one day be treated with “psychobiotics”, or intestinal bacteria with mental health benefits, and that conditions such as obesity and irritable bowel syndrome could be cured with faecal transplants. Gut microbes are now being touted as the new frontier of personalised medicine. They are the subject of dozens of TED talks claiming that they will eventually provide the solution to all our health problems, and that they even make us who we are.

Should we believe the hype, or are our gut microbes being oversold? The gut microbiome does indeed show huge potential, but the research is still in its infancy, and many of the big questions remain unanswered. One of the great unknowns is exactly how gut microbes communicate with the brain. We are beginning to unravel these lines of communication, to reveal several possible candidates. One potential route is via the vagus nerve, which branches into the gut and sends sensory nerve endings into its lining. Neuromodulatory chemicals synthesised and released by gut bacteria can signal to the brain by influencing the activity of these nerves, and the brain could signal to gut bacteria by the same route in the opposite direction.

One of the great unknowns is exactly how gut microbes communicate with the brain

Gut bacteria also release chemicals that can act on the hormone-producing cells and glands in the gut, causing them to secrete certain peptides, which may then reach the brain and influence its workings, via the bloodstream. They may also influence brain function by releasing metabolites that activate the immune system. Likewise, our understanding of the diversity of the gut’s microbial community, and of how and exactly why the composition of the gut microbiome differs between individuals, is lacking.

We have made many incremental advances, discovering that certain strains or species synthesise neuromodulatory chemicals, but the big picture is as yet far from complete. We are also beginning to understand the factors that influence the initial process by which bacteria colonise the gut in the days following birth. We know, for instance, that a child born by caesarean section will have a different composition of gut microbes to one born vaginally. But exactly how life events and experiences alter the composition of the gut microbiome, and how these events and changes might have a life-long impact on health and behaviour, remains unclear.

Learning more about the gut microbiome could eventually lead to other, novel forms of treatment. We have known for a long time that gut bacteria play a vital role in digesting the food we eat, and it is becoming increasingly clear that our diet can alter gut microbe composition in significant ways. But how, exactly, does our diet affect our gut bacteria and how, in turn, might that influence our thoughts, mood, and behaviour? A clearer understanding of this could eventually lead to what some have called nutritional psychiatry, or eating foods that prevent or treat mental illness.

A number of clinical trials testing the efficacy of faecal transplants for obesity and irritable bowel syndrome have already been conducted, but the results so far have been mixed. The research into the role of gut microbes in anxiety and mood disorders has, however, been done almost exclusively in rodents, and is being translated to humans very slowly. So randomised, double-blind, placebo-controlled clinical trials in humans are urgently needed.

The research done to date provides proof of concept that both mental and physical health conditions could be treated or at least alleviated by manipulating our gut microbes in one way or another. But developing effective treatments will not be possible without first gaining a far better understanding of the bacterial community in our guts, the chemicals and metabolites they produce, and the mechanisms underlying the beneficial effects of specific bacterial strains.

As we begin to answer these questions we could eventually tailor probiotic, nutritional, and faecal transplant treatments to each individual’s specific needs. Science is a time-consuming process, and it often takes decades for major discoveries to translate into treatments; so despite the many important advances in gut microbiome research, don’t hold your breath for that probiotic that will lift your mood.

• Mo Costandi trained as a developmental neurobiologist and is a freelance science writer. He is the author of Neuroplasticity and 50 Human Brain Ideas You Really Need to Know