Illustration by Dale Edwin Murry

In April 2011, Robert Froemke and his team were reprogramming the brains of virgin mice with a single hormone injection.

Before the treatment, the female mice were largely indifferent to the cries of a distressed baby, and were even known to trample over them. But after an injection of oxytocin, the mice started to respond more like mothers, picking up the mewling pup in their mouths. Froemke, a neuroscientist at New York University's Langone Medical Center in New York City, was monitoring the animals' brains to find out why that happened.

At first, the mice showed an irregular smattering of neural impulses when they heard the baby's cries. Then, as the oxytocin kicked in, the signal evolved into a more orderly pattern typical of a maternal brain. The study showed in unusual detail how the hormone changed the behaviour of neurons1. “Oxytocin is helping to transform the brain, to make it respond to those pup calls,” Froemke says.

Oxytocin has been of keen interest to neuroscientists since the 1970s, when studies started to show that it could drive maternal behaviour and social attachment in various species. Its involvement in a range of social behaviours2, including monogamy in voles, mother–infant bonding in sheep, and even trust between humans, has earned it a reputation as the 'hug hormone'. “People just concluded it was a bonding molecule, a cuddling hormone, and that's the pervasive view in the popular press,” says Larry Young, a neuroscientist at Emory University in Atlanta, Georgia, who has been studying the molecule since the 1990s.

“What we need to start thinking about is the more fundamental role that oxytocin has in the brain.”

That view has led some clinicians to try oxytocin as a treatment for psychiatric conditions such as autism spectrum disorder. But the early trials have had mixed results, and scientists are now seeking a deeper understanding of oxytocin and how it works in the brain. Researchers such as Froemke are showing that the hormone boosts neuronal signals in a way that could accentuate socially relevant input such as distress calls or possibly facial expressions. And clinical researchers are starting a wave of more ambitious trials to test whether oxytocin can help some types of autism.

The work is leading to a more sophisticated view of the hormone and its complex effects on behaviour — one that will take many types of expertise to refine. “The oxytocin field has just matured and ripened enough to draw in researchers from traditionally separate fields, catapulting this forward,” says Young.

Birth accelerator

Oxytocin's story starts back in the early 1900s, when biochemists discovered that a substance from the posterior pituitary gland could promote labour contractions and lactation. When scientists later discovered the hormone responsible, they named it oxytocin after the Greek phrase meaning 'rapid birth'. Oxytocin is produced mainly by the brain's hypothalamus; in the 1970s, studies revealed that oxytocin-producing neurons send signals throughout the brain, suggesting that the hormone had a role in regulating behaviour.

In a landmark 1979 study3, Cort Pedersen and Arthur Prange at the University of North Carolina in Chapel Hill showed that giving oxytocin to virgin rats could trigger maternal behaviours: the animals would build nests, lick or crouch over unfamiliar pups and even return lost pups to the nest. Researchers went on to show that oxytocin signalling in the brains of prairie voles (Microtus ochrogaster) helps the animals to form lifelong pair bonds4 — a rarity among mammals. In 2012, researchers even found a version of oxytocin in the tiny roundworm Caenorhabditis elegans, where it helps the animals find and recognize mates5.

“This is a very ancient molecule,” says Sue Carter, a neuroscientist at Indiana University in Bloomington, whose lab pioneered many of the early studies of oxytocin in voles. “It has been used and reused for many purposes across the evolution of modern animals, and almost everybody who's tried to look at an effect of oxytocin on anything like social behaviour has found something.”

In mammals, many mysteries remain. Oxytocin is difficult to measure reliably in the brain, making it hard to know exactly where, when and how much is normally released; nor do scientists understand precisely how it works to alter behaviour. “What we need to start thinking about is the more fundamental role that oxytocin plays in the brain,” Young says. The determination to find out has been strengthened by a growing move in neuroscience to characterize circuits that are important in brain operations. “That's the level that's critical for understanding how the brain is regulating behaviour,” says Thomas Insel, director of the US National Institute of Mental Health in Bethesda, Maryland, who has studied oxytocin in voles.

“This is not a molecule that people should be self-administering or playing with.”

At Langone, Froemke focused on the circuits underlying the maternal response to pup cries — a behaviour that helps females to retrieve helpless newborns that can get lost when a mother is moving her nest. He focused on the left auditory cortex, a brain area thought to be involved in detecting the pups' ultrasonic cries.

Froemke's study1, published in April, showed that oxytocin temporarily suppresses inhibitory neurons — those that dampen neural activity — which allows excitatory cells to respond more strongly and reliably. “Our hypothesis is that the virgin brain is a blanket of inhibition, and that pairing the pup calls with oxytocin allows the network to be reconfigured,” says Froemke. The hormone may serve to amplify incoming signals and allow them to be recognized as behaviourally important. (It is at least possible, he says, that this same mechanism could explain why some human mothers feel they are uniquely tuned to a baby's cries.)

“The study is kind of a high-water mark for the field, putting different levels all together: a robust behaviour, a brain region, and a cellular basis for it,” says Richard Tsien, a neuroscientist also at Langone. Tsien has been studying the action of oxytocin on neuronal circuits in detail, by examining slices of the hippocampus, a region involved in learning and memory. In a 2013 study6 of rats, Tsien's team found that oxytocin selectively acts on a type of cell called an inhibitory interneuron in a way that quiets background chatter within the neuronal circuit. “Oxytocin improved signal transmission, almost doubling the ability of information to flow through the system,” Tsien says. In effect, it is producing more signal and less noise.

Froemke's and Tsien's work fits into a broader theory: that one way oxytocin helps social interaction and recognition is by enhancing the brain's response to socially relevant sights, sounds or other stimuli. Young has shown that the hormone helps mice to recognize and pay attention to the smells of other mice7; others found that it promotes people's ability to recognize faces8.

The hormone does not act alone. In 2013, neuroscientist Robert Malenka at Stanford University in California and his colleagues showed that oxytocin works together with the neurotransmitter serotonin to reduce the excitability of neurons in the nucleus accumbens9, a brain region involved in reward. This process seems to support the preference of mice to return to environments where they had rewarding social interactions with other animals. “Oxytocin is part of a system,” Carter says, “and it's not the only molecule that matters, but it's one that in some way is regulatory over a large number of other systems.”

Matter of trust

The rapid evolution in basic research has been accompanied by a boom in clinical interest. Oxytocin has been used since the 1950s to accelerate childbirth, so many researchers consider it relatively safe to use in experiments.

About ten years ago, psychology studies started to show that single doses of oxytocin, delivered through an intranasal spray, could promote various aspects of social behaviour in healthy adults. People who inhaled oxytocin before playing an investment game were more willing to entrust their money to a stranger than were placebo-treated players10. A dose of the hormone also increased the amount of time that people spent gazing at the eye region of faces11, and improved their ability to infer the emotional state of others from subtle expressions12.

The idea that oxytocin is central to social cognition made it an attractive candidate for treating psychiatric disorders, especially autism spectrum disorder. People with this condition, who often have problems with social interaction and communication, may not process social stimuli appropriately — and scientists theorized that oxytocin might reverse some of the symptoms. Beginning in 2010, results emerged that seemed to support this theory: researchers found that single puffs of oxytocin could temporarily improve measures of empathy and social cooperation in people with autism spectrum disorder.

“People got quite excited,” recalls clinical neuroscientist Evdokia Anagnostou, who co-directs the Autism Research Centre at Holland Bloorview Kids Rehabilitation Hospital in Toronto, Canada. But Anagnostou says that some preliminary steps were skipped over as researchers rushed to test oxytocin as a psychiatric drug. “To be honest, if we had done it properly, we wouldn't have done it the way we did. It went a little bit too fast,” she says. Because oxytocin had cleared the early, standard steps of drug development decades earlier, some researchers did not systematically test a range of doses to see whether they had differing psychological effects.

Many early studies of oxytocin for autism were limited because they assessed only a single dose and had relatively few participants, and later experiments with more doses failed to show the same promise. In 2010, clinical psychologist Adam Guastella at the University of Sydney in Australia studied 16 male adolescents with autism spectrum disorder, and found that one dose of oxytocin could improve their ability to gauge the emotions of others by looking at their eyes13. But when he tried giving twice-daily doses of the hormone for two months, he found no significant improvements in social interaction or social cognition14. “Studies to this point have really shown limited benefit of oxytocin in improving psychiatric illnesses over time,” he says. Guastella says that getting to the bottom of oxytocin's complex neurological effects will take time. “If we want a simple answer, we're not going to get it.”

All in the detail

So far, few studies have definitively linked autism to problems in oxytocin signalling. Some of the clearest evidence emerged in February, from a team led by neurogeneticist Daniel Geschwind of the University of California, Los Angeles. The group showed that mice that lacked a working copy of the Cntnap2 gene — which has been implicated in a small subset of human autism cases — had fewer oxytocin-containing neurons in the hypothalamus and socialized less with other mice than did control mice15. After receiving doses of oxytocin every day for two weeks, the mice behaved normally again. “Until this, there was no evidence that there was a subtype of autism that had to do with oxytocin deficits,” Geschwind says.

His study points to a more targeted approach in the clinic. “Autism is highly heterogeneous, but if you can find subsets of individuals — those who have oxytocin-signalling deficits — they may be the best candidates for oxytocin therapy,” says Karen Parker, a behavioural neuroscientist at Stanford.

A handful of large-scale clinical trials are now getting under way to test oxytocin and oxytocin-based therapies for autism spectrum disorder, and to work out who could benefit. Linmarie Sikich, a child psychiatrist at the University of North Carolina is heading the largest of these trials. Sikich plans to recruit 300 people with autism spectrum disorder, ranging in age from 3 to 17, and give them 6 months of either oxytocin or a placebo, followed by 6 months in which everyone will receive oxytocin.

Unlike previous studies, the trial will include people with a wide range of symptoms — and one of its major aims is to uncover the set of factors that influence whether and how strongly people respond to oxytocin. Sikich will analyse many measures of cognition and social functioning, and collect blood samples to look for biomarkers — such as levels of oxytocin and the receptor it binds to — that are associated with a response. “Lin has really been trying to create conditions under which you could study the potential beneficial effects of oxytocin and really do this right,” says Carter.

But Carter and other scientists are concerned by reports from the physicians and parents of children with autism spectrum disorder who say that they are already using oxytocin off-label — before it has been thoroughly tested. “We do not understand how the hormone works yet, or have enough information about what happens when it's given repeatedly,” Carter says. “This is not a molecule that people should be self-administering or playing with.”

Some work has pointed to a potential dark side to oxytocin. Carter's group found that a single low dose of the hormone given to baby prairie voles improved their pair bonding as adults, but that higher doses interfered with that behaviour — possibly because oxytocin started to activate other receptors16. And human studies have suggested that in certain contexts, a puff of oxytocin can cause people to be more aggressive in defending themselves against outsiders or competitors17. In patients with a psychiatric condition known as borderline personality disorder, a single dose of oxytocin has been found to hinder trust and cooperation18.

Young says that the oxytocin field would benefit from closer collaboration between basic and clinical researchers. If basic scientists can work out how oxytocin helps the brain to process social stimuli, then that might help in the design of stimuli — in the form of behavioural therapies — that could be given alongside the hormone to change behaviour, just as oxytocin and pup calls together affect virgin mice. “I think in the future these two branches need to have more communication,” Young says.

But long before that, say researchers, oxytocin could use a rebranding. “It doesn't induce love; it doesn't induce massive amounts of trust,” Guastella says. “The problem we've got ourselves into is that we're trying to look for a simple answer: either oxytocin does or does not work in a patient population, or it does or does not enhance a certain social process.”

But the science of life is rarely as simple as that. “Oxytocin is known to affect circuits in different ways, and it's not going to affect everyone in the same way,” Guastella says. “The sorts of biology we're studying here are incredibly complex.”