When a researcher presented his latest findings on the effects of birth control pills on the brain at a conference, a somewhat predictable chain of events followed. Though Michael Lipton, the study’s author, emphasized that his research had only established an association between the size of the hypothalamus and oral contraceptive use—rather than a definitive cause-and-effect relationship—within days, several outlets had run stories claiming that birth control could “shrink” parts of the brain.

Lipton doesn’t usually study birth control—he’s a neuroscientist. But when he discovered a significant difference in the way women and men respond to traumatic brain injuries, he wanted to investigate how sex hormones affect the brain. For the study, Lipton recruited 50 women, 21 of whom were already taking oral contraceptives. After giving them all MRIs of the brain, he measured their hypothalamuses, a brain region that helps regulate hormones, and found that the women who had been taking birth control pills had significantly smaller hypothalamuses than those who didn’t.

Still, in order to establish that taking oral contraceptives causes the hypothalamus to lose volume, Lipton would have to conduct additional studies with people who hadn’t used oral contraceptives before, assess their hypothalamus volume, have them take the pills, and then assess their hypothalamuses again before reaching any conclusions.

“We haven’t demonstrated that this is clearly a causal relationship, and to do so would require a very different study,” said Lipton, who is also the director of radiology research at the Albert Einstein College of Medicine in New York City. “It’s a leap to say birth control ‘shrinks’ your brain.”

Some might feel panic at seeing a headline about their brains “shrinking,” but the study found no correlation between hypothalamus size and cognitive performance. And while it did find that having a smaller hypothalamus can be associated with feelings of depression and anger, Lipton did not determine that birth control caused either of those things.

While Lipton's research is under review for publication in an academic journal, he still doesn’t recommend making any medical decisions based on his findings—he said there’s no reason to believe a smaller hypothalamus would make a meaningful difference in a person’s life.

Lipton’s findings are consistent with what we know about birth control’s effects on the brain: Over the last decade, research has shown that oral contraceptives can alter people’s brain structure, making some regions bigger or smaller. A 2010 study found that people who take oral contraceptives tend to have more gray matter—brain tissue that helps with muscle control and sensory perception—in several parts of the brain, including the prefrontal cortex and the hippocampus. A later study, conducted in 2013, found that hormonal birth control had the potential both to increase and decrease the volume of different brain regions.

Experts say it makes perfect sense that hormonal birth control could alter the structure of people’s brains: If your body is producing less of a certain hormone, your brain will reduce the number of receptors for that hormone, which could in turn reduce the brain volume in that region.

“If you’re keeping women in a constant state of low estrogen”—hormonal birth control prevents estrogen levels from spiking, which causes ovulation—”there’s no way that won’t correspond to structural changes in the brain,” said Sarah Hill, a research psychologist and the author of This Is Your Brain on Birth Control. “But that doesn’t mean it’s bad. The brain is a very dynamic system, and it’s changing all of the time.”

Still, it remains the case that scientists know very little about the relationship between hormonal birth control and the brain. Lipton said his recent study is the only study there is on the hypothalamus, which he called the most “obvious suspect” for scientists looking to explore this relationship.

And out of that dearth of information about the subtler effects of hormonal birth control has emerged a growing anxiety about how the pill could be shaping people’s moods, behaviors, and relationships to other people.

“Some women who’ve quit the Pill told me they’ve seen a reduction in anxiety or depression; some have seen an increased libido,” Anna Silman wrote for The Cut earlier this year. “For others, the effects were more nebulous: I heard phrases like ‘more alive,’ ‘more clear,’ ‘more myself,’ and “like a fog had lifted. The more I spoke to people, the more obsessed I became with the idea that one little tweak to my hormonal network could be affecting everything I take for granted about myself.”

There’s also worry that scientists aren’t interested in finding out more about the pill. That may be true to some extent, Lipton said: “There isn’t a huge motivation to go looking for any untoward effects [of hormonal birth control],” he said. “There was the sense that it’s a safe drug that has been around for a long time and that already has other well-known risks like blood clots and stroke.”

As scientists uncover more about the relationship between birth control and the brain, it’s possible that little of it will have any bearing on people’s practical decisions—if having a smaller hypothalamus doesn’t cause any noticeable changes, it’s likely the vast majority of people will continue taking the pill. Still, Lipton believes it’s a worthy line of inquiry, since understanding more about how the brain responds to sex hormones has implications beyond birth control. In his case, it will help him better understand traumatic brain injuries, and could lead to doctors learning how to treat them more effectively.

With birth control agnosticism already in the air, stories about negative-sounding findings catch on quickly, perhaps especially among those who have experienced bad side effects from the pill, or who have had doctors who didn’t listen to them. When faced with new information about hormonal contraceptives, Hill said she recommends people check in with themselves first before jumping to any conclusions.

“We’re in a cultural place where we can start having open, honest, and even critical conversations about the birth control pill, but that doesn’t mean we need to abandon it,” Hill said. “It’s important to take research seriously and try to understand all the different ways the pill can affect women, but I don’t think we should let it overwhelm us: If you feel good and things are going well in your life, then your birth control pill is probably working for you.”

Sign up for our newsletter to get the best of VICE delivered to your inbox daily.