(CNN) Listen up, gen-Xers and millennials, and, well, everybody who has a brain. If you live a high-stress life, you could have memory loss and brain shrinkage before you turn 50, according to a study published Wednesday in the journal Neurology.

"Higher levels of cortisol, a stress hormone, seem to predict brain function, brain size and performance on cognitive tests," said study author Dr. Sudha Seshadri, professor of neurology at UT Health San Antonio.

"We found memory loss and brain shrinkage in relatively young people long before any symptoms could be seen," Seshadri said. "It's never too early to be mindful of reducing stress."

Too much 'flight or fight'

Cortisol is one of the body's key stress hormones, best known for its role in our "flight or fight" instincts. When we are stressed and on high alert, the adrenal glands produce more cortisol. The hormone then goes to work shutting down various bodily functions that might get in the way of survival.

Once the crisis has passed, cortisol levels should drop, and the body systems should return to normal. But if your alarm button stays pressed, the body can continue to malfunction, leading to anxiety, depression, heart disease, headaches, weight gain, trouble sleeping, and, of course, memory and concentration problems.

The brain is especially vulnerable, say experts, because of all of the nutrients it needs to function optimally.

"The brain is a very hungry organ," said Keith Fargo, who directs scientific programs and outreach for the Alzheimer's Association. "It requires an outsized amount of nutrients and oxygen to keep it healthy and functioning properly. So, when the body needs those resources to deal with stress, there's less to go around to the brain."

High stress connected to memory loss

Previous studies have found a relationship between cortisol and dementia risk, but the focus has been mostly on the elderly and the memory area of the brain, called the hippocampus.

Among the new study's strengths, said Seshadri, is that it looked at a group of men and women with an average age of 48 and did MRI brain scans of the entire brain, not just the hippocampus.

Researchers chose more than 2,000 people with no signs of dementia and gave them various psychological exams to measure their thinking skills.

All were part of the Framingham Heart Study , a long-term study sponsored by the National Heart, Lung and Blood Institute. It's been following the health of residents of Framingham, Massachusetts, and their offspring since 1948.

About eight years after initial testing, the group was reevaluated. Blood serum cortisol was measured before breakfast. Then brain MRIs were done and the series of memory and cognitive tests repeated.

After adjusting the data to consider age, sex, body mass and smoking, the study found people with the highest levels of cortisol had the most memory loss.

"I wasn't surprised by the changes in cognition," said Fargo who was not involved in the study. "If you have higher cortisol you're probably stressed out and likely to have more difficulty on cognitive tasks."

Stress affects brain structure, too

What was surprising, said Fargo, was the study's findings of the effects of cortisol on brain structure.

JUST WATCHED Why we need to work less, rest more Replay More Videos ... MUST WATCH Why we need to work less, rest more 01:07

High levels of cortisol were associated with more damage to the parts of the brain that move information throughout the brain (corona radiata) and between the two hemispheres of the brain (corpus callosum).

In addition, the brains of people with higher cortisol levels had smaller cerebrums, the two hemispheres of the brain responsible for thought, emotions, speech and muscle functions, the study found.

The average total cerebral brain volume in people with high levels of cortisol was 88.5 of the total brain volume, compared to 88.7 in people with normal levels of cortisol.

"I was surprised you would be able to see such a large change in brain structure with high cortisol levels compared to moderate levels of cortisol," said Fargo. "If you're seeing structural brain changes in midlife, you can imagine what is happening by the time you get old enough to develop dementia."

Interestingly, the effects of high cortisol on cerebral brain volume appeared to only affect women, not men.

"Estrogen can increase cortisol," said Dr. Richard Isaacson, who directs the Alzheimer's Prevention Clinic at Weill Cornell Medicine, "and about 40% of the women in the study's high cortisol group were on hormone replacement." Isaacson was not involved in the study.

Get CNN Health's weekly newsletter Sign up here to get The Results Are In with Dr. Sanjay Gupta every Tuesday from the CNN Health team.

Seshadri said the study did adjust for the use of hormone replacement therapy.

"That does not completely rule out an adverse impact of estrogen replacement," Seshadri said, "but makes it less likely to the main story."

Seshadri also emphasized the study results only show an association, not a cause, and more research is needed to tease out the relationship between high cortisol levels and dementia. While that is occurring, she suggests people consider lifestyle modifications to combat the stress of modern life.