This relationship held even after the authors controlled for things like mental and physical health, cognitive ability, and even their socioeconomic status in childhood, rather than adulthood. That is, growing up rich or poor didn’t necessarily affect the brain health of the middle-aged people. But it seems something about their lives in adulthood did.

What could those things be? People who had lower-paying jobs might have had worse access to health care and healthy food. They might live in more polluted neighborhoods or have lives that are less intellectually stimulating. The stress of being low on the socioeconomic totem pole raises levels of allostatic load—a measure of stress hormones that cause wear and tear on the body, including the brain.

“We’re starting to learn more about the impact of both stress and continuous learning on the brain,” says Gagan Wig, a neuroscientist at UTD and the principal investigator of the study. “It’s consistent with the idea that lifelong experiences might influence brain health.”

Past studies have also suggested that being low in socioeconomic status can affect the way we think. A paper in Science in 2013 found that “a person’s cognitive function is diminished by the constant and all-consuming effort of coping with the immediate effects of having little money, such as scrounging to pay bills and cut costs.” The cognitive cost of poverty, that study found, was practically like losing an entire night of sleep. Another study from last year found that people who had lived in poverty performed worse than those who had never been poor on tests of verbal memory, processing speed, and executive functioning.

“I think this [PNAS paper] builds on the past work on cognitive function and poverty,” says Jiaying Zhao, a psychologist at the University of British Columbia and a co-author of the 2013 study. “This shows how chronic poverty can influence brain anatomy. This paper provides the neural evidence.”

In the PNAS study, the relationship between socioeconomic status and the brain measures didn’t hold up for the very youngest adults (those aged 20 to 34) or the very oldest adults (those over 64) in the sample. It could be that only the healthiest people among the low-socioeconomic-status groups survived to old age, or that by the time people reach their 70s and 80s, social and economic determinants are less important to their brain health than is the biological aging process. Also, this study included very few people below the poverty line, which is the condition the earlier papers were largely studying.

Still, together, this line of research suggests that being poor (or at least not rich) might be at least in part a self-perpetuating cycle. The people who are less wealthy struggle to cover their expenses. They get stressed, remember worse, and thus perform worse at the very same cognitive tasks that tend to increase wealth in today’s information economy.

Or, as Zhao put it in 2013, “Previous views of poverty have blamed poverty on personal failings, or an environment that is not conducive to success ... We’re arguing that the lack of financial resources itself can lead to impaired cognitive function. The very condition of not having enough can actually be a cause of poverty.”