Hourly shift work has been linked to all kinds of ills, from obesity to heart attack, and now a new study shows it might also have serious implications for your brain.

The study in Occupational & Environmental Medicine looked at more than 3,000 people living in France, about half of whom had experience working shifts. Those who had done so, either in the past or present, had lower scores on tests of memory, processing speed and overall brain power than those who worked normal office hours, the study finds.

These effects persisted even after researchers controlled for effects of sleep deprivation and they got even stronger after people had worked nights for 10 or more years. Those long-term shift workers had worse memory than those who had always worked days, plus cognitive deficits so steep that the study authors equated them to 6.5 years of age-related decline.

There is a bit of good news, though. After stopping shift work for five years, cognitive abilities returned to levels of people who had never worked shifts.

So why does shift work appear to be so bad for the brain? The authors stress that the study is observational, so it can’t determine that shift work causes brain decline. But they do have a favorite theory: “If it’s not sleep,” says Dr. Philip Tucker, study co-author and senior lecturer in the psychology department at Swansea University in the U.K., “the strongest candidate would be destruction of circadian rhythms.”

Working the night shift challenges the body’s natural circadian clock, which is linked to all sorts of health problems. Though the study didn’t look at the brain structures of the participants, a small study in 2001 found that flight attendants who were chronically jet lagged actually had smaller temporal lobes.

Shift workers, who sleep during the day, may also have a vitamin D deficiency, the study authors say, or may be more prone to metabolic disorder—but the disruption of circadian rhythms is still the main contender.“You could argue,” says Tucker, “that if you start messing about with those clocks, there’s going to be all sorts of effects.”

Read next: The 12 Worst Habits for Your Mental Health

Get our Health Newsletter. Sign up to receive the latest health and science news, plus answers to wellness questions and expert tips. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Mandy Oaklander at mandy.oaklander@time.com.