(CNN) Drinking in moderation can help our health , some research has showed. Many doctors recommend a glass of wine or beer a night as part of diet plans such as the Mediterranean diet and the DASH diet , which have been proven to keep your heart and brain healthy. However, a new study suggests that even moderate drinking may not be great for your brain.

As part of the study, which was published Wednesday in the BMJ , researchers looked at people's weekly alcohol intake from the Whitehall II study , which tracks disease and social behaviors in a group of British civil servants for 30 years. University of Oxford and University College London scientists studied how participants fared with regular brain function tests and an MRI.

What they noted was that the people who drank the most had the highest risk of hippocampal atrophy , a form of brain damage that can impact spatial navigation and can be associated with memory-loss conditions like Alzheimer's and dementia. The heavier drinkers saw a faster decline in language skills and had poorer white matter integrity , which is crucial to processing thoughts quickly.

If you're starting to worry and are afraid to drown your sorrows, note that there are many caveats, and more research needs to be done. Some experts suggest you shouldn't change your drinking behavior based on this one study, but the results of these brain scans and memory tests for moderate and lighter drinkers were not what researchers expected.

"We were surprised that the light to moderate drinkers didn't seem to have that protective effect," said study co-author Dr. Anya Topiwala , a clinical lecturer in the Department of Psychiatry at the University of Oxford. "These are people who are drinking at levels that many consider social drinkers, so they are not consuming a lot."

Even the heaviest of the drinkers aren't big nightly bingers. The "drank the most" group in this study consumed about 30 units of alcohol a week, with a unit considered to be 10 milliliters or 8 grams of pure alcohol. A medium glass of wine has about two units of alcohol, and so does a pint of some beers, depending on the alcohol content.

If you do the boozy math, the study's heaviest drinkers had a little more than two medium glasses of wine or two beers every night of the week.

The moderate group was drinking about 14 to 21 units of alcohol per week, or about a medium glass of wine each night, plus a little extra on the weekends.

Researchers discovered that the moderate group was three times more likely to have hippocampal atrophy compared with people who didn't drink at all. However, in the heavy and moderate drinkers, there is no evidence to show how clinically significant this change is, and there is no evidence linking this loss to any negative general cognitive effects, even the ones for which the participants were tested.

With the light drinkers, those who had a small glass of wine a night or up to seven units per week, researchers didn't see a significant difference compared with the abstainers, but they didn't see any protective qualities, either.

The abstainers may be one subject that needs further development, according to Eric Rimm , a professor of medicine and director for the program in cardiovascular epidemiology at Harvard T.H. Chan School of Public Health.

Rimm, who was not involved in the new study, has researched the impact of alcohol for years. He said that although the study has an interesting hypothesis, the abstainer group (22 men and 15 women) is tiny and may be throwing off the results.

If you are a moderate drinker, he said, you don't have to give up the booze based solely on this report.

"There are so many other lifestyle factors that are not taken into account in this study, like nutrition. Eating whole grains and fruits and vegetables have been linked with slower cognitive decline," Rimm said. Attributing mental decline to alcohol is too limited, he said.

Tom Dening, a professor of dementia research and Director of the Centre for Old Age and Dementia at the University of Nottingham, called the study "most impressive" and suggests it may be a good reminder that "perhaps we should all drink a bit less," but he also questioned its results. People typically are not honest about how much they really drink, he noted.

"People tend to underestimate their actual consumption, partly to appear more respectable," Dening said. "If actual consumption was under-reported, then the apparent adverse effects of modest amounts of alcohol could have been magnified."

Earlier studies have also shown that people tend to drink less as they age, said Carl Heneghan , director of the Centre for Evidence-Based Medicine at the University of Oxford, who was not involved in the new research.

"These type of studies also cannot account for all the (factors), and therefore they cannot, and should not, conclude causation," Heneghan said. "Using all the available evidence provides a much more balanced approach for the public on deciding how much to drink."

Rimm wondered whether there were so few abstainers in the study because, after 30 years, they hadn't lived as long as the others.

When it comes to the brain, this is not the first research to question whether drinking even moderate amounts had an impact, though results have been mixed. One 2008 study found that unlike with the heart, for which alcohol has protective qualities, the more people drank, the smaller their total brain volume, meaning their brains aged faster.

Join the conversation See the latest news and share your comments with CNN Health on Facebook and Twitter.

"An observational study cannot truly prove that alcohol causes dementia, but the findings are in keeping with my clinical experience," said Dr. Elizabeth Coulthard , a consultant senior lecturer in Dementia Neurology at the University of Bristol who was not involved in the new research. "Hopefully this research will contribute to a greater understanding of true safe limits for alcohol consumption that ensure protection from future dementia. Until we have further studies, the good news is that low alcohol intake was not associated with brain or memory decline in this sample."

Topiwala said she and her co-authors want to try to replicate the results, particularly with a more diverse population. Most in this study were men with similar backgrounds, with generally higher IQs than the population as a whole, all factors that can affect the results.

"We think, though, this study asks some really important questions in an area that does have a knowledge gap," Topiwala said.