If you thought smoking a joint occasionally was OK, a new study released Tuesday suggests you might want to reconsider.

The study, published in the Journal of Neuroscience, is the first to link casual marijuana use to major changes in the brain. And according to the researchers, the degree of abnormalities is based on the number of joints you smoke in a week.

Using different types of neuroimaging, researchers examined the brains of 40 young adults between the ages of 18 and 25 who were enrolled in Boston-area colleges. Twenty of them smoked marijuana at least once a week. The other 20 did not use pot at all.

The marijuana smokers were asked to track their cannabis use for 90 days. All were given high-resolution MRIs, and users and non-users' results were compared.

Researchers examined regions of the brain involved in emotional processing, motivation and reward, called the nucleus accumbens and the amygdala. They analyzed volume, shape and density of grey matter - where most cells in brain tissue are located.

"I think the findings that there are observable differences in brain structure with marijuana even in these young adult recreational users indicate that there are significant effects of marijuana on the brain," says Dr. Jodi Gilman, lead author and a researcher in the Massachusetts General Center for Addiction Medicine. "Those differences were exposure-dependent, meaning those who used more marijuana had greater abnormalities."

More than a third of the group - seven of the 20 - only used pot recreationally once or twice a week. The median use was six joints a week, but there were four people who said they smoked more than 20 joints a week. None of the users reported any problems with school, work, legal issues, parents or relationships, according to Dr. Hans Breiter, co-senior author of the study and a professor of psychiatry and behavioral sciences at Northwestern University Feinberg School of Medicine.

"There's a general idea out there that casual use of marijuana does not lead to bad effects, so we started out to investigate that very directly," Breiter said. "This research with the other studies we have done have led me to be extremely concerned about the effects of marijuana in adolescents and young adults and to consider that we may need to be very careful about legalization policies and possibly consider how to prevent anyone under age 25 to 30 from using marijuana at all."

Researchers have long been concerned about the effects of marijuana on the developing brain - teens and adolescents under the age of 25. Preliminary research has shown that early onset smokers are slower at tasks, have lower IQs later in life and even have a higher risk of stroke.

Dr. Staci Gruber, director of the Cognitive and Clinical Neuroimaging Core at McLean Hospital in Boston and a professor of psychiatry at Harvard Medical School, has conducted numerous studies on marijuana use and brain function.

"This data certainly confirms what others have reported with regard to changes in brain structure," she said. "When we consider the findings of the Gilman ... study with our own and other investigations of marijuana use, it's clear that further investigation is warranted, specifically for individuals in emerging adulthood, as exposure during a period of developmental vulnerability may result in neurophysiologic changes which may have long-term implications."

Gruber says we need to take a closer look at all pot users whether they smoke once or twice a week or four or time times a week.

And she had this advice for adolescents: "Don't do it early–prior to age 16. That's what our data suggests, that regular use of marijuana prior to age 16 is associated with greater difficulty of tasks requiring judgment, planning and inhibitory function as well as changes in brain function and white matter microstructure relative to those who start later."

According to the National Survey on Drug Use and Health, in 2012 nearly 19 million Americans used marijuana. It's the most-used illegal drug in the country and use is increasing among teenagers and young adults.

Results of the new study match those of animal studies, authors say, showing that when rats are given tetrahydrocannabinol, or THC - the ingredient in marijuana the gets you high - their brains rewire and new connections are formed.

Gilman thinks when people start to become addicted to substances, their brains form these new connections too.

"The next important thing to investigate is how these structural abnormalities relate to functional outcomes," she said. "Currently we don't know how much marijuana is safe and I think this study shows that we should be cautious about marijuana use in adolescents and young adults whose developing brain may be even more susceptible to cannabis-induced changes."