Background and Aims Chronic alcohol use is associated with lower gray matter volume, and we reported recently that alcohol use showed negative associations with widespread gray matter (GM) volume even among young adults. The current study aimed to test the strength of association between (1) alcohol use and GM volume; (2) alcohol use and white matter (WM) integrity; (3) cannabis use and GM volume; and (4) cannabis use and WM integrity among adults and adolescents.

Design and Setting General linear models within large pooled cross‐sectional samples of adolescents and adults who had participated in studies collecting substance use and neuroimaging data in the southwestern United States.

Participants The current analysis included adults aged 18–55 years (n = 853) and adolescents aged 14–18 years (n = 439) with a range of alcohol and cannabis use.

Measurements The dependent variable was GM volume or WM integrity, with key predictors of alcohol use [Alcohol Use Disorders Identification Test (AUDIT) score] and cannabis use (past 30‐day use).

Findings Alcohol use showed large clusters of negative associations (η p 2 = 0.028–0.145, P < 0.001) with GM volume among adults and to a lesser extent (one cluster; η p 2 = 0.070, P < 0.05) among adolescents. Large clusters showed significant associations (η p 2 = 0.050–0.124, P < 0.001) of higher alcohol use with poorer WM integrity, whereas adolescents showed no significant associations between alcohol use and WM. No associations were observed between structural measures and past 30‐day cannabis use in adults or adolescents.