Recent studies suggest that being overweight or obese is related to worse cognitive performance, particularly executive function. Obesity may also increase the risk of Alzheimer's disease. Consequently, there has been increasing interest in whether adiposity is related to gray or white matter (GM, WM) atrophy. In this review, we identified and critically evaluated studies assessing obesity and GM or WM volumes either globally or in specific regions of interest (ROIs). Across all ages, higher adiposity was consistently associated with frontal GM atrophy, particularly in prefrontal cortex. In children and adults <40 years of age, most studies found no relationship between adiposity and occipital or parietal GM volumes, whereas findings for temporal lobe were mixed. In middle-aged and aged adults, a majority of studies found that higher adiposity is associated with parietal and temporal GM atrophy, whereas results for precuneus, posterior cingulate, and hippocampus were mixed. Higher adiposity had no clear association with global or regional WM in any age group. We conclude that higher adiposity may be associated with frontal GM atrophy across all ages and parietal and temporal GM atrophy in middle and old age.