In this brief review we summarize the promising effects of physical activity and fitness on brain and cognition in children and older adults. Research in children finds that higher fit and more active preadolescent children show greater hippocampal and basal ganglia volume, greater white matter integrity, elevated and more efficient patterns of brain activity, and superior cognitive performance and scholastic achievement. Higher fit and more physically active older adults show greater hippocampal, prefrontal cortex, and basal ganglia volume, greater functional brain connectivity, greater white matter integrity, more efficient brain activity, and superior executive and memory function. Despite these promising results, more randomized trials are needed to understand heterogeneity in response to physical activity, mechanisms, and translation to public policy.