Previous research in animal models suggests that brain-derived neurotrophic factor (BDNF) is involved in stress-modulated alcohol consumption. However, relatively few studies have investigated this issue in humans, and results of existing studies have been heterogeneous. The primary aim of the current study was to examine the within-subjects effect of acute stress (timed math plus cold pressor) on serum BDNF levels (ΔBDNF: post- minus pre-stress) in healthy social drinkers (N = 68, 20 male). A secondary aim was to explore which heritable and environmental factors in our limited sample might exert the greatest influences on ΔBDNF. Importantly, presence versus absence of the BDNF Val66Met polymorphism (rs6265), which has often been discounted in studies of human serum BDNF, was included as a between-subjects control variable in all statistical analyses. Our results indicated that acute stress decreased serum BDNF. Further, multiple regression analyses revealed that quantitative family history of alcohol use disorder (qFH) and age at first alcohol use together accounted for 15% of the variance in ΔBDNF. Thus, the influences of qFH and age at first alcohol use may explain some of the heterogeneity that exists in previous studies of human serum BDNF. These results parallel findings in animal models and suggest that stress-related changes in serum BDNF are influenced by both heritable (qFH) and environmental (early alcohol consumption) factors.