Effects of oral hormonal contraceptives (OC) on human brain structure and behavior have only recently become a focus of research. Two explorative reports observed larger regional gray matter (GM) volumes in OC users within the prefrontal cortex, ACC and fusiform gyri, as well as parahippocampal gyri, hippocampus and cerebellum. These studies did however not control for the androgenicity of the progestin compound of OC, did not take into consideration how long OC users had been on their OC, and did not control for age differences between the OC group and the naturally cycling group. We compared 20 naturally cycling women during their early follicular cycle phase to 18 users of OC containing androgenic progestins and 22 users of OC containing anti-androgenic progestins. When controlling for age, we found that in users of anti-androgenic progestins relative GM volumes within the bilateral fusiform gyri, fusiform face area (FFA), parahippocampal place area (PPA) and cerebellum, were significantly larger than in naturally cycling women, while in users of androgenic progestins, relative as well as absolute volumes within the bilateral middle and superior frontal gyri were significantly smaller compared to naturally cycling women. These morphological changes were related to performance in a face recognition task. Face recognition performance was significantly better in users of anti-androgenic progestins compared to the other groups and significantly related to absolute as well as relative GM volumes in the FFA and PPA. Total GM volume, as well as absolute GM volumes within the bilateral fusiform gyri, FFA, hippocampus, parahippocampus, PPA, middle frontal gyri and ACC were significantly larger, the longer the duration of OC use, particularly in users of androgenic progestins. Morphological differences between active and inactive pill phase were observed in users of androgenic progestins. These findings suggest differential effects of androgenic and anti-androgenic progestins on human brain structure.