Most likely, it’s both.

A paper recently published in the Journal of Health and Social Behavior examines how these two factors work together and how they split along gender lines. Researchers tracked the long-term impact of obesity by looking at data from the Wisconsin Longitudinal Study, a large-scale project that collected health and social information from more than 10,000 people born in 1939 at multiple points in their lives. Consistent with the findings of the CDC, girls from poorer families tended to have higher body mass at age 18 than their richer female peers, a pattern that didn’t exist for boys. But the researchers also found that among the overweight teens of both sexes, the girls—and not the boys—tended to perform worse in school than their thinner classmates.

Later, at age 54, the subjects demonstrated two similar patterns. First, the women who had come from poorer families as children were more likely to be obese adults; second, women with higher body-mass index were more likely to be poor, regardless of their childhood status. As in adolescence, the effects were less pronounced and less consistent among the men of the study.

These discrepancies, explains study author Tetyana Pudrovska, may be rooted in our notions of what attractiveness looks like: For women, specifically, it looks thin.

“[Thinness] is more closely tied to physical attraction in our culture for women than for men,” she says. “Obesity violates cultural standards of female beauty.” As a result, the stigma of obesity is lopsided across gender lines, with women bearing the brunt of it.

The higher social price begins to take its economic toll as early as adolescence, she says, as the stigma of obesity can lower teen girls’ self-esteem—which, in turn, can lead to a host of problems that impede academic success. “Girls who are obese, they’re less likely to go to college and less likely to graduate from college,” Pudrova explains. “And one of the reasons is that they have higher levels of substance abuse in high school. They have higher levels of truancy.” Among the subjects of the Wisconsin Longitudinal Study, their grades also tended to be lower.

The same forces persist in the workplace as well. Already at an educational disadvantage, obese women face rough treatment at the hands of the labor market, where attractiveness is often rewarded: Women with higher body-mass index, earn less, on average, than their thinner counterparts, and have fewer opportunities for advancement. Wages for women tend to peak among those with a BMI of 21.8 (a healthy weight is considered to be a BMI between 18.5 and 24.9); for men, by contrast, the wage-weight connection punishes only those at the highest end of the BMI scale.

As Olga Khazan reported earlier this year, obesity is expensive. Obese men lose an average of $2,646 each year to things like additional insurance costs, sick leave, disability, and lost productivity. For women, that figure is $4,879. The difference? Lower wages.