There are many reports on a positive relationship between tallness and socioeconomic success, and between tallness and health in the human male. Accordingly, tallness is an explanatory variable in many studies on health or behavior. Recently, a positive correlation of tallness with fitness has been reported. However, whether this fitness advantage is the effect of the socioeconomic success of tall men (making them good providers) or of body height itself (tallness being directly associated with some genes, i.e., not requiring the father's presence, that are favorable for offspring number or survival) remains unclear. The exact type of selection (against short men, favoring men around some above-average height, favoring only very tall men) also remains unclear. Here, for a cohort of military officers, we show that tallness had a indirect effect on male lifetime fitness, independent of socioeconomic success. The crucial factor was not that tall men had more fecund wives, but that tall men more often had a second family. Selection worked strongly in favor of very tall men, not just against short men. Since there were no hints of any evolutionary check on this selection, these findings suggest unconstrained directional selection for tallness in men.