The belief in physiognomy—the art of reading character from faces—has been with us for centuries1–3. People everywhere infer traits (for example, trustworthiness) from faces, and these inferences predict economic, legal and even voting decisions2,4. Research has identified many configurations of facial features that predict specific trait inferences2,5–14, and detailed computational models of such inferences have recently been developed5–7,15–17. However, these configurations do not fully account for trait inferences from faces. Here, we propose a new direction in the study of inferences from faces, inspired by a cognitive–ecological18–20 and implicit-learning approach21,22. Any face can be positioned in a statistical distribution of faces extracted from the environment. We argue that understanding inferences from faces requires consideration of the statistical position of the faces in this learned distribution. Four experiments show that the mere statistical position of faces imbues them with social meaning: faces are evaluated more negatively the more they deviate from a learned central tendency. Our findings open new possibilities for the study of face evaluation, providing a potential model for explaining both individual and cross-cultural variation, as individuals are immersed in varying environments that contain different distributions of facial features.