It is genuinely difficult to sustain a casual conversation that includes more than four speakers. Add a fifth speaker, and the conversation often quickly fissions into smaller groups. Termed ‘the dinner party problem,’ this four-person conversation size limit is believed to be caused by evolved cognitive constraints on human mentalizing capacities. In this view, people can mentally manage three other minds at any one time, leading to four-person conversations. But whereas existing work has posited and empirically tested alternative accounts of what drives the conversation size constraint, to our knowledge, no work has explored the question of why this capacity is specifically four? In this theoretical paper, we (a) review research demonstrating this cognitive constraint in sociality, (b) review the relevant working memory literature, which has explored the “why four” question at some length, and (c) we begin to pose possible answers to our specific social “why four” question. Using simple mathematical models of small-scale sociality, which we imbue with evolutionarily-relevant content, we present one novel possible explanation for the four-person conversation size constraint.