Previous research indicates that people identify the sexual orientation of a target quickly and mostly accurately. We investigated the influence of the sex and sexual orientation of the observer on the accuracy of detecting homosexual men and women. Additionally, we examined observer confidence when rating a target as heterosexual or homosexual. Homosexual and heterosexual male and female observers (N = 619) were presented with 80 facial photographs (20 homosexual women, 20 heterosexual women, 20 homosexual men, and 20 heterosexual men). Participants were asked to identify whether the target was “gay” or “straight” and rate how confident they were with their judgement. The results showed that pictures of women were judged more accurately than pictures of men by all observers, and there was a general tendency for homosexual observers to use the “gay” category more frequently than heterosexual observers, although this bias was qualified by sex of picture. Finally, confidence ratings were more closely related to response bias than rating accuracy.