Ever since the first televised presidential debate in September 1960, when Richard Nixon was said to have been deemed the winner by Americans listening on the radio and John F. Kennedy was preferred by those watching TV, conventional wisdom has dictated that looks matter in politics. For decades political observers have pointed out that in presidential match-ups, it’s often the taller man who wins. In recent years, social and political scientists have begun looking seriously at this question, trying to quantify the effects of physical attractiveness in the lab. In presidential politics, does it help to look like Mitt Romney? Or, put another way, how much does Newt Gingrich’s face hurt him?

The answer will be disappointing to those who believe in the myth of the rational voter. Looks do indeed matter. But they don’t matter in exactly the way we thought—it’s not attractiveness alone that counts, but a cluster of traits people believe we can read into faces. It appears that voters, particularly those who aren’t paying much attention, don’t know much about politics, and don’t have strong partisan affiliations—which is to say, a solid number of Americans—operate like 19th-century phrenologists, believing on some not-quite-conscious level that that they can read a politician’s character by glancing at things like his eyebrows and jaw line.

For a long time, social scientists looked no farther than attractiveness when measuring the importance of candidate looks, and for good reason. Studies going at least as far back to the 1970s tended to show a positive correlation between handsomeness and how subjects assessed a candidate. Scholars sometimes reasoned that the attractiveness effect took place because voters without a lot of information about a candidate used good looks to infer other good qualities—a kind of “halo effect.” The beauty advantage makes intuitive sense because we see it in other, non-political arenas. Attractive people appear to benefit in all sorts of situations, like in the workplace and in legal situations. Heck, even babies are predisposed to focus on good-looking faces.

And then, in 2005, Princeton psychologist Alexander Todorov and colleagues published an astonishing study suggesting that beauty didn’t tell the whole story. Rather, voters appeared primarily drawn to faces that suggested competence—so much so that the effect could actually be seen in election results. In the lab, subjects glanced for a single second at the faces of congressional candidates. They didn’t know anything else about the candidates, and they didn’t recognize them. Almost 70 percent of the time, the face that subjects judged as more competent-looking actually won the election.

What does competence look like? Working with subjects rating photos of hundreds of faces, Todorov and colleagues have developed computer models of how faces can suggest character traits like trustworthiness and likability. The competent face shape is masculine but approachable, with a square jaw, high cheekbones, and large eyes. When people say Romney just looks presidential, this is the image they’re summoning.

Todorov and other psychologists believe that otherwise expressionless faces can appear to show emotion based on how they’re formed—the shape of the eyebrows can suggest anger, for instance, while a long distance between the eyes and the mouth can suggest sadness. On Todorov’s computer model of an incompetent face, beady, close-together eyes paired with high eyebrows suggest fearfulness, even through the face is expressionless. Todorov believes our tendency to read expression into neutral faces amounts to an “overgeneralization” of a healthy trait—human beings’ ability to judge others’ intentions from a brief glance. We need to know in an instant if the guy approaching us on the street poses a threat. Todorov has found that people make judgments about the faces of strangers after just 34 milliseconds of exposure to an image.

Following up on Todorov’s studies, political scientists Chappell Lawson and Gabriel Lenz found that the subjects most likely to rely on assessments of competence from a face alone are voters who don’t know much about politics but watch a lot of television. In other words, they’re the folks most likely to see images of the candidates, even if they’re not taking in information about those candidates, and even if they’re switching the channel the instant presidential debates come on.

Todorov and other scholars have also looked at how clusters of traits interact. They asked subjects to look at photos of candidate pairs from real elections and judge how good-looking and competent they appeared, as well as whether those politicians appeared likely to lie to them or to physically threaten them. They found that attractiveness is not an unalloyed asset to a politician. Candidates who were good-looking but also appeared incompetent faired quite poorly in real elections, as if the traits exacerbated one another. (Paging Rick Perry.) They also found that those candidates who appeared threatening in the lab actually tended to lose elections 65 percent of the time.

There’s some evidence that our preference for certain facial traits is situational. A British study compared images of George W. Bush and of John Kerry that had been blended with other faces in such a way that their basic face shape and features were maintained but the politicians themselves were no longer recognizable. Facing hypothetical elections, subjects preferred the Bush-like face during wartime, and the Kerry-like face during peacetime. The authors speculated that Bush’s face had more masculine and dominant characteristics, which subjects appeared to prefer during hawkish periods.

Taken all together, these new studies suggest that how a politician’s face appeals to voters, or doesn’t, can’t be boiled down to just one factor. Rather, voters look at a candidate and make a series of instant judgments based on a number of traits. Then, of course, in many cases, they listen to the candidate, they consider the issues, and they do all the things rational voters are supposed to do. Skin-deep inferences aren’t all that voters rely on, though they may have an outsized effect on the decision-making process.

“I tend to think that what happens is we get an emotional reaction to someone and that sets the stage for everything else, for the way we process information about them,” says Lee Budesheim, a psychologist at Creighton University in Omaha, Neb., who has studied the impact of candidate attractiveness.

There may be one other factor that influences our judgments about candidate appearance, and it is highly individual. In a 2008 study, Stanford University actually morphed images of unfamiliar political candidates with pictures of their lab subjects, unbeknownst to their subjects. The effect was subtle, maintaining the basic look of the candidate, with just a hint of the subject’s face superimposed on it.

As researchers expected, subjects preferred the image of the candidate morphed with their own picture. They didn’t recognize their own faces, but on some level they recognized something familiar and similar to themselves. And in politics, it seems, familiarity breeds attraction. After all, who’s more competent and trustworthy than you?