Whether we see another person as trustworthy or competent depends in large part on the way they look, fair or not. It’s normal to associate certain facial features with certain personality traits, no matter how silly or arbitrary the link may be. But it’s how deeply we interpret facial features that’s cause for concern. A new article from Carnegie Mellon University provides evidence of our ingrained bias in favor of specific features, while also laying out the possible consequences of relying on these stereotypes. The research, published in Trends in Cognitive Sciences, lays out the serious implications of judging a book by its cover.

What We Like Individual preferences aside, humans generally find some traits particularly pleasing. By gathering results from numerous studies, the researchers discovered which features are usually considered trustworthy or untrustworthy, as well as competent or incompetent. They found several patterns, including: a link between feminine features and perceived trustworthiness

a link between mature and masculine facial features and perceived competence

a link between masculine features and negative expressions and perceived dominance Learn More: Why Plastic Surgery Isn’t Necessarily the Way to Go »

Why We Think the Way We Do We’re wired to take in visual information quickly, making faces ripe for interpretation. “The ease of processing faces combined with a general human tendency to simplify the world and categorize is one reason why we stereotype,” said Christopher Olivola, co-author of the study and an assistant professor of marketing at Carnegie Mellon University’s Tepper School of Business. He uses the example of two candidates applying for the same job with a photo and a resume. The resume should hold the most value for the purpose of fairness, but the photo influences our decision making more than we’d like to admit. “We’re making all these really big important decisions, and we’d like to think we’re doing it in a deliberate, rational way, but research shows us we really are biased,” Olivola said. Learn How to Tell If Your Friend Is Faking a Laugh »

The Downside of ‘Face-ism’ When we automatically assume that people have certain traits based on their facial features, it can cause serious problems. Not only can individuals be negatively impacted by “face-ism,” but others might also suffer inadvertently if they confuse what’s considered attractive with a good personality. The potential impact is huge: “The way a person looks in terms of their facial structure does predict their outcomes in life,” Olivola said. The review shows that the biases are especially apparent in politics and the justice system, two areas that, in theory, should be grounded in facts, not aesthetics. Politicians who appear more competent get a leg-up in elections, even from constituents who don’t share their political views. The people making these superficial decisions rely more on media than on strong political knowledge when voting. And people with untrustworthy faces, Olivola says, are more likely to be pulled from police lineups, to go to trial, and to receive harsher sentences than their sweet-faced peers.