Israeli startup Faception made headlines this year by claiming it could predict how likely people are to be terrorists, pedophiles, and more by analyzing faces with deep learning.

It's an unsettling idea. Experts and research in the field, however, suggest that it is more fantasy than reality.

Faception assigns ratings after training artificial intelligence on faces of terrorists, pedophiles, Mensa members, professional poker players, and more. Through deep learning—that emerging technique found in everything from Alpha Go to Siri to Netflix—the AI can supposedly predict how likely a new face is to belong to any given group.

While this may sound believable, there's no evidence that face-based personality predictions are more than a tiny bit accurate. And there are many reasons to fear it might promote bias and other dangerous effects.

Face-based predictions of personality are extremely limited.

There's a long history of trying to predict personality from faces. Called physiognomy, it's been largely refuted.

"Impressions from faces are very, very low quality evidence," said Alexander Todorov, a Princeton psychologist who specializes in facial perception. "You’re going to make important decisions, whether someone’s competent or fits the job, based on their appearance? That’s a ridiculous notion."

Although Todorov's work focuses on human perception of faces, AI doesn't seem to be much better.

Forthcoming papers out of Switzerland and the US show that deep learning models using social media profile pictures were able to explain only 2-to-3% of variation in personality scores. By comparison, predictions using Facebook likes were able to explain over 30%.

What Faception is trying to do is even harder. Rather than use carefully curated profile pictures, which give clues to personality in style, scene, and pose, it uses relatively neutral images like passport photos and security camera stills.

The startup tries to get around this limitation by looking at a wider range of variables, notably facial structure and other inherent physical differences, hoping to uncover genetic influences in personality.

For now, there's little proof that this works.

"Current psychological research shows that by adulthood, personality is mostly influenced by the environment," wrote Daniel Preotiuc-Pietro, a post-doctoral researcher at the University of Pennsylvania who has worked on predicting personality from profile images, in an email. "While it is potentially possible to predict personality from a photo, this is at best slightly better than chance in the case of humans. I seriously doubt the 80% accuracy for personality [claimed by Faception]."

Faception is also on uncharted territory when it comes to predicting behavior, rather than personality, from facial traits. It's one thing to spot an extrovert and another to spot a terrorist.

It's true that Faception has access to cutting-edge research. One of its advisors is Michal Kosinski, the Stanford social psychologist who showed that you can effectively predict personality traits using Facebook likes. Kosinski is currently finalizing a study that he says will show that you can make pretty good predictions based on faces too.

"It is easy to infer age and gender from your face. It turns out you can also infer things like personality, sexual orientation, and so on," said Kosinski.

Kosinski said his models can make predictions based on facial traits that are tied to individual differences in people's hormones and genes, but he wouldn't say how accurate those predictions are. He also wouldn't comment on Faception's approach, noting that he doesn't know the inner workings, except to say that it could work in theory.

Faception claims to be highly accurate. The company says its models would have flagged eight of 11 of the Paris attackers and that in a 50-person poker tournament, its four picks included two of the three finalists. CEO Shai Gilboa also tells us it has reached over 90% accuracy on some classifiers. It's not clear what these claims really mean, however, and it's impossible to verify them.

Faception claims its terrorist classifier would have identified 8 of 11 Paris attackers. Faception "The problem with this kind of commercial application is that it’s very hard to evaluate the evidence because nothing is published and you don’t know what they have done," said Todorov.

All that's to say there's little evidence that Faception’s methods would yield reliable results. Given that, it's frightening to imagine how they might be used.

Facial profiling with deep learning could be dangerous.

Where to start? For one, there's the risk of false positives.

"Even if there’s good evidence, better than chance, what are the costs when you make an error?" said Todorov. "It’s stigmatizing and it has a lot of ramifications."

In fact, false positives are inevitable with classifiers looking at things like terrorists.

"Even the most accurate model aimed at a rare outcome will produce a great majority of false positives—extremely rare 'true positives,' like being a terrorist, will be hidden among thousands or hundreds of thousands of 'false positives,'" said Kosinski.

If you don't think that's a problem, imagine how you'd feel if word leaked that you were at at high risk of being a terrorist or pedophile.

Another risk is that algorithms might fail to disassociate ethnicity, gender, and other factors. Then a model supposedly looking at facial traits could really be a cover for even more controversial kinds of discrimination.

"The possibility of encoding biases is very real," wrote Preotiuc-Pietro. "An algorithm is trained to optimize its accuracy and this may lead to over-representing stereotypes."

Examples of algorithms falling into this trap are increasingly common. A ProPublica investigation earlier this year found that software used to predict criminality is biased against black people. Another study found that women are shown fewer online ads for high-paying jobs. Then there was the AI beauty contest that was biased in favor of white people.

Beauty.AI admitted its AI-powered beauty pageant was biased in favor of white people. beauty.ai There are ways to try to guard against bias in facial profiling.

"One is to separate models for different groups of people," Kosinski wrote in an email. "Second, is to analyze the results separately for different groups. One could, for instance, look more closely at the suspects in a given group that are in top x% of the scores, regardless of the fact that even the most 'terrorist-looking' female face is probably much less terrorist looking (in [the] model’s eyes) than most 'terrorist-looking' male."

Gilboa wouldn't go into details about how Faception avoids bias but insisted it wasn't a problem: "We don't [make] such mistakes."

The risk is that this kind of discrimination can reinforce stereotypes, alienate broad groups of people, and distract from other information that is more useful.

"Do you know the book, 'Moneyball?' Todorov asked, referring to the account of a baseball executive who questioned assumptions about what makes a good player. "That’s a great book and this is exactly, exactly the opposite point of view. What explains the success of Billy Bean? Well, because he didn’t believe in appearance."

Facial profiling is on the rise

Whatever you think about facial profiling, it's not going anywhere.

Faception, for one, claims to be off to a good start. Gilboa told us the startup has been hired to analyze security risk for unnamed governments and default risk for unnamed financial clients. He said it is also focused on developing applications to improve human-robot interactions through personalization.

Faception could succeed, even if it is highly inaccurate, even if it is highly unsavory. If its models can make banks, security agencies, and other groups feel they are ever so slightly better as evaluating people, then it might have a market. After all, facial profiling has the advantage of working in the absence of any private data.

And this could be just the start.

"There’s a fast-growing interest in this technology," said Kosinski. "I’ve seen some startups [in the area] and I’m also sure that companies like Facebook and Google and Microsoft have plenty of people who are computer vision specialists and also are very strong on the deep learning front."

"Once the knowledge is there," said Todorov, "I wouldn’t be surprised if some people, companies, businesses buy into it."

No doubt some of these applications will help the world.

"The benefits are mind-blowing," said Kosinski. "For instance, predicting illness—if you have a technology that can take a picture of your face and say you might have diabetes or something."

Other applications are more controversial.

"If crossing a border to a country that is less liberal and less concerned with human rights than the United States, they don’t even need to get data," said Kosinski. "They can just take a picture of your face and then have a prediction of political affiliation."

One thing is for sure, said Kosinski: "It’s going to be very bad for privacy."