You can tell when someone's faking a smile or pretending to be in pain, right? Sure you can. But computer scientists think they can build systems that do it even better. There's already a Google Glass app in beta testing that claims to provide a real-time readout of the emotional expressions of people in your field of view. And a new study finds that the same technology can detect fake expressions of pain with 85% accuracy – far better than people can, even with practice.

Granted, the study was done in a carefully controlled laboratory setting, not a messy real-world situation like a dive bar during last call, but the findings still look impressive.

Computers have long been better than humans at feats of logic, such as winning at chess, but they've lagged far behind humans in perceptual tasks like speech recognition and identifying visual objects, says Marian Bartlett, an expert on computer vision and machine learning at the University of California, San Diego and an author of the new study. "Perceptual processes that are very easy for humans are hard for computers," Bartlett said. "This is one of the first examples of computers being better than people at a perceptual process."

>'This is one of the first examples of computers being better than people at a perceptual process.'

There are several efforts underway to use computer vision and machine learning algorithms to decode human facial expressions, something that could have uses ranging from interrogating criminal suspects, to A/B testing car commercials, to gauging the mood of people as they shop.

The method Barlett's team has developed is based on the idea that genuine and fake expressions of emotion involve different pathways in the brain. Real emotional expressions are executed almost reflexively by the brainstem and spinal cord, the thinking goes, whereas fake expressions require more conscious thought and involve motor-planning regions of the cerebral cortex. As a result, the movements produced are different in subtle ways that a computer vision system can detect – even if people typically can't.

More specifically, Bartlett's system is based on something called the Facial Action Coding System, or FACS, which was popularized by the psychologist Paul Ekman in the '70s and '80s and is used today by everyone from TSA screeners to animators trying to imbue their characters with more realistic facial expressions. It's a way of describing virtually any facial expression that's anatomically possible by breaking it down into its component movements – a wrinkle of the nose, a tightening of the eyelid, a lowering of the brow, and so on. The idea is that each of these movements maps onto a specific muscle or set of muscles.

Bartlett's team has been working for years to create a computer vision system to automate FACS and to develop machine learning algorithms that can learn to recognize patterns of facial movements that correspond to particular emotions. (They also founded a company, Emotient, based on the same technology – more on that later). The new study is the first to assess how well the system distinguishes genuine from fake facial expressions and compare its performance to that of human observers.

First, Bartlett's team recruited 25 volunteers and recorded two videos with each. One video captured the subject's facial expression as he or she experienced real pain from submerging one arm in a bucket of ice water for a minute. For the other video, the researchers asked subjects to fake being in pain for a minute while they dipped their arm in a bucket of warm water.

To set a benchmark for testing their computer system, the researchers first showed these videos to 170 people and asked them to distinguish fake from real pain. They did no better than chance. And they didn't improve much with practice: even after watching 24 pairs of videos and being told which were fake and which were real, human observers only achieved about 55 percent accuracy – statistically better than chance, but just barely.

The computer system, on the other hand, got it right 85 percent of the time, the researchers report today in Current Biology.

The system has two main elements: computer vision and machine learning. The computer vision system can identify 20 of the 46 facial movements described in FACS, virtually in real-time. (Coding the movements in a 1-minute video by hand would take up to 3 hours, the researchers write). The system also captures information about the timing of the movements, such as how quickly the lips part and how long they stay that way.

Information gathered by the computer vision system then gets fed into a machine learning system that learns to identify patterns of features that distinguish real from fake expressions. For example, the researchers trained the system by feeding it 24 pairs of videos – with each pair showing the same person's facial expression during real and fake pain. Then they tested it on a new pair of videos it had never "seen" before. Then they repeated this with additional videos to come up with the 85 percent figure.

A readout from Emotient's automated emotion detection system. Image: Emotient

When Bartlett's team queried the system to find out which features it was using to make the distinction, they found that the most important features had to do with opening the mouth. Whether they're experiencing pain or faking it, people grimace on and off during the minute-long videos, Barlett explains. But they did it a little differently. "When they’re faking it their mouth opening is too regular," she said. "The duration is too consistent and the interval in between mouth openings is too consistent."

"The numbers they're getting are definitely very good, probably better than I would have expected," said Matthew Turk, an expert on computer vision at the University of California, Santa Barbara.

There's a significant caveat though. The videos used in the study were carefully controlled and constrained. "The visual real world is just more complex – the brightness is changing, the background is changing, the face is moving back and forth," Turk said. "That can overwhelm a system like this that works really well in the lab."

The challenge, he says, is to make these systems work really well in the real world.

That's exactly what Bartlett is trying to do. She thinks automated pain detection could be useful for doctors and nurses working with children. Research suggests that pain is often underreported and under treated in kids, she says.

She's also developing systems that detect more than just pain. The company she co-founded, Emotient, recently released an app for Google glass aimed initially at salespeople looking for insight into their customers' mood. Presumably, any Google Glass wearer will eventually be able to use it.

A realtime color-coded display indicates which emotions the system is supposedly picking up in the people around you. The company claims it can accurately detect joy, sadness, anger, fear, and disgust. And if you're being a Glasshole, the app just might clue you in: It's also programmed to detect contempt.

In the image on the left, the woman is faking pain. In the other two, she's not.