Kevin Hardy

The Des Moines Register

IOWA CITY, Ia. — Bill Hedgcock knows it sounds a little creepy.

Tucked into the white ceiling tiles, the ceiling camera he had installed at the Pappajohn Business Building at the University of Iowa scans the faces of all that pass under it and instantly calculates their moods — collecting readings for joy, frustration, confusion, fear, anger and sadness.

“We nicknamed it ‘the creepy study,’ because we just wanted to be out about it, just so everyone’s aware,” said Hedgcock, an associate professor of marketing. “It sounds creepy.”

The facial encoding data is part of a larger student research project underway in the Tippie College of Business that uses automated technology to read emotions by measuring slight movements in the facial muscles, such as a movement of the eyebrow or a widening of a lip.

Hedgcock believes the University of Iowa is the world’s only business school with this kind of real-time software that converts images of people’s faces into readings for different moods. If it works, experts see huge potential for facial encoding in the worlds of marketing, advertising and political campaigns.

Facebook brings face recognition to Messenger to speed up sharing

But that’s no sure bet. Hedgcock makes it clear that the technology’s accuracy remains unproven — it’s one reason why his students have spent months picking it apart, getting a better understanding of the technology’s limitations.

If his students eventually deem the technology a stinker, it’s no big deal, Hedgcock said. But if the process is proven and adopted by marketing firms, his students will have a leg up when they go on the job hunt.

“If it does work, they walk out there and they know something that no one else knows how to do,” he said. “It’s not written in a textbook. It doesn’t exist anywhere.”

Finding a niche

Hedgcock wanted to emphasize privacy with his students, so the camera doesn’t record video or images of the people it measures. It simply kicks out data measuring what it sees — and it sees a lot — millions of rows of data so far.

Students have used the mood measurements in a variety of ways. From the first-floor camera, they crunched the data to see whether the weather affects mood and if mood affects food sales at the business school’s snack bar.

Do sunny skies make people happier? Do people purchase more caffeine when they’re down?

But students say the best use of the technology isn’t in a generic space such as a hallway on campus. People walking to and from classes or meetings aren’t exactly the most expressive.

Students say the technology is probably most effective with a more specific aim, such as judging reaction to a short advertisement or a political debate — occasions more likely to elicit an emotional response.

Students paired up with Frank N. Magid Associates, a market research firm with offices in Marion, Iowa. The company is well-known for its work gauging viewer sentiment about local television stations and their on-air personalities.

While it may never fully replace more conventional methods of judging consumer preference such as surveys and dial tests, Magid leaders say they believe facial encoding could be a supplement to their current suite of measurement tools.

“We don’t see it as a replacement for common sense,” said Bob Crawford, senior vice president for research at the firm. “But it’s just another arrow in our quiver that we could take a look at.

“We don’t know the accuracy yet. But it is a big deal.”

Hedgcock believes facial encoding may be best suited for answering questions consumers are unwilling or unable to answer themselves.

Realizing the limitations

The biggest problem with the business school’s mounted camera is that most people aren’t walking around with outright frowns or glowing smiles on their faces.

“Everyone in this building is doing the same thing,” said senior marketing major Lynn Tatge. “Most of the time people aren’t going to be super excited to be here."

So, it’s no surprise that “neutral” ranks as the program’s most frequent mood output.

“Really, there’s a lot of just nothing,” Hedgcock said. “There’s a lot of neutral.”

And there are lingering questions about the tool’s accuracy. How can you be sure that someone is actually happy, sad or frustrated?

“We believe the actual accuracy, if we’re being generous, is probably around 35%. So it’s got some distance to travel in our opinion,” said Dan Hill, president of Sensory Logic, a market research firm in Minneapolis that has used the manual version of measuring facial expressions to read emotions.

But that doesn’t mean it doesn’t have value.

“The students are making the right journey,” Hill said. “It’s a cool idea, it’s moving there, but it’s not really there.”

Seeing the possibilities

Hill says the dictum that “actions speak louder than words” is the very reason why reading facial expressions is so powerful. He’s used the technique to evaluate professional athletes’ attitudes during draft time, to gauge witnesses and juries during trial and to inform would-be bosses on hiring decisions.

“Your emotions turn on when something is meaningful,” he said. “You can figure out when people are emotionally engaged.”

That’s why he and other experts believe a more proven facial encoding software will be invaluable in politics, because it could possibly judge which stances or verbiage voters are drawn to from a candidate.

“I think it’s a good application because politics is very emotional,” Hill said. “There are a lot of so-called independents who really tilt one way or the other, but they like the idea that they’re independent.”

Hedgcock said political strategists are already interested in neuromarketing, the marketing field that uses measurements of the human body — brain scans, respiratory rates and changes in the skin — to measure a consumer’s reaction.

But not many campaigns are interested in talking publicly about those ideas, Hedgcock said, because they can be viewed as too Big Brotherly or manipulative.

For now, the professor is putting aside any concerns about the “creepy” factor.

“Every academic I know in this area of neuromarketing would say the last thing we need to worry about is over manipulating,” he said. “The bigger concern is does it even work?”

