Google Glass app being designed to read emotions Feelings tool may transform digital-human interactions

Last week, I stood in the cramped office of a Sand Hill Road venture capital firm, staring through Google Glass eyewear at an 18-year-old with a blank expression.

In the display above my right eye, the word "neutral" appeared. Then he cracked a wide smile and the word changed to "happy."

The young man with plenty of reasons to grin is Catalin Voss, an entrepreneur and Stanford student from Germany who has been working on iPhone apps since he was 12 - that is to say, pretty much since there has been an iPhone.

Now, with a small team of mostly fellow students, he's working on emotion-recognition tools that could improve education and training by monitoring engagement. But there are other interesting use cases as well: Voss, who has a cousin with autism, thinks the Glass app could help those with difficulty discerning emotions to interact in more natural ways, easing their path through the world.

The company, Sension, based in the Menlo Park offices of Highland Capital for the summer, is among a handful of businesses making strides in emotion-recognition technology. The tools can analyze facial expressions and vocal patterns for signs of specific emotions: Happiness, sadness, anger, frustration and more.

There's a broad array of potential applications, including potentially creepy commercial ones: If my TV knows I'm feeling depressed, might it load up an ad for fast food?

Catalin Voss, founder of Sension, who has a cousin with autism, thinks the Glass app he's developing could help those with difficulty discerning emotions to interact more naturally. Catalin Voss, founder of Sension, who has a cousin with autism, thinks the Glass app he's developing could help those with difficulty discerning emotions to interact more naturally. Photo: Lea Suzuki, The Chronicle Photo: Lea Suzuki, The Chronicle Image 1 of / 6 Caption Close Google Glass app being designed to read emotions 1 / 6 Back to Gallery

But the broader goal is to make machines communicate with humans in more natural ways. In that sense, it can be seen as the latest step in the long history of human-computer interaction, a layer on top of motion sensors like Microsoft's Kinect controller or voice-recognition services like Google Now and Siri. The machines can understand more than the defined meaning of words or gestures, putting them into the context of the feelings with which they're expressed.

"It could really help improve digital interactions, to the point where people feel not just listened to, but heard," said Susan Etlinger, an industry analyst with Altimeter Group.

Automated customer service systems could, for instance, escalate calls to human operators when the tone suggests your blood is beginning to boil. If you're screaming at or swiping emphatically on your smartphone, as opposed to speaking or tapping calmly, apps can adjust their reactions.

Responds to mood

"If Siri understands not just what I say, but how I feel, it will come back with an answer that matches my mood," said Dan Emodi, vice president at Beyond Verbal, an Israeli startup that has built technology that can detect the emotional states suggested by vocal intonation. "It's adding a totally new dimension. It really could change the relationship we have between us and machines."

The company also sees opportunities to train people to be better interviewers, managers or even parents, by helping them understand their own emotional state and how they're coming across.

The general approach to developing these tools is to use machine-learning algorithms, training the software by feeding it existing video or audio where people's emotions are clear: a big smile, a cheery lilt, a furrowed brow, etc.

Facial expression analysis is an active research area in academia. The state-of-the-art approach to face tracking is known as a "constrained local model," which learns by tracking dozens of points on the face. It allows the expression to be read whatever the angle of the head.

Sension's tool looks at 78 points on the face, including the center of the pupil, the arch of the eyebrow and corners of the mouth.

But following and analyzing the movement of dozens of points in three-dimensional space generates a huge amount of data very rapidly, which makes it difficult to crunch and deliver useful results in real time, especially on a small device like a smartphone or Google Glass, Voss said.

Studies face

To speed things up, Sension targets specific points and focuses on the common ways that the shape of the face changes.

"We took state-of-the-art facial recognition technology, accelerated the hell out of it and from that we can compute a score in real time," said Voss, who serves as chief executive of Sension.

By specifically measuring how engaged students or employees are during a lecture or video training, they believe they can help schools or companies to improve the effectiveness of their educational content.

As more education moves online, this could be an important tool for maintaining interest among students. In fact, part of the inspiration for the company was the boring videos Stanford students have to watch for certain classes.

"Training and education went from the classroom to the Internet and lost a lot," Voss said. "We want to make it more like the classroom, more natural again."

Help with autism

Meanwhile, Voss envisions the app for those with autism and similar conditions as a temporary training measure, with patients wearing the device for a few months to improve their understanding of expressions. He said the company will explore tools for the other of side equation as well: Helping those people understand how to respond appropriately to the emotional state of others.

He hopes to eventually do full trials of the technology to test its efficacy, which is, of course, at this point unproven.

Sension already has several paying customers, but has only named one to date, an online training company in Palo Alto known as Mindflash. Sension raised a small amount of money through that time-honored first round of "friends, family, fools." In addition, Highland Capital handed them $18,000, along with free office space and food as part of the Summer@Highland program for student entrepreneurs.

At the end of the summer, they'll move back to StartX, Stanford's startup accelerator program. They plan to raise a round of capital and accelerate hiring later this fall.

Voss stresses that they're building privacy protections into their apps: They don't upload facial images, store anything on the phone or attempt to identify individuals through facial recognition (which is banned by Google for Glass).

He added that the team has no interest in pursuing any marketing applications of emotion recognition.

"That's not what we want to do, and I don't think people will trust us if we do that," he said. "We want to make the way you interact with technology more natural."