Researchers at the University of North Carolina at Chapel Hill and the University of Maryland at College Park are working on a new data-driven model and algorithm to identify the perceived emotions of individuals based on their walking styles.

Exploiting gait features to classify emotional state

Through RGB videos of an individual walking, the team extracted his/her walking gait in the form of a series of 3D poses. The aim was to exploit the gait features to classify the emotional state of the human into one of four emotions: happy, sad, angry, or neutral. The researchers’ perceived emotion recognition approach is based on using deep features learned via long short-term memory (LSTM) on labeled emotion datasets.

A representation of the novel algorithm to identify the perceived emotions of individuals based on their walking styles. Given an RGB video of an individual walking (top), the researchers extracted his/her walking gait as a series of 3D poses (bottom). The used a combination of deep features learned via an LSTM and affective features computed using posture and movement cues and classify using a Random Forest Classifier into basic emotions (e.g., happy, sad, etc.). Credit: Randhavane et al., Fair Use.

Moreover, the team combined these features with affective features computed from the gaits utilizing posture and movement cues. Such features are classified using a Random Forest Classifier (a type of algorithm). The team showed that its mapping between the combined feature space and the perceived emotional state provides 80.07% accuracy in identifying the perceived emotions. In addition to classifying discrete categories of emotions, the algorithm also predicts the values of perceived valence and arousal from gaits.

The visualization of the motion-captured gaits of four individuals with their classified emotion labels. Gait videos from 248 motion-captured gaits were displayed to the participants in a web-based user study to generate labels. The researchers used that data for training and validation. Credit: Randhavane et al., Fair Use.

The researchers also presented an “EWalk (Emotion Walk)” dataset that consists of videos of walking individuals with gaits and labeled emotions. To the best of their knowledge, this is the first gait-based model that identifies perceived emotions from videos of walking individuals.

The team showed that its mapping between the combined feature space and the perceived emotional state provides 80.07% accuracy in identifying the perceived emotions.

A long way to go

“Our research is at a very primitive stage,” stated Aniket Bera, a researcher involved in the study. “We want to explore different aspects of the body language and look at more cues such as facial expressions, speech, vocal patterns, etc., and use a multi-modal approach to combine all these cues with gaits. Currently, we assume that the walking motion is natural and does not involve any accessories (e.g., suitcase, mobile phones, etc.). As part of future work, we would like to collect more data and train our deep-learning model better. We will also attempt to extend our methodology to consider more activities such as running, gesturing, etc.”

What a fascinating study… that could have both positive and negative ramifications depending on how it is applied. 😻😕