Tom House

USATODAY

Graphics Interchange Formats – more commonly known as GIFs -- have become an increasingly fun and addictive way to express our feelings on social media, but can they be used to map the language of all emotions?

Travis Rich and Kevin Hu have channeled animated image sharing through a website called GIFGIF, an ongoing interactive project that presents users with two GIFs at a time and asks them to choose which better expresses one of 17 emotions (happiness, anger, guilt, etc.). The data collection is presented similarly to a game, inviting users to participate as many times as they want by voting on more than 6,000 GIFs sourced from the site Giphy.

The result of the process is a collection of assorted GIFs ranked and scored based on how well they portray one of the 17 emotions.

The MIT pair met each other while working at the same lab space and found similarities in their interest with what Rich refers to as “non-verbal communication.”

“A lot of non-verbal communication happens through GIFs with websites dedicated to that emotional context,” Rich says. “We wanted to build a map of that emotional context.”

Currently GIFGIF has more than 2.5 million votes and the research has already produced some interesting results.

“We found some emotions are strongly correlated, like anger and fear, while others were more loosely correlated, like sadness and happiness,” Rich notes.

They also found that body language better expressed certain emotions. For instance, those featured in GIFs portraying excitement almost all had their arms in the air while those portraying anger commonly featured their arms in a downward motion.

These observations may seem obvious but the implications of such findings suggest that GIFs could even be used to train computer algorithms how to read human emotions through what Rich and Hu call a “text to GIF translator.”

Because GIFGIF assigns and categorizes GIFs based on how well they portray a specific emotion, computer programs can use their emotional content to translate short lines of text, such as a poem or email, into GIFs. The computer may not be able to fully read human emotions but it would be able to at least begin connecting language to emotion.

Rich and Hu are also working on a GIF translator called “Mirror Mirror” that reads human facial expressions. The translator will allow users to search for GIFs based on facial expressions utilizing a facial feature-tracker along with the GIFGIF library.

However, the research has proven useful beyond just the MIT lab.

Michael Shehane, a professor at the Academy of Art University in San Francisco, teaches English for art purposes. Since many of his students are from China with English not being their first language, Shehane says he uses the GIFGIF program to help his students “learn adjectives about emotions.”

“I found [GIFGIF] through BuzzFeed…I use it for class to teach about facial expressions and emotions” Shehane says. “The [class] is all about connecting music and images to emotional responses.”

By utilizing the website, Shehane’s students spend up to 15 minutes in class selecting which GIF best corresponds to a different emotion.

“The students see it as a game,” he says with a laugh.

By using the program, the students are able to “critically engage and recognize different emotions” in order to better understand the lines of facial expressions so they can draw them for artistic purposes.

Since certain emotions – like embarrassment and disgust -- are more nuanced to culture, GIFGIF provides another means for students to “recognize facial expressions,” Shehane says.

“They’re learning language not just through words.”

The biggest issue Travis and Kevin face with their research, however, is how it will translate with older generations. Most teenagers and 20-somthings can tell you what a GIF is but what about older or less social media-savvy users?

The answer, according to Hu, is what he calls the “democratization of GIFs.”

“The older generation doesn’t have the vocabulary, but they have the emotions,” Hu says. “[GIFGIF] is a dictionary for those without the vocabulary.”

The pair says they plan on releasing a private API (a set of instructions for accessing a web-based software) to mobile app creators, website designers and researchers, but ultimately the two think it’s fully possible to capture all emotions with just GIFs.

“GIFs are richer than text, they are more empathetic to the human condition,” Kevin says.



Thomas House is a rising junior at American University

This story originally appeared on the USA TODAY College blog, a news source produced for college students by student journalists. The blog closed in September of 2017.