Alamy

EVEN when one person is doing all the talking, a conversation is an interactive process. The listener participates through expression, posture and movement, and the speaker interprets what he sees to determine what the listener does and does not understand. Computers, however, cannot make such interpretations. Or, at least, they have not been able to do so until now. But a study by Louis-Philippe Morency of the University of Southern California and his colleagues may change that. In the latest edition of the Journal of Alternative Agent and Multiagent Systems they demonstrated that a computer can be given the ability to understand at least one significant human gesture: the nod.

The nod is one of the most important signals a person can send. When made at the right time, it means “I am engaged in what you are saying” or “I understand”. For computer programs and robots that are designed to interact with people, being able to notice nods and respond with nods of their own (through an on-screen avatar in the case of a computer, and with actual nods in the case of robots) would be a boon. Unfortunately, nobody has yet been able to get it right.

Psychologists have spent years analysing human interactions to try to work out what it is that makes someone nod. The results have been poor. Studying interactions is gruelling and time-consuming work. The rule of thumb in the field is that each recorded minute of interaction takes an hour to analyse. Moreover, many social cues are subtle, and not easily noticed by even the most attentive psychologists.

Dr Morency therefore set out to develop a computer system that can automate the process. It is able to analyse video and audio recordings in order to recognise gestures of both posture (moving the head in particular ways, for example) and voice (such as changes in pitch). The system logs the sequence of these cues, and then compares sequences from different speakers to see which combinations routinely lead to a listener nodding, and which do not. The result is a “cookbook” detailing which recipes (combinations of cues) are most likely to make listeners nod.

To try the system out, the team established an experiment with 100 participants, all of whom were asked either to watch a short video and describe what they saw, or to listen to the description and describe what they heard. Dr Morency knew from past experiments that lowering of vocal pitch and pausing were both cues likely to draw a nod from a listener, so these were things that he programmed the system to look out for. He and his colleagues suspected, however, that the cues for nodding were far more complicated than the past studies had indicated. They therefore programmed their system to look at a great many other characteristics, including body posture, direction of gaze and even the specific words that were being spoken.

Their paper reveals that there is much more to eliciting a nod than simply lowering the pitch of the voice and pausing. While these two things are certainly relevant, the new system also picked up two more cues that nobody had noticed before. One was that speakers frequently make a gaze shift towards a listener just before a nod is given. The other is that they often use the word “and” to generate a responsive nod.

These findings, Dr Morency hopes, are only the beginning. Although nods are important, they are but a small part of human social interaction. He now plans to use the technology to analyse more subtle patterns of human interaction, some of which may—like the gazing and “anding” that trigger nods, not yet be known about.

Even just with nodding, though, the new system should have applications. Added to teaching programs it will be able to help such software work out whether the lessons being proferred have been learned. The armed forces, too, hope to benefit. America's army is already using the technology to analyse interactions between people in other countries, with a view to including this information in programs designed to teach cultural differences to soldiers stationed in foreign lands. For once, then, the politics of gesture may be positive rather than negative.