The evolution of Oculus Avatars

Video games have had realistic-looking, customizable humans for decades — this should be simple, right?

It is not simple. When you look at another player in a traditional multiplayer game, there’s generally no expectation for the character to move exactly like a human would — you know it’s being piloted by someone sitting on a couch, holding a gamepad. This lack of expectation is a saving grace.

In VR, when you see an avatar moving in a realistic and very human way, your mind begins to analyze it, and you see what’s wrong. You could almost think of this as an evolutionary defense mechanism. We should be wary of things that move like humans but don’t behave like us. (Cue fifty years of sci fi filmmaking and paperbacks).

Human skin, we learned, is really difficult to fake. When humans talk, the skin stretches over cheekbones and coloration changes, all of which is very hard to reproduce in a convincing manner, particularly in the tightly constrained compute budgets of VR experiences.

Eye and mouth movements are equally important to get right. It’s not enough to make the eyes blink and to articulate the jaw when you are talking. There are other social cues delivered by the brows, cheeks, and lips. These are nonverbal and, without a face-tracking camera, difficult to simulate.

It’s because of challenges like these that our teams at Facebook Reality Labs are having to invent completely new technologies to enable more realistic avatars over the next decade.

In the absence of mature technology, however, our initial approach was to abstract away from these problems.

Making humans look significantly less human and breaking the expectation of realism worked well. Our colleagues working on Facebook Spaces pursued a fully articulated avatar with simulated eyes, mouth, and arms, using a cartoon style to abstract away from anything that might come across as uncanny or heighten the discontinuity of simulated and tracked behaviors.

At the same time, we learned the value of greater realism. Presented with a more human likeness and proportions, people automatically understood the space someone was occupying; the distinct shape of the nose bridge could indicate attention and facilitate conversational turn-taking, even when viewed side-on. And realism opened up the use of avatars in contexts where toon avatars felt less appropriate, such as business meetings in VR.

For the launch of Oculus Avatars in 2016, we started with a volumetric and sculpturally accurate human representation, and we used a monochrome texture to abstract away from anything that felt too much like skin. We faded or covered areas of the body we couldn’t reliably simulate, all to deemphasize what was not really there and to focus attention on the very human motion of the head and hands that we could track using our hardware.