Facing the Future

Can a New Zealand-based lab take virtual assistants from the realm of marketing gimmicks and endow them with real intelligence?

One day in the spring of 2010, a computer scientist named Mark Sagar sat down to compose an email to his boss, film director Peter Jackson. Sagar had been the special projects supervisor on Jackson’s 2005 film, King Kong, pioneering the facial motion capture technology that allowed the great ape to finally go beyond chest-beating and really emote.

With two Scientific and Engineering Awards from the Academy of Motion Picture Arts and Sciences already on his shelf, Sagar’s career seemed secure. The technology he’d helped invent — “an expression-based, editable character animation system,” as the academy put it in bestowing one of the Oscars — had played an indispensable role in the creation of the highest grossing movie of all time, James Cameron’s Avatar. He was poised to jump from one tentpole blockbuster to the next.

But the Kenyan-born engineer — whose family moved to New Zealand when he was a child — had been quietly hatching a more ambitious plan. The way he saw it, the motion capture technology he’d helped perfect was little more than puppetry — a way for technicians to map an array of points on an actor’s face to corresponding points on the face of an animated character. When the actor smiled, the character smiled. The results were impressive, but they were only skin-deep. For a character to be truly lifelike, Sagar knew, its expressions would have to come from within. They’d have to be motivated, driven, responding to an array of internal processes, like those of a living creature.

“I got to a point in my career where I’d gone as far as I wanted to,” he recalls, sitting in his office at the headquarters of Soul Machines, the Auckland-based startup he co-founded in 2016. “I thought, ‘Okay, I might actually want to work on different things than making computer-generated apes.’”

Sagar developed a lofty goal: to build an entity that would learn, feel, remember, and interface with people in much the way we interface with each other. He wanted to build a digital human.

In the email, he pitched the idea to Jackson as a tool for immersive storytelling — worlds filled with digital beings who could operate semiautonomously. “Rather than watching a story passively, imagine being face-to-face with a ‘live’ character and having every small action that you took affect the outcome,” he wrote. “For that, the character has to have senses,” Sagar went on, “and a brain that responds to you.”

He knew the idea sounded a little out there, but he had the road map all figured out. He had a background in computational biology, having created digital models of the eye and the heart for use in medical training. Sagar thought he could use the same approach for building a digital “brain:” identify each element and its function, build a mathematical model of it in code, and stitch the results together. “I believe technology is at the stage where we can do this,” he wrote to Jackson. The result would be “an entire new medium of social interaction with digital characters, stories that write themselves, with virtually infinite paths.”

Maybe Jackson didn’t quite get it, or maybe he was just preoccupied gearing up for the next Hobbit movie. Or maybe, given the widespread agreement among neuroscientists that our understanding of the human brain is still far too limited to contemplate building an authentic digital one, he just thought Sagar was nuts.

In any case, the director never responded. Sagar recounts the story sitting in his office in the Soul Machines headquarters, just off the Auckland waterfront. Dressed in cargo shorts, neon blue running shoes, and a T-shirt from a trip to South by Southwest, the 52-year-old computer scientist is fit and tan, with spiky auburn hair. There’s a bit of the mad professor about him, a slight aura of cheerful befuddlement. “At that point,” he says, “there was no going back.”

Sagar moved on, finding a more receptive audience at the University of Auckland’s computer science department, where he’d earned his doctorate in engineering. The school promptly set him up with an office and a small research budget. He took a pay cut, but he had total freedom to pursue his vision. Sagar christened his new concern the Laboratory for Animate Technologies, built a research team, and started hacking away at a virtual brain and nervous system. A few years later, Chris Liu of Horizons Ventures — the Hong Kong-based VC firm known for its investments in DeepMind, Siri, Impossible Foods, and other once fantastical seeming projects — stopped by for a visit. In November of 2016, Horizons put up $7.5 million to help Sagar launch a startup, Soul Machines, with the goal of continuing his research and identifying commercial applications. (Daimler Financial Services is another large investor.)

For the last two and a half years, the company has been tinkering away at a series of elaborate virtual avatars for several major corporate clients. And if you believe Liu, Sagar, and other proponents of the company’s work, the entities Sagar has been building — the “digital humans” — could one day change the world.