There are a couple of really big shots with 20 or 30 puppets all in the scene at Santa Monica beach in broad daylight with avatars and actors. We shot it all wild on a Technocrane, analyzed that move, then scaled that for motion control, and shot the greenscreen puppets that needed to go into the scene on a separate set using scaled motion control that would match the full-scale scenes. There was a lot of complicated programming that went into that.

VFX Voice: How did you approach scenes that required CG puppets, or avatars, as you call them?

Nicholson: Anytime you see a full-height puppet, it’s generally an avatar. That avatar has to be an exact match to the look and feel and animation of the real puppet itself. You have hair and fabric, and you need to match the way the light absorbs into felt. Skin is actually pretty easy compared to puppet felt!

The close-up avatars were scanned and had a complete texture model done, and others were done with photogrammetry. Some of the puppets on set that had their heads explode or get shot, we’d just do a bunch of photogrammetry of it so that we have a digital duplicate to match to the live action.

Phil Phillips is puppeteered on the greenscreen set.

“The close-up avatars were scanned and had a complete texture model done, and others were done with photogrammetry. Some of the puppets on set that had their heads explode or get shot, we’d just do a bunch of photogrammetry of it so that we have a digital duplicate to match to the live action.” —Sam Nicholson, Visual Effects Supervisor, Stargate Studios

We also did some motion capture of the puppets on set with Xsens MVN sensors on the puppets. And we had a puppeteer who was operating the main character, Phil Phillips, wearing a suit so that we could capture his leg movements. I wanted to capture all of Phil’s movements because, keep in mind that the shadow reconstruction is very important here.

Generally, the floors are all built up four feet high on a puppet set because puppets are operated above people’s heads. So even when you do something as simple as walk through a door into a room, the door is split in half so that the puppeteer can articulate his body through the door, but you have only the upper part of the door. You have the upper part of the torso, the puppet grabbing the doorknob, but then everything below the door knob is puppeteers and junk. So sometimes you have three people trying to fit through a doorway at the same time. It’s split, and there is no floor at all, so you have to reconstruct the floor, reconstruct the door, and then take the motion capture that we got on the day and add it and blend it into the puppet.

Puppeteer Bill Barretta (left) and director Brian Henson on the set of The Happytime Murders.

“Generally, the floors are all built up four feet high on a puppet set because puppets are operated above people’s heads. So even when you do something as simple as walk through a door into a room, the door is split in half so that the puppeteer can articulate his body through the door, but you have only the upper part of the door.” —Sam Nicholson, Visual Effects Supervisor, Stargate Studios

VFX Voice: Puppets move in a certain way because of the way they’re built – how did you match that with any CG movement?

Nicholson: Yes, one of the interesting things is that, when you think about it, if a puppet is on your wrist or on your arm and you rotate your wrist one way, your wrist only rotates one way for a certain amount. So when a puppet turns its head, depending on which hand the puppeteer’s using, the puppet can look around 180 degrees in one direction, but it might only be 30 degrees in the other direction. So we really had to match that kind of movement.

VFX Voice: What was your toughest shot?

Nicholson: There’s some shots in there with multiple puppets in full daylight and the camera’s moving, and you’ve also got to remember the whole point of the scene is that it needs to be funny. One of the funniest scenes is in the porn shop – the octopus milking a cow has got to be one of the classics of all time. But I mean, you’ve got to make that look spontaneous. Where do you find that in the rule book? I just don’t know.

Another greenscreen element featuring Phil Phillips.