Looking back at ten years of creative experimentation to discover the future of filmmaking for virtual reality.

Previously, I described how breakthroughs in computation and imaging have opened up a new dimension for filmmakers — today I’m looking back to trace the history of volumetric video capture, a key element of this burgeoning medium.

T he theory goes like this —

Life’s most universal stories feature drama between real people. If it were possible to depict real actors in virtual reality (as opposed to computer generated characters), the medium’s storytelling possibilities would expand to encompass a dramatic potential bigger than today’s cinema. A new narrative format would emerge where the entire range of human emotions and experiences could unfold within an interactive simulation. Resulting in something like…

HBO’s Westworld depicts a far-future vision of what virtual reality hints at today

It turns out there are some serious technical challenges to putting lifelike human performances into interactive virtual reality. Unlike watching a movie, a spectator in VR is able physically look and move through the scene. This freedom renders 2D video (and even stereo 3D video) completely obsolete. A traditional video placed into VR can only be viewed from one perspective, and looks obviously flat as soon the viewer departs from the camera’s original position.

Traditional 2D video placed into true 3D VR environments isn’t going to fool anyone.

The solution? Volumetric Video

Volumetric video is an emerging format of video featuring moving images of real people that exist truly in 3D — like holograms — allowing them to be viewed from any angle at any moment in time. The trick is that this media requires a fundamentally different video technology capable of capturing 3D images of actors at fast framerates.

The tried and true reference for volumetric video— Star Wars Princess Leia Hologram

The last few years have given birth to an explosion of technology startups offering ways to capture and present true-to-life 3D holograms of people in virtual reality. Among them is our very own DepthKit. Catalyzing this frenzy is the burgeoning VR industry’s desire to showcase immersive experiences that appeal to audiences broader than traditional gaming.

The aesthetic progression of volumetric video. Left: Radiohead House of Cards (2007) Right: Microsoft Research (2015)

As the prevalence of volumetric video technology continues to explode into 2017, I want to take a moment look back at the years of cross disciplinary collaborations between artists and researchers that have laid the technical and aesthetic groundwork for this new medium to emerge. Inspired by cyberpunk thrillers and space operas of the prior century, these pioneers have worked through the complexities and embraced the glitches in pursuit of discovering a new dimension of storytelling.

By tracing how these ideas emerged, we can continue to make progress towards realizing their creative potential.

2005 to 2009: Re-appropriation of Research Technologies

Turning back to the clock to the mid-2000’s, three dimensional video wasn’t more than a science fiction concept popularized by visual effects wizards in scenes from Star Wars and Minority Report.

Minority Report (2002) featured holographic home movies

However, deep behind the secure walls of academic research institutions, breakthroughs in computational photography and computer graphics were giving way to the technology that would take volumetric video from science fiction into reality.

During those early days, a few pioneering artists managed to scale the ivory tower to collaborate with researchers. Applying their techniques towards popular culture media. In 2009, Director James Frost collaborated with media artist Aaron Koblin to 3D capture point-clouds of Radiohead’s Thom Yorke performing House of Cards.

House of Cards — Radiohead Directed by James Frost, 2007. Technique: Structured Light

Inspired by the Radiohead video, artist and computer scientist Kyle McDonald figured out how to recreate the technique himself. Kyle translated the academic research into an open source Instructables on how to build a scanner oneself using an off-the-shelf projector and camera. He applied the technique in collaboration with director Alan Poon to create a video for Broken Social Scene’s track Forced to Love.