A Behind the Scenes Report on the Making of the Show Visuals and Delivery Systems

Image credits Valerio Berdini | liveon35mm.com

Amon Tobin's show ISAM premiered at MUTEK June 1st to incredulity and awe from fans and the media alike. The general consensus from all camps was that this show was something new, futuristic, not experienced before; it exceeded all possible expectations and was very, very exciting. Suspension of disbelief had been achieved and audiences transported with it.

Our friends at V Squared Labs and Leviathan responsible for the show visuals "killed it" - that's pretty much the word.

The team made up of V Squared director Vello Virkhaus, designer and programmer Peter Sistrom and Leviathan chief scientist Matt Daly along with the help of Bryant Place built a custom application that runs the visuals of the entire Amon Tobin show end to end. The application is made in TouchDesigner, Derivative's visual programming environment, and does the projection mapping, video playback, Kinect response, realtime effects and more.

If you've seen the live show or even video footage of the performance then you know what we're talking about. If you've been living in a media blackout for the last few weeks here is a compilation of what people have been saying.

Gizmodo called it "The Concert of the Future, Today" and CHARTattack said that "it was a live show one might have said could only be created and performed by a cybernetic organism." where "groundbreaking modeling and mapping techniques were used to take the audience at the Metropolis on a trip to dozens of worlds. ... it felt like one was transported into deep space or, alternately, inside the Tron grid, a game of Tetris, an M.C. Escher painting, the fiery pits of Mordor, an acid-coloured kaleidoscope, a Splinter Cell video game..." Wired got straight to the point: "ISAM's live show looks like a mindfuck of the highest order" with "little in the way of precedent."

Image credits Valerio Berdini | liveon35mm.com

Image credits Valerio Berdini | liveon35mm.com

Image credits Valerio Berdini | liveon35mm.com

As for the audience at the MUTEK premiere here is what some had to say: overwhelming, mind boggling, feeling small, the most amazing visual performance i've ever seen in my life.. the amount of work, i can't even fathom the technology behind it because it was just insane. The dynamics of everything, the range of imagery, i was really blown away...

Also after the premiere, MUTEK founder and artistic director Alain Mongeau told Vello he was breathless and would have to talk to Vello after he'd caught his breath.

Get the picture?

Image credits Valerio Berdini | liveon35mm.com

Image credits Valerio Berdini | liveon35mm.com

We had some questions for Peter, Matt, and Vello because even with what we at Derivative know about TouchDesigner and building performative systems and interfaces for large shows and events, it was difficult to construct a complete picture of what this team actually did and how to achieve the scale and complexity of Amon Tobin show. With a single projector!

We reached the team at different times in different locations and various post-show states via Skype. What follows is an amalgamation of their experiences and insights on the making of this show that includes as in-depth a description as possible (without demystifying the whole thing) of the individual components that made up the superb system used to run the show end to end.

ON VELLO, MATT, PETER & BRYANT

Vello Virkhaus, ISAM Live Creative Director

Vello Virkhaus, Founder and CEO of V Squared Labs is an experienced Visual Director, Producer and VJ who takes on hugely complex projects with brilliant results. He is also clearly not afraid of anything. Vello has enormous technical proficiency and a long history of pioneering and innovative work for some of the industry's top talent. Vello has produced visual shows for the likes of Paul Oakenfold, The Red Hot Chili Peppers, Korn, Ministry, Jay-Z, 50cent, and Beyonce. He has also directed and produced unique and technologically advanced audio visual experience content packages for DJ's, nightclubs and architectural installations, most recently, for the Marquee nightclub in Las Vegas.

Vello was introduced to TouchDesigner by his friend and frequent collaborator Matt Daly to create 'Dome 2', a custom 360 visual performance system along with brand content for Heineken Inspire activations. They decided to use TouchDesigner as their "creative tool", as Vello puts it, for its high quality realtime 3D and UI development capabilities.

Vello: "Matt got me completely obsessed with TouchDesigner and we've been using it ever since." - on quite a few engineering-intensive projects included Electric Daisy Carnival, Ultra Music Festival, Roger Sanchez, and last year the Marquee in Vegas where Vello worked with Matt and Bryant to design and produce a pretty hot interactive, multi-screen experience.

Matt Daly, ISAM Live Chief Scientist

Leviathan is celebrating their 1st anniversary the day after our conversation with Matt Daly. In a year the company with a staff of 25 has a lot to show for itself. They're the largest 3D, visual effects shop in Chicago and they're only just getting started.

Animation, broadcast, and traditional cinematic work forms the basis of the studio's practice and the "experiential work" as Matt puts it, is another tangent that allows Leviathan to access different clients. Experiential projects involve producing big systems with lots of projectors, computers and content. Leviathan creates the content as well as the systems, interactive, video serving and so on and according to Matt "TouchDesigner is definitely giving us an edge over everyone else." "Without your software we're up a river." he stresses.

D: So what you're saying is you use TouchDesigner where you wouldn't go without?

"Definitely - there's no other tool that can do the things we've done with it. I won't kid you, I have looked and there's nothing else that can give you a really reliable system for running the kinds projects we produce. We've run some very high-profile shows in front of a lot of people and TouchDesigner hasn't failed us. If there's any failure it's on our part or the computer itself."

Bryant Place, ISAM Live Designer

Bryant Place a.k.a. CyberPatrolUnit is a busy guy and really good at what he takes on - which is considerable. He's an accomplished VJ and visual programmer based in LA, though he spends a lot of time on the road. For the Skrillex summer tour Bryant designed the entire show - the LED layout and stage design as well as the video system itself built from scratch in 5 days at AG Lighting. The system uses TouchDesigner to speak DMX and move LED panels around to animate them for the show's very cool visual effects. Prior to that Bryant toured with Plastikman Live 2010 tour driving the TouchDesigner based show visuals.

Bryant has work extensively with Vello Virkhaus' V Squared Labs team, performing for major musical acts and also developing and installing a highly specialized audio-visualization system for Marquee Nightclub at the Cosmopolitan Vegas. Bryant was an integral part of the V Squared team on the Tobin show assisting in many ways, technically and with two of the pieces creatively.

Bryant was at V Squared when we reached Vello so we were lucky enough to bag 2 birds at once.

Peter Sistrom, ISAM Live Designer and Chief Programmer

Peter Sistrom is an incredibly capable young man schooled in architectural design who 3 years ago developed a passion for motion graphics and visuals, particularly in a synesthetic relationship with music.

We met Peter last year when his impressive entry for the Plastikman Live Visual contest was selected as one of the finalists. At that time he'd been using TouchDesigner for only a few months and prior to that had been working in Processing and Pure Data. "I had no background in computer programming, motion graphics or film before embarking on this journey, so I have been teaching myself everything by the seat of my pants." Peter told us then.

Very quickly Peter has become one of the software's prolific users and most engaged promoters. Earlier this year Peter demoed TouchDesigner and its interoperability with Arduino at the LA Crash Space. Bryant Place who had been working with Vello at V Squared brought Peter into the fold for the Amon Tobin project. It was effectively Peter's first TouchDesigner job and as Vello and Matt have made clear, his role was significant in the execution of the show.

Matt: Peter is the one responsible for putting together what we see as the show. We gave him some tools to use but he definitely deserves the credit for assembling this whole thing and creating a performable show out of it.

Amon Tobin is Peter's favorite electronic music artist and in many ways this was an ideal experience. Peter's role was, as he modestly explained, "to funnel it all in" and keep in touch with all the other artists and renderers performing reality, quality and continuity checks, integrating content from various sources, integrating check and lineup with the timeline and triggering it as well as designing visual content and working on the show's transitions. Impressive!

D: Peter, is this your first mapping project?

Peter: Of a large scale yes. I've done it in my bedroom before!

ON THE MAKING OF ISAM

D: VELLO, you recently said about the show that you loved the emotional response the 4D mapping triggers in people. What did you mean by '4D' exactly?

Vello: 4D in mathematics is a very abstract concept in which this additional dimension is indistinguishable, yet acknowledged. This unknown to me relates to the pronounced visual effect the mapped structure of Tobin creates for the viewer. What is fascinating is that we give viewers an idea of what it might be like to see beyond 3D space, to see all points simultaneously for both the exterior and the virtual interior of the set. The combination of this mapped effect and Amon's music produced some very intense emotional reactions from people after the MUTEK performance.

Bryant: At times you do forget that there's a physical structure there, that's where the magic is.

D: It's a huge project how did it come about?

Vello: Alex Lazarus, production designer at Blasthaus had mistakenly emailed the wrong person at ninjatunes, yielding an inquiry from Jeff Waye about video mapping for Amon Tobin. This humorous, yet powerful inquiry led Alex to contact us, as he believed we could be perfect for the project with our mutual understanding of electronic art and music. We then put together an initial presentation with a set of story boards and written treatments of ideas, along with a reel of LED pixel mapping samples. Before this project, we had only done a few video mapping projects, but nothing of this scale or complexity. We had been primarily working with LED, and mapping in planar space.

Things escalated pretty quickly from the initial set of images with developments by Alex and Heather Shaw of fantastic looking physical set designs. Once it seemed like we were actually in the running against our competition, I got Matt Daly at Leviathan involved and collectively we began to storyboard out the whole show.

Storyboards supplied by VSL

Amon liked it, so I started expanding a written creative of everybody's design ideas and then added Amon's feedback. Once we had the contract, I flew up to Amon's house in the Bay Area where we pounded out a complete written treatment with final reference materials in a day. Two months later the show launched at MUTEK June 1st.

D: It sounds as if you were fully invested, more even...

Vello: Yes Matt Daly and I both put in a great amount of time and resources to get the project to this state.

D: Were you involved in the design of the structure in view of what you were going to be projecting on it? We've been hearing a rumor you're using a single projector on this show...

Vello: I was very fortunate to have some technical input early on with the set design Heather and Alex were developing. My involvement in this process was more from a technical video projection standpoint. I was looking at the structure(s) and determining if they could be mapped, with how many projectors etc. Seeing their ideas come to life was also an added inspiration in creative direction. I knew from the get-go that steering the show in an achievable direction meant going with a single projector because there'd be no time for mixing and blending 2 projectors together. So, I made technical evaluations regarding these matters.

D: What was the process, working with Amon? He appears to be a very considerate artist.

Vello: It's exciting to work with an artist who's interested in working with and building a team. They're active and creative and interested in pushing the project into the fine art medium so there's a lot of territory to grow into with this type of experience because it's transcended the commercial barrier.

ON VISUALS

D: There's a vast amount of content in the show and a lot of variety in that. Were the show visuals designed per track or thematically?

Vello: The show visuals were designed per track with an overarching, loose narrative hemming it all together. Amon was symbolically represented as a pilot and the structure itself the craft. We designed imagery that helped transform the 2bit structure into these various narrative and abstract constructs. The vast variety of theme content was also chosen to keep things moving and keep the audience interested.

D: Audience reaction to this show has been rather extreme. What did you guys do to get this kind of response?

Matt: What separates this project from others is the cinematic content, people really respond to that and then having the generative material layered overtop. There was a story to be told, spaceship, asteroids, alien planet and other things, we did storyboards, did extensive pre-production with Vello and helped get the project. Leviathan provided a huge amount of HD content, our render farm was cooking for a couple of weeks.

Image credits Valerio Berdini | liveon35mm.com

D: Did you have any references, overarching concepts?

Matt: The original references provided for this were Chris Cunningham's work where he took these very expensive (at the time) renders and then completely messed them up, just ripped them to shreds without any regard for how beautiful the rendering quality was. That's something we were trying to achieve.

We took these really high-end renderings - it took weeks of machine hours to render these scenes - and then completely disregarded them and tore them up with our glitching and realtime dynamics. I think that was a welcome sight to people because it didn't have that "Oh look how beautifully rendered and perfect this is". It had a bit of Atari style video game appeal to it. Low resolution and very fluid, computers and digital and organic in a way I think a lot of people haven't seen before.

D: Very much in line with Amon's musical composing style then? So much of his source material is organic which is then treated as you're describing. As he told FastCompany, his work seeks to strip away the familiar, he"breaks things down and reorders/transforms them to create hybrids that lead the imagination into other unsettling and disorienting worlds."

Matt: Yes, very much so. Amon's music is very sculpted. When you break it down each song is like a sculpture in a way because it's very carefully crafted and he's using these effects that maybe in other music would be a lot more repetitive. He's a classical composer of the electronic age.

D: Peter, what was your experience working with Amon and designing visuals for this show?

Peter: He's my favorite electronic artist so it was a thrill. Amon pushes but respects and understands. He's got a particular vision in terms of what it feels like but he's not a visual artist so he has to trust us. Great teamwork.

At MUTEK Amon said, "hey can we get something a little more happening here because it seems a little slow?". And we could do that, we used these live effects - face mapping, procedural noise, broken scanning lines etc which are used in the DJ set but also in transitions.

D: Bryant and Peter, your contributions to the show visuals were produced somewhat differently. Can you describe that process please?

Bryant: Both Peter and I set out with the notion that there would be curtain songs, that is, songs that would run in real-time; this was great because right away it set limits as to how one could approach the graphical design. Working in realtime was a huge benefit because it really allows you to tune into the audio to create something tightly coordinated. For the most part we worked with the Animation component.

Peter: Yeah, the Animation component came into play in a few different ways, either for animating the whole song or little triggered bits as Bryant used it.

Once I found out we were not going to run in realtime, I actually really beefed things up and did things I wouldn't do for a realtime piece. What was nice though was that it still rendered out the whole 3.5 minute song in 1080 hd in about 10 minutes! Technically it could have run, just maybe not within the rest of the system at full bore.

This created a different workflow, making TouchDesigner into a sort of hybrid 3d animation and compositing system. I pulled lots of textures from real life, like paper and rust for Lost and Found. All the geometry was procedurally based on the model of the structure. Lots of depth rendering/compositing, 3d texture lookup animation for individual instanced objects, lighting tricks, quite a bit to get the mood and depth, fun stuff!

ON BUILDING YOUR OWN SYSTEMS

D: Vello, you've been at the forefront show by show and project by project building your own applications vs. going with turnkey solutions. How'd you go about building the system for the live show?

Vello: We'd never done such a huge mapping project and there was no precedent for the way this system had to operate – it would need to be a unique, new system. I don't know what we would have done without building our own solution in TouchDesigner to do this project. There existed a small number of previous TouchDesigner parts from other projects that we could pull from, but pretty much everything was built from scratch. We wanted to make our own proprietary engineering accomplishments for this show, to push ourselves further and prove that we could do it.

"It's a relatively simple system in the sense that it just runs" says Peter Sistrom but anybody who's pushing the envelope is building their own application: a user interface with custom-made controls, compositing and rendering for that show. Something completely unique that gets you to a level that's impossible to achieve with turnkey stuff.

This is indeed the case with the Amon Tobin show where TouchDesigner was used to design and build a Show Previewing System, a Show Sequencing System, the Show Rendering Unit and Alignment System. We are not able at this time to reveal much about these at request of the artist.

Show Sequencing System

Peter: The show's timeline or the Show Sequencing System is its own component and really the heart of the show. It's essentially the 'central show unit' which was invaluable for streaming all of that content and making it all manageable. Jarrett Smith from Derivative was instrumental in helping us with setting that up.

Show Previewing System

Peter: The show previewing system allows you to see what it looks like in 3D before the set is made. This enables the team to see how the perspective illusion works and allows for tweaking, trading off depth illusion vs the left/right position of the observer.

Show Rendering Unit

D: I guess it's a little nail-biting creating all that refined content before the cube set structure is built?

Peter: We'd been working off a digital model provided by Heather Shaw while the cube structure was being built and doing all our renderings based on that model. I had tried to instill in everyone producing content - the big Leviathan crew using Maya and Cinema 4D, to be somewhat procedural in their work... that adjustments might have to be made to the model. But then it turned out that everything was built perfectly and matched up and we never adjusted our virtual model - that was one of my big "Aha" moments which was more of a "Thank God" moment really!

ON KINECT

D: Matt, Kinect was used sparsely but very effectively on this show, what did you do?

Matt: One of the ways Amon appears during the show is projected on the entire set as a Kinect point cloud. What the audience sees is a custom rendering of the point cloud using the Kinect image. We have some other ideas on how to use the Kinect and it feels as though we've barely scratched the surface with this project. But we were also careful to not over-do it and risk losing the interest factor here.

ON "AHA" MOMENTS

D: On a show of this scale and complexity and you're the creative director, were there any actually memorable moments that were illuminating?

Vello: I was working with Peter in a hotel room in Vegas doing a Heineken event and working on the Roger Sanchez system and my big "Aha" moment came when I asked Peter to make a box with a sphere inside it because I wanted to prove that we could recreate a faces and edges (of the cubes) map look with a floating interior. A big part of what I'd sold to Amon was that the show would have a floating cinematic feel combined with mapping effects. Peter was just noodling around and I looked over his shoulder and he'd re-projected the box and the sphere on the box and the sphere and as soon as I saw that I instantly knew that we had the technique to do ISAM. That was the big "Aha" moment. That was "Oh shit, there it is!" and I felt fully confident.

Vello: The second "Aha" moment came when we actually saw the calibration tool project onto the 2bit set for the first time at the rehearsal stage. This was one of the most powerful moments which provided me the most assurance and relief. I had been really concerned about the overall technology and how things were coming together, how they were going to work. I was actually super paranoid having heard a few video mapping calibration horror stories. The projectors were literally dropped down off of the forklift and onto the truss stands, plugged in, powered on and calibrated in less than 30 minutes - the first time.

Peter: A lot of "Aha" moments were centered around the illusion - the perspectival projection mapping illusion. We had got it all set up in the warehouse we'd rented for a week and Matt did the first map, the first calibration and said "Okay, throw something up so we can see." and literally the first thing I had was a rendering of the structure with just a soft noisy gray - almost stone tone, and two lights. It popped up and we just kind of all looked at it and "Woah!" Suddenly this thing looks like it's made out of stone and it's just sitting there. It was the most simple thing but it was really remarkable. There were enough cues to make it believable that it was angled stone cubes.

Matt laughs "When we finally saw things light up for the first time going from the virtual to real, that was a really good feeling like, "Wow! This thing is actually going to work!" Everything was playing back, we were getting good frame rates and it all looked good. We're using SSD drives and the video's playing back flawlessly, I don't think we ever really hit a bandwidth for that - you guys (Derivative) are really efficient with the resources of the system.

ON BIG SHOWS, SINGULAR SYSTEMS AND SPECTACLES

D: There's been an apparent trend recently, at least, 3 of the artists using TouchDesigner to run their shows - Richie Hawtin's Plastikman, DJ Shadow and now Amon Tobin have nested themselves right inside the performance set. Amon said that the idea for ISAM was to integrate himself "quite literally, into an audio and visual presentation of the album." What do you make of this?

Vello: The performance set is an extension of the individual musician experience. DJ'ing has been and is a very individualistic thing, so it seems like a natural extension of the DJ as superstar to have them be inside a larger structure and have visual representation of the sonic going on around them and be inserted into it.

Image credits Valerio Berdini | liveon35mm.com

Bryant: I think there's a common goal now where everyone is trying to achieve the effect of a singular system where all devices are in communication working as a singular system. We're eliminating technical boundaries where something might look as if it's on its own operational system and not quite doing the same things as another element. So yeah, I think there's some common ground there between all these different projects to create the effect of a singular system.

D: Amon Tobin sees tools as a starting and not an end point and has said that music technology can appeal to our inherent laziness. You can now buy what you might lack through your own efforts and creativity. Shows are definitely getting bigger and technologically more complex... where do you think that's going and what is it about?

Peter: Big shows have worked their way into the zeitgeist. Big electronic artists - Amon, Richie Hawtin, Deadmau5 et al. These days you've got to do a big 3D show. Amon came in at a good time after lots of experimentation in all of this and brought his amazing depth and texture.

Bryant: I think also that the difference between Amon and a lot of previous shows, is that he's generating some of the visuals on his own. There's a point in the show where he's interacting with the Kinect camera, painting waveforms onto the structure by waving his hand around.

Matt: Everyone's always looking for the next big thing. Someone will make a tool and then everyone else will do it and it becomes commonplace. Augmented reality for example. It's always going to be bigger and better.

Vello: Everyone wants a show and it's about expanding the immersive performance space. Everyone has a different approach and no matter what they're using and how they're doing it the desire is to entertain an audience. Mediums are converging in the entertainment sphere and they have been for a long time as far as what you do. The job description's getting a little hazier these days.

Bryant: But it keeps it exciting. Every day it seems more and more possibilities are coming into fruition.

Vello: What makes this show so nice is that it has feeling. It's cinematic and a feast for your eyes. Projection mapping is definitely a future direction - it has great transformative qualities. However there are also more complications doing this in a festival environment.

ON HORIZONS

D: What's next on your radar Peter? It's been an action-packed few months. What do you have coming up and what are your current sources of inspiration, things you'd like to do?

Peter: Right after the Roundhouse show in London there's the Electric Daisy Carnival in Vegas. It will be a big LED extravaganza and 3 days of straight VJing. Then in July there's a lot of personal work I want to get back into - the next version of my Name the Machine project with my good friend and collaborator Matthew Davis.

I personally love the expanse of big flat HD screen but there's so much out there. I like surfaces and spaces, maybe it's my architecture background. I'd love to work on the integrated design of a space, a venue - not unlike the Cinechamber but more of an architectural thing that integrates video and media as a venue space and then you can either use those spaces or not as artists.

D: What's next for you guys? Now what?

Bryant: Well one of the things I'm going to take to the field very soon is doing things beyond video and video mapping. I'm getting more into DMX and trying to control servos and send information to other devices rather than a video signal and integrating those two. Like doing some sort of LED video mapping but then on top of that sending information to lighting or other devices.

Matt: Definitely work on more experiential projects. We want to do bigger more integrated projects using everything from projection system technology and LED wall technology to integrated systems large multi-screen venues, digital signage integrated into building systems - that sort of range.

Vello: We plan on evolving the technology and system structure created for the Tobin experience as far as we possibly can. I feel as if we are just scratching the surface on the potential of the immersive entertainment market. I hope to gain more financial success and artistic recognition.

We've all had a lot of response from the Amon Tobin show, the phones are ringing and we'll be knocking on your door for more software licenses really soon. ;-)

ON TOUCHDESIGNER: VISUAL THINKING IN AN AGE OF VISUALIAZTION

TouchDesigner is WYSIWID: what you see is what it's doing. Its open, broad and highly-visual procedural architecture boosts and expands discovery, creativity and productivity making it the most complete authoring tool for building interactive 3D art, visualizations, user interfaces and rapid prototypes.

No stranger to large, complex live events and mapping projects, TouchDesigner was recently used to map two icons of modern architecture: the Sydney Opera House and The Guggenheim Museum. Both were wrapped inside and out, the Opera House as visual accompaniment of a live symphonic performance and the Guggenheim for YouTube's first Biennial awards. TouchDesigner was also used to build the audio-reactive visual show system for the Plastikman Live Tour, voted RA's top live show of 2010.

The ambitious and complex Amon Tobin Show is an excellent example of not only what TouchDesigner can do but also exemplifies how it can be used - the spirit of working with a tool if you like. The V Squared team used TouchDesigner for prototyping and then building the show's UIs and components; as the glue that puts all these systems together; as an interface through data bases, XML etc.; to produce show visuals; as a tool for rapid visualization and to render and then run the show.

Performing high-profile complex live events involving tons of technology often in front of millions of people you need a dependable system.

Matt: "When it comes to the glue that puts all these systems together and lets us interface through data bases, XML etc. TouchDesigner has all those tools ready for us so we can prototype and get things up and running really fast."

Dedicated to advancing the way we make art and visualize information and ideas, Derivative has produced live visuals and interactive art projects for a roster of international superstars that include Prada, Herzog & de Meuron, Raster Noton and Rush. They have also developed major theme park attractions globally.

TouchDesigner FTE (Free Thinking Environment) and its counterpart TouchDesigner Pro puts a free development environment with an extremely rich feature set into the hands of artists, animators, educators and "everyone else". "TouchDesigner is our vision of what is possible in tomorrow's software tools for building interactive applications and exploring data, imagery and sound", Derivative founder and CEO Greg Hermanovic states. "It exploits what is possible in today's computing technology and is positioned to grow with the foreseeable advancements in computer, graphics and mobile technologies."

TouchDesigner 077 goes Gold in July 2011 – join Facebook "TouchDesigner", download TouchDesigner… and a big, big congratulations to the V Squared team! Stay tuned.