Film Music & The Future Of The Sampled Orchestra Mixing / Production By Mark Wherry

Creating the score for Pirates Of The Caribbean: At World's End required an incredible amount of music technology, and demonstrated why 64-bit computing offers new levels of realism for orchestral sample libraries.

Film music is perhaps the single area where music technology has its largest application. Not only does film music require all the usual elements of sequencing and recording, but there are the added complications of synchronisation, frame rates, submitting mixes to dub stages, and all sorts of nonsense that other musicians simply don't have to worry about. This complexity makes film musicians ever more reliant on technology for reasons of efficiency: so many tedious tasks need to be performed that it would be painful to consider having to do these things without the aid of technology. As painful, in fact, as it was to carry out such tasks before the technology came along in the first place.

At the same time, technology can also help those working in film music to be more creative — or at least that's the hope. Whether it's creating a new sound that inspires an idea, or working with a computer-aided orchestra that allows for more experimentation, technology can take a musician to more places creatively than they could otherwise explore. And this is especially important when you have to be creative under a time pressure, as is usually the case in film music. So, when it works, music technology allows film composers to be creative, efficiently.

I get to spend my days worrying about this issue, working for the film composer Hans Zimmer. Just recently, while working on Pirates Of The Carribean III: At World's End, Hans decided he wanted to push our use of technology further than normal. This resulted in arguably the most technically accomplished, realistic-sounding orchestral mock-ups ever created, and I think some of the best music written for the Pirates series so far. It also led to Hans' day-to-day technical coordinator referring to the film as 'Samples of the Caribbean'...

Hans is well known for having created his own private collection of sampled orchestral instruments, and the core of this library was originally recorded back in 1994 onto eight-track digital Tascam DA88 format. After the recordings had been edited, instruments were programmed for Hans' sampler of choice at the time, Roland's S760, which wasn't a bad option, considering that each unit took only 1U of rack space and Hans needed over 30 of them to play back the orchestra. Over the years, more samples were added to the library, and the library itself went through many conversions as more powerful samplers were released, such as Emu's E4. In 2002, the original recordings were transferred again from the original DA88s, remastered and programmed natively into Tascam's Gigastudio, and these newly created versions still make up a large part of the palette five years on.

All This Effort For A Few Samples? One of the reasons samples are so important in film music is that their use allows the director, and other people involved in the film-making process, such as the editor or producers, to hear how a piece of music (usually referred to as a cue) is going to sound against the film. As Hans himself has commented many times, using samples of real orchestral instruments is going to give a better impression than the composer sat at the piano going "but you know, this is going to sound great when the horns come in!" These days it's quite common for sampled demos to be used by editors to cut against while they're working on the film, or for showing previews to test audiences, before the final versions of the cues have been approved and performed by a real orchestra. There are a couple of advantages here, but the main one is that it allows for ideas to be tried out easily before the expense of trying them out with a real orchestra. So good demos, and thus good samples, are important for modern film composers to get their ideas across.

Although they still sound pretty good, Hans really wanted to start again with new recordings: partly because the technology has improved radically since 1994, but mostly because he was simply bored with hearing the same old sounds every day. So in 2004, a year after I'd started working with Hans — and 10 years after the original library had been recorded — we started again. The new library was recorded using the same studio as its predecessor (AIR Studios in London), and largely the same bunch of musicians as before, but there were also some key differences.

While the original recordings were made at 44.1kHz with a 16-bit resolution, the new recordings were captured at 88.2kHz/24-bit using Apogee converters. Originally we had planned to record at 176.4kHz, and we did actually record the first sessions using two Pro Tools systems: one at 176.4, and another at 88.2kHz. The differences between the two recordings were very subtle indeed, and although I would have liked to keep the higher sampling rate (I even wanted to record a DSD archive version of all the sessions as well!) ultimately it would have been too expensive. It wasn't just the cost of recording the data to consider, but the cost of backing up that data, the systems required to edit the data, and so on.

One of the reasons we were creating so much data is that instead of recording a single surround mix of the samples, we recorded each sample using 16 separate mics, each on its own channel, just as it might be heard in an actual film score recording. In recent years there have been commercial libraries that allow you to balance, say, close, room and stage mixes; but our goal was to build a system that would allow us to mix from all 16 microphone channels in real time, even when the composer was playing back the samples. While this might sound a little crazy (and that's partly because it is crazy), the result would enable us to work with the mix of a piece of music containing samples in exactly the same way we would work with a piece of music recorded by a real orchestra. Perhaps our secret weapon in the whole recording process was AIR's chief engineer, Geoff Foster, and we relied completely on his experience for the choice of microphones and positioning.

Once the samples had been recorded, the next task was to process them using noise-reduction techniques. Noise reduction is quite a sensitive issue: after taking so much time and care recording the samples, the last thing you want to do is make those samples sound inferior. But the fact is that noise reduction is a necessary compromise when you're creating samples, because no matter how small the amount of noise present in your recording, when that noise is multiplied by the number of times the recording is going to be layered up in the sampler as you play notes, it becomes a real problem.

So at the end of 2004 I flew back to England with my partner on the project, Abhay Manusmare, and visited noise-reduction specialists CEDAR Audio. Geoff Foster came up from London and we spent a day with one of CEDAR's specialists, investigating how their high-end Cambridge system could be used to clean up our samples while preserving as much of the natural reverb of the room and the tone of the instruments as possible. We were pretty impressed, and we decided to purchase a couple of Cambridge systems for the processing of our samples.

This left the Herculean task of editing the samples and turning them into playable instruments, and for this Hans and I turned to a mutual friend, Claudius Bruese. Claudius lives in Germany and is best known for his early work at Waldorf, as product manager for the Wave, and his recent work on products such as Steinberg's The Grand and Halion Symphonic Orchestra. He's also an accomplished composer and musician and a brilliantly organised person to deal with the large task that lay ahead!

However, what do you use to edit high-resolution, multi-channel samples into playable instruments? Well, to be honest, we couldn't find anything. It's not just about the cutting, for which an application like Pro Tools would be fine, but the looping and assigning of metadata as well. So at that point we gave up on the whole project. Just kidding. Fortunately, by this time, Hans had invested in a company called Wizoo (which is now part of Digidesign) and a team of programmers created an editing system that would allow us to create the content for our new library.

When I'm 64-Bit There's a big change going on right now in the computer world as we all start adopting 64-bit operating systems, and the big advantage of using a 64-bit operating system, as you've probably heard, is its ability to access a large amount of memory. When you run an application on 32-bit Windows, for example, that application can only use around 2GB of memory (or up to 3GB in some circumstances). If you run any plug-ins within that application, such as a software sampler plug-in, that plug-in will be included under the application's umbrella of memory usage. So if you're running a sequencer and a bunch of plug-ins on 32-bit Windows, the maximum amount of memory they can collectively access is 2GB. Under a 64-bit operating system, an application has no such limitation (well, there is a limitation, but it's so damn high that you're likely to run out of money to buy more memory before you reach that limit, and most current workstations can only physically accommodate 16 or 32GB of memory anyway). However, running a 64-bit operating system isn't an instant solution. Not only will you need 64-bit drivers for all your MIDI and audio hardware, but the applications you run need to be specially compiled for 64-bit operating systems in order to make use of all the memory. At present, as far as I'm aware, Cakewalk's Sonar is the only major DAW available on Windows commercially in a 64-bit version. Assuming you have the appropriate drivers for all your hardware, your current 32-bit Windows applications can perform better on 64-bit Windows. I briefly mentioned earlier that some applications can be made to use up to 3GB on computers with 4GB of memory running 32-bit Windows, and 32-bit applications can actually use up to 4GB if run on 64-bit Windows, provided the developers set a certain flag correctly in the program's executable file. Even though Cubase isn't available in a 64-bit version, for example, the current 32-bit version will already take advantage of this option. For this reason, we were running our GVI host software (and later our own samplers) on systems with 64-bit Windows. Our audio hardware was RME's Fireface, and although this was overkill, since we only needed the ADAT outputs, it was one of the few bits of hardware available last year with good 64-bit drivers and a couple of ADAT ports — although that situation has now changed. Fortunately, since we were already using a custom-developed system for sending MIDI over networks, we could build that technology straight into the host application without having to worry about installing MIDI drivers. (Commercial alternatives like Music Lab's MIDI Over LAN now support 64-bit Windows.) Even though we had developed the host in-house, though, it wasn't possible to make it a true 64-bit application, because one feature of 64-bit applications is that they can only load plug-ins compiled for 64-bit systems. Similarly, 32-bit applications can only use plug-ins compiled for 32-bit systems. So if you're running 64-bit Windows and you want to use a 32-bit plug-in, you have to run it within a 32-bit application. (Cakewalk actually have a feature in Sonar called Bit Bridge that allows those running the 64-bit version of Sonar to use 32-bit plug-ins, but that wasn't an option for us, and a full explanation is a little beyond the scope of even this article!) So even though we could have used a 64-bit application like Sonar to access a large amount of memory in a music context, there still weren't any 64-bit sampler plug-ins that we could run within Sonar to play back our orchestral sounds.

During 2005 we didn't get much chance to work on the new sample library, because of other film projects; but towards the end of 2005, and throughout 2006, Claudius and his team really started ploughing through the content we'd recorded so far. Finally, in the Summer of 2006, we began to receive some test instruments of the new library that we could play: short strings and long strings.

The short strings sounded great, and with six or seven velocity layers and up to six round-robin variations on each note (incorporating up and down bows), produced surprisingly realistic performances. Unfortunately, though, Hans absolutely hated the long strings! On the day he heard them, we ended up sitting in the studio until six in the morning, and he was pretty depressed about the whole thing. Part of the problem was that they weren't very playable, and the fact that they weren't very playable made it difficult to judge their quality when compared to the old library.

The reason for this lack of playability was the way each note had been cut. The full natural attack had been preserved, which was great for playing individual notes, but made it hard to perform sweeping legato lines. However, this turned out to be a blessing in disguise, because Claudius later came up with a solution. This was to provide three different attacks for the long notes (original, short, and something in between) that could be selected by the person playing the samples, using keyswitches.

The new attacks solved the problem of playability and still gave the option of having the natural attacks, but it was an interesting lesson. One of the things we really tried to achieve with the samples was to create the most natural, realistic sound possible, and the idea of manipulating the recordings too much as they became sampled instruments was initially quite abhorrent. However, the experience of the long notes served as a reminder of the paradox of recreating acoustic instruments electronically: sometimes you have to do quite unnatural things to recreate something natural. Playing in a legato style is quite natural for a real violinist, but quite unnatural for a sampler.

Even though these test instruments were still far from the ultimate potential of the new library, they were pretty good and sounded, as you would have hoped, significantly better than the old library. So Hans decided he really wanted to use them on Pirates 3: if he was going to have to do another sequel, went the reasoning, we might as well try to make the music sound as different as possible and really push the boundaries. At the time, I had no idea how far that idea would take us.

The first draft instruments we received from Claudius were not the full-quality final versions, partly because we had no way of playing back 16-channel samples with a decent amount of polyphony at high sample rates. So instead, four-channel Gigastudio versions were created (at 16-bit/44.1kHz), consisting of two stereo Giga instruments (one for the left and right front speakers, and one for the left and right rear speakers) that needed to be played back as a stacked instrument on the same MIDI channel in Gigastudio. Although this worked well, there were a couple of big problems.

To say that memory is one of the problems we had with Gigastudio might sound a little odd — isn't the fact that it streams from hard disk meant to save memory? Well, yes and no. Like all streaming samplers, even though Gigastudio does indeed stream samples from hard disk, in order to do this efficiently it needs to pre-load the first part of every sample used in an instrument into memory.

The reason for this is that the seek time of a hard drive — which is to say the time it takes for the hard drive to locate an item of data that's required — is about seven or eight milliseconds. Now think about the buffer size on your soundcard, which is probably set to somewhere between 1.5 and 12 milliseconds. At the point when you ask Gigastudio to play a note, there wouldn't be time to fetch that data from hard disk and process it in time for there not to be a tremendous amount of latency — especially when you start to play more and more notes. So, instead, what happens is that the first part of the sample is played from memory, giving the hard disk sufficient time to start streaming the remainder of the sample.

With all our round-robin variations, different velocity layers, and alternate attacks, there were quite a large number of samples to pre-load, even just for one instrument. And we had separate long and short instruments for violins, violas, celli and basses, all needing to be loaded and played back!

In addition to memory, another problem we encountered was that playing back one of our instruments required hundreds of voices, using about half the processing resources of one fairly beefy computer running Gigastudio. On a modern system, you might get between 600 and 800 voices when running Gigastudio 3. This may sound like a large number, but it's the voice count for mono voices. For our quad samples, each note required a minimum of four voices, so that's only 150 to 200 notes all of a sudden, which easily get used up when you consider that all the samples have four seconds of natural ambience after them. Play a trill, and with the amount of overlapping notes you've just lost a quarter of your polyphony!

Worse still, one of the key features we use in Gigastudio for long strings is the ability to crossfade, using a MIDI controller, between different velocity layers, and this creates a real nightmare for the voice count. If we go back to the discussion of streaming and how it takes too much time to find something on a hard drive at the precise point at which the sampler requires it, and if you consider that the sampler never knows what's coming next, the only solution is that the sampler always has to be ready. Part of being ready is to cache the first part of every sample, as we've already described; but in the case of crossfading between multiple samples in real-time, what happens is that all velocity layers must be played simultaneously, as the user could move the MIDI fader at any time to crossfade to another velocity layer. Our samples have seven velocity layers, and since each note requires four voices to play back (because the samples are in quad) that means each note requires 7 x 4 = 28 voices to play back. Suddenly, 600 to 800 voices per system becomes 21 to 28 notes per system!Although these types of issues are inherent to all samplers, a particular restriction with Gigastudio 3 is that its sample playback engine can only make use of one processor core. So even if you buy a more powerful computer with four or eight cores, Gigastudio will still only use one of those cores to play back samples.