Marvel’s Spider-Man: Dynamic Music System

With great audio comes great gameplay

Marvel’s Spider-Man was a huge success. Not only did it sell 9 million copies (as of November 2018), becoming the PS4’s fastest-selling exclusive title, but it was also nominated for 36 game awards. While I could write about the many reasons I love this game, this post is going to focus on its impressive adaptive music system, specifically how it matches the cinematic music tropes of a typical superhero film and tailors them to a player’s unique gameplay experience.

In case you haven’t played it, Marvel’s Spider-Man is a triple-A, open-world action-adventure game following a story of Spider-Man in which the player controls the titular hero and is free to roam New York City. Whenever the player begins to swing around Manhattan, John Paesano’s epic score swells to create incredibly cinematic gameplay, comparable to Spidey’s movie iterations.

Kotaku’s Jason Schreier describes this feature as having a “…massive impact, making it feel like you’re entering a superhero movie every time you start moving. It’s the type of thing you might not even think about, but it’s just as important to the overall sensation as the way Spider-Man’s webs look or feel. And it’s just one reason this game is so great.”

Here’s an example of the gameplay:

So, what is adaptive music?

Video games are increasingly referred to as being ‘cinematic’. With this, there is a desire for video game music to also use techniques found in cinema and film, such as emphasising dramatic moments and establishing character motifs. The difficulty with this is that unlike movies, games follow a non-linear format.

Whether it’s a film score or a song on an album, most music is linear as it typically has a beginning, a development and an ending that sound the same no matter how many times the piece is played.

Games are non-linear because the narrative structure, pacing and even outcomes of the game are constantly changing because they are controlled by the player. As this is unpredictable, adaptive music systems must be used to respond to the player at any given moment.

Here’s a great example of an adaptive music system:

There are a few different types of adaptive music, though the two most popular techniques are ‘parallel forms’ and ‘transitional forms’. I’ll keep the explanations brief.

Parallel techniques

Parallel form (sometimes referred to as vertical re-orchestration) is the adaptive technique in which composers break up their score into separate musical layers, such as by instrument families (strings, percussion, horns, etc.). These layers are then linked to various game parameters, such as the player’s health, the time left in a level, the time of day and the number of enemies around. During gameplay, layers of the score are stacked on top of each other and played together, following the same tempo and key signature as each other to work coherently.

So, if the player was engaged in combat with many enemies, there may be lots of layers of instruments playing at once to build the intensity. If there were only a few enemies, there would be fewer instruments playing, making it less dramatic.

The result is music adapting to the player’s actions and encounters which, if done effectively, will seem aware of the player, creating an exciting personal soundtrack to their gameplay.

Transitional techniques

Transitional forms (sometimes referred to as horizontal re-sequencing) involve stitching together different pieces of music based on the player’s actions.

These cues are often cross-faded (fading the volume out of one cue as the new cue fades in), though there are a variety of other methods, such as ‘bridging’ which involves playing a small transitional music segment between the other pieces, or ‘phrase branching’ which waits for a musical phrase to finish before beginning the next one.

The transitional technique typically favours the composer over the programmers, with the focus placed on the music itself rather than how it will integrate with the gameplay. The fact that the score isn’t typically layered allows the composer to use different tempos and key signatures too which can often make for more variable but slightly less interactive soundtracks than parallel forms provide.

(A more in-depth explanation of these techniques can be found here.)

Spider-Man’s music system

So which technique does the Spider-Man music system use? It actually uses a mix of the two. I emailed Rob Goodson, the music editor on the game, to ask about the system and he very kindly detailed how it worked.

“The music system in Spider-Man utilizes a combination of techniques. In a nutshell, the system is tracking a number of variables that determine music intensity, including player altitude, player speed, and length of time spent swinging. Each piece of music that you hear in the open world has 3 levels of intensity that transition between one another depending on how high you’re swinging, how fast you’re going, etc. When your speed drops to 0, we trigger the end stinger that brings the music to a natural-sounding conclusion.”

From this, we can see that the Spider-Man music system uses a combination of the aforementioned techniques. It uses parallel forms with three layers which are activated by various parameters and then uses a transitional technique to crossfade into the end stinger or a different piece entirely. Building the intensity of the score in relation to the players swinging is an effective way to let them know they are performing well and is one of the driving forces behind making you ‘feel like Spider-Man’ (something mentioned so much by reviewers of the game that it became a meme).

Goodson acknowledged the difficulties of using the transitional technique, explaining that the composer will write their music to make the editing process much easier.

“It can be challenging to make these transitions sound natural, which is why we try to have our plan for the adaptive music system inform the composition itself. We give a composer a set of parameters to work within that makes things easier on the editing and integration side. Once the suites are composed/recorded/mixed/etc, we carefully edit them and extract the pieces we need to make the music systems sound as seamless as possible.”

I then asked if composers are happy to write within these parameters or if they see them as a hindrance to their music.

“In our experience, composers are usually happy to work within whatever parameters are necessary. In the fact, they often appreciate very specific direction because it cuts down on iteration on their end if they know exactly what we need. Also, outside of the suites that we use for the music systems, there are plenty of opportunities for them to get creative — writing character themes, custom scoring cinematics, etc. In the case of Spider-Man, when it came to composing the open world gameplay systems, we had already worked with John Paesano for over a year developing character themes, so then it was just a matter of adapting those themes into the framework of the music system suites (you may have noticed that the 5-note Spider-Man theme itself is woven into almost every piece of music in the game, and the open world music in the second act re-casts the Martin Li/Mr. Negative theme in several ways).”

While it’s slowly becoming more common, composers aren’t usually brought in to work on the project until the later stages of development. However, John Paesano explained in an interview that he was brought on from a ‘very, very early stage’ which will have allowed the team the opportunity to develop and tweak the system to such a high standard.

Goodson actually stated:

“The music system did take a while to perfect, and went through a number of iterations throughout the development cycle. In fact, we spent the better part of a year just tweaking and fine tuning the system, all the way up until the game was released”.

Most of this tweaking and fine-tuning likely took place in the middleware, ensuring that every potential transition the player could hear would sound as natural and smooth as possible.

“Once the editing is complete, we use audio middleware (Wwise, in the case of Spider-Man) that, among other things, allows us to track the tempo and meter of each piece of music and set rules for how different pieces of music will transition between each other. We can set them to transition on a beat, barline, or any other subdivision we want, set the fade in/fade out times, and even play a transitional stinger to bridge the gap between the two pieces”.

A good example of the finely tuned transitions can be heard when diving from a tall Manhattan skyscraper. As Spider-Man falls closer to the ground the score builds in intensity. When the player fires their first web to begin swinging, the music system immediately reacts and begins playing the next cue, though it still sounds natural. Here are two examples I recorded:

Final Thoughts

The adaptive music system featured in Marvel’s Spider-Man set a very high standard for AAA Superhero game scores. By bringing John Paesano in early, Insomniac clearly demonstrated they understand how important good dynamic music is within a game and I believe it is a huge contributor to what makes the overall game so great.