The Problem

I tend to be a big advocate of strong and frequent communication with video game players.

Partially, this is because I think frequent communication provides a more human face for a development team, making players more tolerant of the inevitable missteps and more understanding of the limitations of the development process, as well as tending to improve the overall tenor of a game’s community. People like knowing that they’re being listened to, even when you often have to say, “Sorry, we can’t do that for X and Y reasons”. Partially, as well, frequent communication helps the developer by providing a strong stream of subjective feedback from a game’s most passionate players which is a good way of identifying legitimate problems that should be investigated and addressed.

It is true that the players who comment on forums, for example, represent a distinct minority, but even if they are not a demographically precise sampling of a game’s user base, if a significant majority of these players, self-selecting as they are – are evincing pain, it’s probably something worth looking at. Players (and sometimes executives) frequently lack the necessary context to be able to identify what are realistic solutions and what are not, so where a developer is willing to open up and communicate more aggressively, the average quality of proposed solutions tends to go up.

One of the areas where players and developers often clash is over the speed of changes, new content, and bug fixes. Developers are (rarely) actually lazy, but the process and the pipeline to minimize risk of deployed issues is generally quite intense, but most players have very little insight into this. To a player, a bug that sits around for a month before being fixed can conjure up an engineer sitting with his or her feet up sipping Mountain Dew while playing League of Legends, when the truth is (usually) that there are a lot more steps a fix has to go through than players are aware of.

The exact process varies from development studio to development studio – length of release cycles, platform, company development philosophy, and other considerations all have a major impact on the exact structure, but here’s one example – in this case, for Gazillion‘s Marvel Heroes, where I currently work as the game’s Lead Content Designer – of some of the specifics of a development and deployment pipeline for those who have never been exposed to something like this.

The Pipeline

Gazillion’s development pipeline can be thought of as the synchronizing of two interlocking wheels, sort of like an Aztec calendar. One wheel is the code and data set that developers (designers, engineers, artists, audio folks, etc.) are all working on (the Development wheel), while the other wheel is the production deployment (the Deployment wheel).

Both wheels are turning at the same time, but they interlock with each other only occasionally; periodically, the deployment wheel takes a snapshot of what is on the development wheel, kicks that off to QA who then reports on what critical issues there are. Developers then have to fix – sometimes on each wheel separately through what is called a merge, sometimes directly into the wheel – any found issues.

The deployment wheel turns again and kicks off another wheel, it goes back to QA, rinse and repeat until you have a “good” build. Obviously, this takes a variable amount of time based on what issues arise. Once there is a good build, the deployment wheel turns yet again, and this time pushes it to the Test Center on a white list. Rinse, fix, repeat. Then another turn of the wheel and it goes to the Test Center sans white list. Rinse, fix, repeat. Once that’s finally good, then it goes to the Live service.

All this time the development wheel is turning, but you can see here how the development and deployment wheels can – depending on the timing – either be in synch or out of synch by as much as two – or even more – weeks.

A two week cycle is in the industry very aggressive and very risky; more common (and more safe) is more like a month to two months. Any single gamestopping issue can halt the entire deployment wheel, and until that issue is unjammed, it can’t go forward.

Looking at the patch notes is, as well, only a part of what is going on. The patch notes include everything that is (or is supposed to be, at least) player-facing, but there are a ton of things that go on under the hood from fulfillment services, integration with outside services (like Steam), website integration, server and client optimization, and so on. Any of these things can halt the entire process which is why it is impossible for us to absolutely guarantee a deployment date for any specific piece of content.

For major releases, as you can imagine, things can get very tight and very stressful since there you have an enormous amount of external visibility from the community, from Marvel, from Gazillion’s board, and from game sites. To be sure, we bake in as much extra time as we can to account for things, but by definition you can’t anticipate a Black Swan event.