Dev Diary: Path of Exile Alpha Update A few weeks ago, we deployed a major update to the Path of Exile Alpha realm and invited a dozen or so lucky forum members (and many of their friends) to help test it. This development diary entry describes how we’ve handled the transition from development to maintaining a live game service that supports players playing from around the world. History of the Alpha Realm As an online game, gameplay in Path of Exile takes place on a “Realm” – a server farm that stores players’ characters and hosts the game server instances that they play on. For the first few years of development, we only had an internal testing realm in our office. Cheats are enabled on this realm so that developers can easily test new features and skip to the areas that they are working on. Although our developers would often play the game legitimately on this realm, it was difficult to show off items and accomplishments because they could easily be obtained with the developer cheats. In early 2010, Path of Exile was in a stable enough state that we decided to deploy it to some servers running in America, as the “Alpha” realm. Cheats are not enabled on this realm (not even for us), so it’s a fair playing field. Initially we just used this realm to test the game ourselves. We’re based in New Zealand, so we are able to do a lot of lag and synchronicity optimization when testing on a server that is so far away. Running the Alpha realm helped us improve our process of planning, developing, testing and deploying patches. Maintaining the servers as the player base grows larger helps us plan for our server requirements at launch. For the next year, we invited friends and family members to help us test the game. As we’ve discussed in several previous development diary entries, some of them played for scary quantities of time and gave us plenty of great feedback. As you can imagine though, with only a community of 5-6 really active players, the realm endured long periods of stagnation. We’d have a surge of activity each time we released a major patch, but it often dwindled down to only a few people playing. With the release of the 0.8.0 we felt we were ready to open the Alpha up to a larger group of testers. We wanted to try to simulate a much larger trade economy and a community where players would find and discuss balance issues in depth. We have a ladder that ranks people by experience (both in normal and hardcore leagues). This small feature is important because it creates an atmosphere of competition that encourages the testers to try hard to put in more hours than each other. Over the next few weeks, we invited people semi-randomly from both the active pool of forum posters and other people who have signed up for the beta. As it stands now, we have a pool of around 40 people who play often, which results in a concurrent player count of around 7 people online at once. We intend to invite more people in a similar way over the coming weeks. [/frame] [frame width="360px" bMargin="5px"][/frame] [/frame] [frame width="360px" bMargin="5px"][/frame] [/frame] [frame width="360px" bMargin="5px"][/frame] [/frame] [frame width="360px" bMargin="5px"][/frame] Collating Feedback One of the primary reasons for running an Alpha test is to get feedback from external players. This feedback can range from screenshots of visual problems to lengthy essays on how they feel a certain aspect of the game could be improved. It’s very valuable to read this feedback, because the development team is so used to how the game currently plays. Hearing from Alpha testers helps us learn what we’re doing right and find out early about problems that we haven’t noticed ourselves. Collating the feedback from testers is harder than we had predicted. We have an internal bug tracking system with hundreds of open issues, so it takes time to check to see if each issue is already logged (and if so, adding any new information to it). For issues that we don’t currently know about, they often have to be reproduced by Quality Assurance so that we have useful reproduction steps before allocating them to someone to fix. When an obvious problem makes its way into a deployed version, it’s normal for most testers to report it simultaneously. We don’t mind, of course, as it means they’re fulfilling their role as Alpha testers properly. This can result in a large amount of feedback to read through, but will pay off with a polished game. It’s fitting punishment that when the server goes down, we get deluged with IMs. There’s an Alpha forum for both bug reports and discussion of game features. Some systems are heavily under debate and often get redesigned based on player feedback. For example, we’re constantly iterating on the passive skills screen. Our most recent attempt is too confusing and overwhelming for new players. We have a much improved layout in the works which should hopefully make everyone happy. Without the Alpha feedback, we might not have realised so soon that such a drastic change was needed. It's great when Alpha testers post their impressions of the game to our main forum for the other community members to read and discuss. Server Monitoring The other important reason to run an Alpha realm is to prepare the servers for the onslaught of users that our Beta will introduce. We want to monitor and graph everything that we can about the health of both our physical servers and the game server software that we run on them. We’ve augmented our backend servers and game servers with various monitoring hooks that read plenty of information about current load levels and play statistics. These values are exposed in a way that allows monitoring software to access, log and graph them. We initially used Cacti for this monitoring. We’ve recently switched to Zabbix, which meets our needs better. Both are excellent open source server monitoring systems. It’s very easy to have one web page open on a monitor that shows all of the crucial information about our servers. Some of the stats we currently graph are the number of current players/parties/instances and amount of logging/chat/bandwidth used. Based on this monitoring, it’s very easy to see when things are going wrong with the realm so that we can fix them. We’ve used it to diagnose several problems already, and are constantly adding more monitors so that more information going forward. It’s also really handy to be able to get an at-a-glance summary of what players are doing. [/frame] [frame width="360px" bMargin="5px"][/frame] Patch Development and Deployment We are currently trying to deploy updates at least once per week. Each patch starts with a session where we identify the high-priority issues that are currently affecting our players. These could be server stability issues, client crashes, or some gameplay imbalance that needs addressing. Any currently-in-development features such as new quests or skills that are near completion are also flagged for inclusion in the next patch. Using our issue tracker, we identify this set of features/bug-fixes and mark them with the version number of the patch we’re developing. As the team implements features or fixes bugs, they get tested by our Quality Assurance guy. Once most of the issues are complete (or the deadline for the patch is approaching), we “tag” a version of the game with that patch number. This version is essentially frozen in time, allowing extensive testing to occur on it. Fixes to any broken changes (or last minute changes that we want to include) are merged into this tag without introducing instability from untested features that are constantly being added to the mainline development copy of the game. At around this time, we post the patch notes to the Alpha forums to give people advance notice of the changes. Sometimes they remind us about critical fixes we promised and forgot to include. It’s often possible to make last minute tweaks to issues after this stage while it is still being tested. Of course, it’s dangerous to change too much stuff at the last minute without adequate testing. Once Quality Assurance is happy with the build of the game, we deploy it to the Alpha realm. This process can take several hours due to the amount of uploading required from New Zealand, but it can be substantially faster if the changes don’t require lots of new art. With the new build live, players flood back into the servers to test the changes and see how well their characters cope with the new balance. If we’re lucky, they find all of the newly introduced problems quickly (in time for a hotfix that can be deployed within an hour). If we’re unlucky, it’s a phone call at 2am. The Future Our plans are to continue to deploy frequently updates and invite more community members to help us test Path of Exile. We’re intending to enter Beta in August, and will be demoing Path of Exile with a booth at the PAX Expo in Seattle that month. If you’d like to sign up for beta access now (as well as a chance at a lucky alpha slot!) please sign up here.