Hi, I’m Jim ‘Anodoin’ Merrill, and I work on test automation efforts for League of Legends, focused specifically on the in-game experience. I currently serve as the tech captain to the Build Verification System Development (BVS-Dev) team. In large part, our team builds tools for automated testing and helps teams write better tests.

For the past couple of years, we’ve been working on getting our test system and infrastructure up to snuff in order to increase developer efficiency and reduce the number of bugs we ship. We now run approximately 100,000 test cases a day, and automated testing at that volume helps get content to players sooner and with fewer bugs. I’d like to share a little bit of what we’ve done, and hopefully start a conversation about automated testing in the game space.

Why Do We Care?

League changes really, really quickly. On average, we see well over 100 code and content changes checked into source control every day, and providing adequate coverage for all of those changes is a challenge. With a new patch every two weeks, it’s critical that we detect defects quickly. Bugs discovered late in the release process can cause delays, lead to redeploys, or require temporary champ disables—all bad experiences for players. Automation frees up our quality analysts to focus on more creative testing and upstream defect prevention, where they can provide more value.

Automation also provides faster turnaround for test results. It isn’t feasible for humans to run a full test sweep on every new code or content submission, and, even if it were, it would require an army of testers to return results sufficiently quickly.

Our test system runs on continuous integration (CI) and reports back within about an hour of check-in. That means that developers receive results in a reasonable timeframe, which helps reduce context switching; in fact, bugs discovered in automation get resolved eight times faster than the average bug. Better yet, if we need to increase our throughput for tests, we can simply add a few more executors to our test farm.

The Build Verification System

The imaginatively-named Build Verification System (BVS) is our test framework for the game client and server. It's responsible for acquiring artifacts to test, deploying them onto a test machine, starting and managing systems under testing, executing tests, and reporting on their results. The tests and harness are written in Python, and we wrote most of the BVS code to insulate test-writers from the complexities of gathering the required resources. As a result, a few arguments in a test class can specify what map to run, how many clients to include, and what champions should be in the game.

Tests make use of remote procedure call (RPC) endpoints exposed on the client and the server in order to issue commands and monitor game state. For the most part, tests consist of a fairly linear set of instructions and queries—existing tests cover everything from champion abilities to vision rules to the expected rewards for a minion kill. Some of our earlier tests were significantly less linear, but that made working with the system much harder for less-technical developers.

Since all the work of configuring a test workspace is done separately, the tests themselves should look the same whether running in a local workspace or in our test farm. This makes it easy to run tests locally while making changes to the game.

For example, our test for the damage dealt by Kog’Maw’s new W looks like this:

""" Name: BioArcaneBarrage_DamageDealt Description: Verifies the damage modifications from Bio-Arcane Barrage Verifies: - KogMaw deals less damage to non-lane minions - KogMaw deals percentile magic damage - KogMaw deals normal damage to lane minions """ from KogMawAbilityTest import KogMawAbilityTest from Drivers.LOLGame.LOLGameUtils import Enumerations import KogMawStats class BioArcaneBarrage_DamageDealt(KogMawAbilityTest): def __init__(self, championAbilities): super(BioArcaneBarrage_DamageDealt, self).__init__(championAbilities) self.ability = 'Bio-Arcane Barrage' self.slot = KogMawStats.W_SLOT self.details = 'Kog\'Maw deals reduced base-damage to non-minions with additional percentile damage' self.playerLocation = Enumerations.SRULocations.MID_LANE self.enemyAnnieLocation = Enumerations.SRULocations.MID_LANE.angularOffsetDegrees(45, 200) self.enemyMinionLocation = Enumerations.SRULocations.MID_LANE.angularOffsetDegrees(45, 400) def setup(self): super(BioArcaneBarrage_DamageDealt, self).setup() self.enemyAnnie = self.spawnEnemyAnnie(self.enemyAnnieLocation) self.enemyMinion = self.spawnEnemyMinion(self.enemyMinionLocation) self.teleport(self.player, self.playerLocation) self.issueStopCommand(self.player) def execute(self): self.takeSnapshot('preCast') self.castSpellOnTarget(self.player, self.slot, self.player) self.champAttackOnce(self.player, self.enemyAnnie) self.takeRecentDeathRecapSnap(self.enemyAnnie, "annieRecap") self.resetCooldowns(self.player) self.castSpellOnTarget(self.player, self.slot, self.player) self.champAttackOnce(self.player, self.enemyMinion) self.takeSnapshot('minionRecap') self.teleport(self.player, Enumerations.SRULocations.ORDER_FOUNTAIN) def verify(self): # Verify that enemy Annie is taking the correct amount of damage. annieAutoDamageEvents = self.getDeathRecapEvents(self.player, "Attack", "annieRecap") annieAutoDamage = 0 for event in annieAutoDamageEvents: annieAutoDamage += event.PhysicalDamage annieSpellDamageEvents = self.getDeathRecapEvents(self.player, "Spell", "annieRecap", scriptName=KogMawStats.W_MAGIC_DAMAGE_SCRIPT_NAME) annieSpellDamage = 0 for event in annieSpellDamageEvents: annieSpellDamage = event.MagicDamage AD = self.getStat(self.player, "AttackDamageItem") expectedPercentile = (KogMawStats.W_AD_DAMAGE_RATIO * AD)/100 annieTotalHealth = self.getStat(self.enemyAnnie, "MaxHealth") expectedPercentileDamage = self.asPostResistDamage(self.enemyAnnie, expectedPercentile * annieTotalHealth, 'MagicResist', snapshot='preCast') self.assertInRange(annieSpellDamage, expectedPercentileDamage, expectedPercentileDamage * .1, "{} magic damage dealt. Expected ~{}".format(annieSpellDamage, expectedPercentileDamage)) expectedPhysicalDamage = self.asPostResistDamage(self.enemyAnnie, KogMawStats.W_NON_MINION_DAMAGE_RATIO * AD, 'Armor', snapshot='preCast') self.assertInRange(annieAutoDamage, expectedPhysicalDamage, expectedPhysicalDamage * .1, "{} physical damage dealt. Expected ~{}".format(annieAutoDamage, expectedPhysicalDamage)) # Verify that enemy minion is taking the correct amount of damage. AD = self.getStat(self.player, "AttackDamageItem") minionExpectedPhysicalDamage = self.asPostResistDamage(self.enemyMinion, AD, 'Armor', snapshot='preCast') expectedPercentile = (KogMawStats.W_AD_DAMAGE_RATIO * AD)/100 minionTotalHealth = self.getStat(self.enemyMinion, "MaxHealth") minionExpectedMagicDamage = self.asPostResistDamage(self.enemyMinion, expectedPercentile * minionTotalHealth, 'MagicResist', snapshot='preCast') expectedDamage = minionExpectedMagicDamage + minionExpectedPhysicalDamage actualDamage = self.getDamageTaken(self.enemyMinion, 'preCast', 'minionRecap') self.assertInRange(actualDamage, expectedDamage, 1, "{} total physical and magic damage dealt. Expected ~{}".format(annieAutoDamage, expectedDamage)) def teardown(self): self.destroy(self.enemyAnnie) self.destroy(self.enemyMinion)

The first part of Kog’Maw’s suite of tests, including the Arcane Barrage damage test, looks like this:

Your browser doesn't support video.

Please download the file: video/mp4 When a test finishes a run, it provides the results to a separate reporting service, which stores run-data going back approximately six months. Depending on the source of given test data, this service takes different actions. A local run of a test opens a webpage on the executing machine that details the passing and failing cases. A run in the test farm, however, will create new bug tickets for any discovered issues, tag artifacts according to results, and send an email to committers if there are any failing cases. Test data is also aggregated and tracked via the reporting service, allowing us to see when test failures have occurred, how often they occur, and how long it has been since a passing build.

In Wood 5 we don't use wards anyway, so I see no problem with this critical failure