Hey folks! I’m Jane “Wilwariniel” Jeffers, QA Architect on League of Legends, and we interrupt your regularly scheduled Tech Blog to bring you the first installment of the Rito Bug Blog Series!

I’ve been working on League of Legends for quite a while now (seven years, to be exact), and let me tell you: I’ve seen some… stuff.

Why a Bug Blog Series?

So, why do Bug Blogs? Well, at Riot, we believe that a fundamental component of the development process is understanding that failures can and do happen. We make mistakes because we’re humans, and humans are far from perfect. And that’s okay! What’s important is to recognize failures for what they are, work to understand why they happened, learn from them, and make needed changes based on those learnings. That’s how we grow and that’s how we get better.

With this in mind, we want to share some of our more interesting bug challenges so that others can see how we work through them, both in the short-term and in the long-term. After all, in my experience, the true root cause of a defect oftentimes isn’t necessarily in the code or scripts itself, but somewhere in our tools or processes. Don’t worry, though, we’ll cover the nitty-gritty technical details, too. Our Bug Blogs will cover especially impactful bugs, ones that were difficult to triage and solve, and the ones from which we learned and grew the most.

We hope that by sharing these stories, we can help others facing similar struggles to totally avoid making the same mistakes that we did, or provide some ideas for solutions when things have gone sideways. And, I mean, c’mon, who doesn’t like to cringelaugh along with a really good bug story? I know I sure do.

So, for our very first post, let’s take a look at the one-two mega-punch that happened at one of the worst times possible. We learned a lot from this experience, and I hope that you can, too. This is the Bug Blog for League of Legends patch 4.18, aka the Worlds 2014 patch.

Context on Worlds Patches

Generally speaking, during a League of Legends World Championship, we actually have at least two patches to maintain. There’s the patch the pros play on during the tournament, which gets locked down earlier to ensure extra stability, and then the patch for the general playerbase, with more recent changes and content. Usually, this means the tournament is a patch or two behind what’s on live. The patch we’ll be talking about, the 4.18 patch, was what was live for our players for Worlds 2014.

That year, the World Championship grand finals were held in Seoul, South Korea, and a number of folks from the League dev team went there to experience it and to really interact with our playerbase in person. This way they could get a firsthand account of what players want, need, and get the most excited about.



Worlds 2014 Grand Finals in Seoul, South Korea. So many players!

I volunteered to stay behind and be the QA Lead for the Worlds patches with a small crew so that we could jump on any problems that might arise ASAP. We had put a lot of work into these patches to prepare for Worlds and for a chunk of the dev team being gone, so the expectation was that it would be a relatively quiet week.

Heh… quiet.

Round 1: Fight!

The first full week that patch 4.18 was live, we started to see the first reports of things being, well… a little wonky for a couple of champions. Messages began to pop up on the boards and social media about Kassadin and Nidalee in particular not behaving properly. Players said they weren’t doing damage when expected, or they were performing abilities in weird directions. This meant they were losing clutch fights. Not cool.

Our Network Operations Center technicians jumped on these reports and started paging out to the people on-call right away. While Production and Insights pulled the winrate numbers for the affected champions to figure out impact, the rest of us started working on reproducing the issue.

We narrowed in on the problem having to do with using certain abilities in a specific order. Performing a move block spell into a cone spell was key, which explained why Kassadin and Nidalee were experiencing the defect much more than other champions. Their respective kits practically depend on this combo.

When the bug happened, the champion would move to their new position with the move block, and then fire off the cone spell ability backwards, in the direction from which they came. It was confusing to the player, and felt pretty damn awful. With this knowledge, and the fact that the winrates for both champions were down 1-2% (a pretty hefty nerf from an unintended change), we made the call to disable first Nidalee and then Kassadin.



The bug in action. Nidalee performs her move block spell, Pounce (W), into her cone spell, Swipe (E). Watch how she turns when performing the Swipe.

But we were still missing something. The higher the skill level of the player, the more likely the defect was to occur. Did it have to do with timing? Positioning? Did an earlier order of events need to happen? As we continued to investigate, the list of potentially impacted champions kept growing. Although not every champion in the game has their own built-in movement abilities, every player has access to the summoner spell Flash, which blinks a character to a target location. Champions like Darius and Annie frequently use Flash and a cone spell to engage in fights, for example, and they were experiencing the issue too. We weren’t just looking at two impacted champions, but every champion with a cone spell ability.

Fortunately, we had a breakthrough in our testing and realized what the missing element was: it all had to do with spell queuing.



No, not that Q.

Under the Hood

In League, players can queue up abilities to more effectively carry out combos or pull off some sick escape plays. This is particularly effective when a player wants to prepare an ability to execute after a non-instantaneous ability completes. The queued ability is postponed until the first ability completes, and then the queued ability is cast. A move block spell is non-instantaneous because it takes the time to move the champion from Point A to point B. So one popular combo strategy is to queue up the next ability (like, say, a cone spell) so that it triggers as soon as the move block is complete. For cone spells in League, the ability originates from the player position. In the case of a move block into a cone spell, the game should activate the postponed spell after completing the move block, using the player position from that completed move block. Instead, with this defect, an erroneous line of code was capping the target position of the cone spell at the time the ability was queued up, instead of storing the uncapped position and applying that cap when the postponed spell was brought back into play. In other words, it would take the positioning from mid-move and try to cast the ability in that direction due to cast range restrictions. Hence Nidalee, Kassadin, and other champions effectively cast their abilities backwards.

The fix for this was relatively simple: remove the erroneous line of code. But the change impacted a script that was used for more than just cone spells. To make sure we didn’t accidentally break something else, we needed to perform thorough regression testing.

We ran a grep with Agent Ransack and found the script in a whopping 293 .ini files.

In other words, 293 different champion abilities, summoner spells, and items needed to be checked. It was going to be a long day of testing. The Insidiousness of Tech Debt

It was during testing where tech debt really slowed us down. We encountered two big tech debt problems with this fix.

What’s in a Name?

The first big problem had to do with best practices. Today, League of Legends files follow specific naming conventions. This makes it easy for both humans and the game to identify what a thing is and what its purpose should be. AatroxE.lua, for instance, is the .lua file for Aatrox’s ability which is by default on the E hotkey. This makes it easy for anyone to find his E ability in the lua files, even if they aren’t familiar with the ability’s name.

Back in 2014, though, we had only just started really using standardized naming conventions. Out of the 293 .ini files we found in the grep, the names were all over the place. Some followed a pretty clear standard (GarenW.ini and GarenWpassive.ini), some at least tried to be descriptive (MissFortuneViciousStrikes.ini), and some… well, try identifying what ZhonyasRing.ini could be when it’s not an item in the game and you’re racing a ticking clock. Spoiler: only one person on the triage team remembered that it was an old item during League’s beta days. It was split into Zhonya’s Hourglass and Rabadon’s Deathcap, and surprise! Rabadon’s Deathcap kept the ZhonyasRing.ini file.

We made a spreadsheet of all 293 of these .ini files and went through it, identifying each ability, spell, or item based on our collective tribal game knowledge.



More identity than the files had ever felt before.

These all had to be manually tested, mind you. For something so feels-based in gameplay, automated testing was pretty much out of the question (and probably would have taken us longer to set up, anyway). Fortunately, we had some really good League players on the triage team to handle more nuanced test cases. And we also were able to utilize one of our QA outsource partner teams to help out with the heavy manual work. We ran quick checks in-house, and then sent it over to them for more robust confirmation and regression testing.

And yes, all of that regression testing was absolutely necessary. Six other champion abilities broke and designers had to edit their scripts individually to make the fix work.

Majestic Waves of VFX

The second tech debt problem we faced was related to an issue we identified during regression testing. Initially, we thought this was a knock-on (side-effect) bug due to how similar it appeared to the original cone spells bug. We pulled our hair out trying to investigate this new bug and figure out how it could have been introduced by the fix, when it turned out to be an old, old issue that had existed in League since… well, as long as anyone could remember.

Since the fix for this bug was the very first changelist he submitted to League of Legends, I’ve asked the crazy-talented Software Architect Brian “Riot Penrif” Bossé to explain the technical details of this juicy beast. Penrif?

Right! This was a fun one. The kind of fun that sticks with you for around 4 years. After we fixed the case of Nidalee constantly swatting at flies, clever people found that if you flashed and cast the cone spell exactly where you flashed to, strange things happened:

So here’s Cho’Gath after flashing in the direction of the red arrow. His subsequent cone-shaped scream hits where you’d expect, but the client shows majestic waves of VFX headed seemingly wherever they’d like to go. This is one of those cases where the same code exists on the client and the server. If your spell has no discernable direction because you cast it exactly on yourself, the game defaults to using whatever direction the character is facing. Except, clearly, the client thinks there is a real direction, and it’s rubbish. To know why, we need to go on a little journey together.

League uses 32-bit floating point numbers for positional information in the game. Fine and reasonable when computing locally, but constantly chucking around full resolution 32-bit values over the network really adds up, so we use some basic compression to squeeze them down a bit. Basically we just map the extent of the map into a 16-bit integer, and ka-bam, it’s half the size with no noticeable loss in fidelity.

Problem is, we don’t always compress positions like this when we communicate between server and client, we really just do it for unit positions. So, cast positions, to take a random non-specific example, still get the full 32-bit floating point treatment. When the cast position for Flash gets interpreted into a unit position and sent over the network, there's a difference in those positions due to the compression.

Now, when you cast a cone spell with exactly the same cast position, that compression error can now be interpreted by the client as a valid direction for the cone! Easily remedied, thankfully, by changing the shared logic to fall back to the facing direction if the cast position and champ position would both compress to the same compressed position. That way, whatever we do to the network compression set-up, this bug will never come back. And indeed, that’s the first change I ever made to League of Legends. Back to you!

Thanks, Penrif! Wilwariniel here again. This bug’s similarity to the first issue - a cone spell not casting properly - threw us off-track for a while. Once we figured out that it was, in fact, an old, unique issue, we had to weigh whether or not it was worth packaging the fix in with the first bug. How likely was it to happen? Had it disrupted the current state of the game? How risky would it be to include it with the other needed fix?

Ultimately, we decided to defer the fix for that bug to the next patch, when we would have more time to test around it and get things right, instead of potentially introducing more risk while Worlds was going on. But we still spent that triage time and mindshare on the problem. Even smaller tech debt issues like these can really slow things down.

But Wait - There’s More!

I bet you thought that weird compression issue we dealt with was the second punch in the one-two mega-punch I mentioned earlier, huh?

NOPE! Patch 4.18 wasn’t quite finished with us, yet.

Actually, that patch was pretty bumpy. Here’s a picture of our amazing Senior Release Manager Donna “Feithen” Mason in front of the live triage issues board:

Notice the looong column of cards on the right? Those are all problems with the patch. Uh, sorry, Feithen. Anyway.

Enter the Exploit