Wizards' Data Insanity

Tweet by SaffronOlive // Jul 17, 2017

standard

On Monday, Hour of Devastation prerelease events started on Magic Online at 11 am. Right around that time, an announcement slipped out of Wizards, not on the Mothership but on the much less viewed Magic Online website. The nameless article (attributed to "Wizards of the Coast" rather than a specific author) stated that, starting immediately, Wizards would be cutting the number of decklists published from Magic Online in half—from 10 per format per day to five per format per day. Along with reducing published lists, Wizards would also be changing the way the lists were selected. In the past, the 10 lists were chosen at random. Moving forward, the five lists will be random but with the caveat that there won't be any overlap (so the same archetype can't be published twice on the same day).

Wizards cited a couple of reasons for this change. First and foremost, Wizards believes that publishing too many deck lists from Magic Online causes the format to become solved too quickly and that cutting back on lists would solve this problem. Second, Wizards stated: "Since we have been presenting a random selection of top-performing decks, even if a deck doesn't have a particularly high win rate, it can appear to be extremely dominant if it's widely played. With only this information, it's not possible to disentangle win percentage and metagame percentage. This can lead, and at times has led, to feedback cycles where a deck appears more dominant than it would otherwise, which leads to an even greater percentage of play." This basically translates to, "The data we have been publishing is so bad it's messing with the metagame, so we're going to publish worse data to solve the problem."

These changes have been met with mixed reviews. While most people seem to think it's a bad choice, others think that this will open up Standard and make it easier for them to brew decks. There are also some serious questions about the reasoning behind Wizards' decision and the consequences (both intended and unintended) of this choice, so today, we're going to take some time to break down the issue of decklists and talk about the consequences of Wizards' decision to (once again) reduce data.

It's a Long-Term Trend

First, before getting into the nuts and bolts of the decision itself, it's important to realize that this isn't an isolated instance of Wizards trying to stifle data; instead, it's the latest in a long line of changes that work toward removing decklist information from the game, which is partly why this issue is so important. At this point, Wizards has firmly planted its flag in the "data is bad, and we want you to have as little as possible of it" camp, which is a scary place for the game to be.

A few years ago, StarCityGames published a series called Too Much Information, which broke down all of the decks and matchups from their own events (SCG Opens), and before too long, Wizards asked StarCityGames to stop. A couple of years ago, I wrote a series breaking down matchups based on information from Magic Online replays, and once again, Wizards stepped in and asked us to stop. During this same time frame, Magic Online went from publishing every winning decklist to 10 random winning decklists a day to now five curated decklists a day.

The point is that this week's announcement isn't an isolated, one-time incident—Wizards has a long track record of trying to stifle data, and not just its own data that it controls, but in the case of Too Much Information, data that is generated and published completely independently of Wizards on an independent tournament circuit. Based on this history, it wouldn't be at all surprising to see Wizards attempt to cut down even further on data in the future, which is one of the reasons why it's important to speak out on the issue now.

Data and the Metagame

Maybe the most interesting part of this entire issue is Wizards' claim that having published decklists solves the metagame too quickly. First, it's important to note that Magic Online is one of the biggest sources of data available (or at least, it was). While we have one or two big paper tournaments most weekends, the number of successful lists published from Magic Online overshadows the number of lists generated by the SCG tournament scene, the Grand Prix scene, and all of the smaller paper tournaments put together.

Now, it is true that some Magic players will automatically gravitate toward whatever list is putting up the most results, and having access to more and faster data from Magic Online likely causes these players to change decks quickly to the "best deck." The problem is that limiting Magic Online data isn't going to change the behavior of these players—if you're the type of person who wants to play whatever deck has proven to be at the top of the metagame, you'll just turn to Grands Prix, SCG Opens, and Pro Tours for your information. Basically, people gravitating to the top deck isn't a "too much data" issue; it's a style of play that a certain group of Magic players enjoy, and having less Magic Online data isn't suddenly going to turn these players into something they are not.

The other side of the issue is that having data actually helps the metagame evolve and develop. Magic isn't played in a vacuum, and the power level of any deck is based on the metagame you're playing it in. Brewing successful rogue and budget decks is about knowing the metagame inside and out, finding inefficiencies in the best decks, and then finding ways to exploit these inefficiencies.

Maybe the best example of this is the Turbo Fog deck we played on Budget Magic a little while ago. Turbo Fog is a metagame deck—it has the ability to win a lot of games if everyone is beating down with creatures and non-creature win conditions aren't a major part of the meta. On the other hand, there's a reason we didn't play Turbo Fog before Aetherworks Marvel was banned—the format was 40% (or more) Ulamog, the Ceaseless Hunger decks (which destroy Turbo Fog thanks to Ulamog, the Ceaseless Hunger's ability to exile away your library even through Fogs) and another 10% of 15% control decks designed to beat Aetherworks Marvel (which are also rough matchups for Turbo Fog, since they have so many counters and often win with planeswalkers or cards like Dynavolt Tower).

$ 0.00 $ 0.00 $ 0.00 $ 0.00

As such, having a good snapshot of the meta is essential for making Turbo Fog a deck. The same is true of other fun, rogue archetypes like Troll Worship in Modern. The archetype will struggle if everyone is sideboarding enchantment hate or playing Collective Brutality, but the archetype can be close to unbeatable when everyone is playing Affinity and Dredge. These changes will make our primary source of metagame data not just less plentiful but less reliable. It becomes significantly harder to brew new decks, which could very well lead to the exact opposite of what Wizards desires: a more wide open and less solved metagame.

The bottom line is this: there's a group of players who are going to play the top decks no matter what, and these changes don't especially hurt this group of players. Sure, maybe their "top deck" is a week old from a Grand Prix rather than a day old from Magic Online, but everyone else in the "top deck" group is also playing a week-old Grand Prix deck, so the playing field is equal. On the other hand, there's another group of players who want to brew decks to beat the meta, and this group suffers the most from the change, since with worse data, this group won't have as clear of a picture of what they are brewing to beat.

Older Formats

The metagame problem is even more significant with older formats like Legacy and Vintage, for which Magic Online data is the primary way that the metagame evolves and develops. While losing Standard and Modern data hurts, at least these formats are regularly supported in paper Magic, so Grand Prix and SCG data can help fill in the gaps. Meanwhile, Legacy data comes primarily from Magic Online, and Vintage data comes almost exclusively from Magic Online, which means these formats will screech to a halt along with the reduction of deck lists. While Wizards may prefer (rightly or wrongly) to slow down the development of Standard thanks to the "solved format" problem, older formats are already so slow to develop thanks to very powerful, established decks; a huge card pool; and a lack of high-level tournaments to incentivize pros to brew for them that losing such a huge chunk of data is a major negative. There's a real possibility that Vintage and Legacy become less fun to play and staler thanks to this change, which in turn could lead to fewer players buying into and playing these formats.

The One Percenter Problem

Wizards has repeatedly shown over the past couple of years that it wants a consistent group of players at the top of the game through changes like giving Platinum pros automatic invitations to the Magic Online Championship Series and handing out pro points at the very exclusive World Championship, which gives a small number of the best players a huge leg up on their competition. Cutting back on data just further strengthens the grip of this 1% at the top of the game.

I talked about this a bit on Twitter, and one of the common responses I heard was, "Well, the pros work harder, so they deserve the advantage," so let me see if I can explain the problem in a way that answers this argument. Let's say you are an aspiring pro player looking to "play the game and see the world." You manage to qualify for a Pro Tour thanks to the PTQ system or a Grand Prix. You take two weeks of vacation to prepare for the tournament and jam endless games on Magic Online. Let's say you play eight hours a day, six days a week for those two weeks. You've probably gotten in about 300 matches—a pretty good number to learn the meta.

The problem is that competing against you at the Pro Tour are a bunch of big teams of established pros who band together thanks to a combination of friendship and connections. Maybe you have 12 of these players working together. Even if they work half has hard as you (let's say, four hours a day for two weeks), they generate a dataset of nearly 2,000 games—six times as much as you generate working twice as hard—which is a much more meaningful sample size from a statistical perspective. As each player finishes their match, they type the result into a spreadsheet for the rest of the team members to see, and they have a metagame breakdown even better than what we had under the old "10 decks a day" system from Magic Online.

For the first player, having access to deck lists is essential. Even though they outworked the pro team, they are at a significant disadvantage because they are part of the out-group and not a member of the good ol' boys club that gets access to testing teams. They might at least stand a fighting chance to compete with the teams if they can put in 300 matches on their own and then fill in the games with data generated by other players, but their odds of performing well at the Pro Tour and getting "on the train" drop significantly without the supplemental data. Meanwhile, the pros (working half as hard, for the sake of this example) don't really care. The data they can generate on their own is already comparable to or even better than what is publicly available, so nothing really changes for them.

The bottom line here is that reducing data increases the gap between the "haves" and "have nots" in terms of competitive play and works to reinforce the same few names at the top of the standings. It hurts the less-established players, the grinders, and the PTQ winners since they don't have access to a testing team even if they want one, because testing teams aren't as much about skill level as they are about who you know. We can't really know for sure whether this change is an intended consequence of limiting data (which is possible, based on a string of previous changes designed to give established pros a leg up) or an unintended consequence, but there's no doubt that this change is great for the 1% of the competitive community and harmful for everyone else.

How Do You Know?

Another huge impact of the change is that it turns subjective, experiential knowledge into an even more important part of the game, which means content producers (especially pro players) will have an even more significant voice. In the past, having good data worked as sort of a safety net. Let's say someone writes an article saying that card X is going to bring deck Y to the top of the metagame. In the past, looking at results would give players a way to confirm these sorts of claims. Is the card putting up a bunch of good finishers? Then this claim is probably true. Are there zero lists playing this card? Maybe it's better to wait until we have some results before spending our hard-earned money.

Now, we simply don't know. Considering that five lists a day is likely less than 10% of the lists that 5-0 competitive Standard leagues each day, it's very possible that a card / deck is winning like crazy and you simply don't know about it. On the other hand, it's going to be very easy for people to overrate a card or deck that manages to show up a couple days in a row because the sample size is so small.

Beyond simple card evaluation, the system is ripe for manipulation. We've actually had instances in the past where people have published fake deck lists with the goal to get cards to spike for financial gain (I believe Dramatic Entrance was one example), and now this will be incredibly easy for unscrupulous people to do. Since there's less official data, the community, by default, will be forced to rely more on self-reported finishes on social media, and the way Magic Online is set up, there's no way to take a screenshot that definitively proves you 5-0ed a league with a certain deck. While it was possible to do this in the past, it was also much riskier because if you put together a string of good finishes with a deck, sooner or later, the odds were in favor of your list getting published, but now with such a tiny number of the 5-0 lists being published, it's very possible that you could 5-0 10 leagues and not get your decklist published once. Likewise, it's very possible that someone could say, "I've 5-0ed 10 straight leagues by adding random card X to random deck Y" and back it up with a convincing picture of treasure chests and Magic Online trophies (won with another deck) in an attempt to move the market on a specific card.

Magic Online Becomes Less Fun

One major reason to play lists on Magic Online is to get your decklist published on the Mothership. Not only does it provide a sense of accomplishment to see your name in lights, but it also gives players a way to legitimize their deck building and brews. As I mentioned before, it would be fairly easy for someone to "fake" a 5-0 in a league, but having your decklist published on the Mothership is undeniable proof of your accomplishment you can show your friends.

Getting a list published is also a huge boost for new decks. A lot of decks that become "real" in Modern and Standard start off as a single 5-0 list that was published on the Mothership. Then, more people start playing the deck, working on it, and developing it, and suddenly we have a new tier deck. A great recent example of this is the Mono-White Monument deck in Standard. The archetype started with a single 5-0 in a competitive league. The deck was published on the Mothership, more people picked it up, and it eventually put up a couple more finishes. Todd Anderson and some other SCG players then picked it up with a blue splash, and it's now the second most played deck in Standard. It's very possible that, under the new system, we wouldn't have the Monument deck in Standard at all. Imagine that the initial person 5-0's their league, but rather than being in the 10% of lists that get published, it's in the 90% that don't get published. No one picks up the deck because no one knows about it, and as a result, the entire evolutionary process of the Monument deck is skipped; instead of having an exciting and fun new archetype in Standard, everyone's still just bashing Temur Energy against GB Energy against Mardu Vehicles. This is a pretty clear example of how publishing lists makes Standard fun and diverse, instead of the opposite (which, remember, was Wizards' claim in its announcement).

Finally, this change may also have some very unintended and detrimental changes to Magic Online as a whole by pushing better players down into the lower level of leagues. I know for me, and probably for many others, the primary reason I choose to play a competitive league over a friendly league is that I'm hoping to 5-0 and get my list published on the Mothership. The expected value is actually better in friendly leagues at a 50% match win percentage, so now, with such a small chance of getting my sweet brew published even if I manage to 5-0, I'm likely to play more friendly leagues because it makes financial sense. If a large group of former competitive league players drops down to friendly leagues for the same reason, this will make these leagues less desirable for their intended audience and maybe even drive some newer and less experienced players out of leagues altogether.

Wizards: The Gatekeeper

$ 0.00 $ 0.00 $ 0.00 $ 0.00 $ 0.00 $ 0.00

There's no doubt that Standard has gone through a rough few months, with multiple bannings and some very poor formats thanks to Wizards making mistakes in research and development through cards like Felidar Guardian, Aetherworks Marvel, and Emrakul, the Promised End. While Wizards says this change is to improve the meta by keeping it from being solved as quickly, whether intended or not, what this change actually does is work to hide problems.

It's pretty easy for Wizards to argue against experiential, subjective experiences. Maybe it's not that Felidar Guardian combo is ruining Standard or Aetherworks Marvel is too good but that you had a run of bad luck, didn't build your deck right, or just aren't that good of a player. Having access to metagame data provides sold proof for players' arguments that Felidar Guardian and Aetherworks Marvel were too good for Standard. Not only did they feel bad, but when 40% of the decklists being published from Magic Online are of one deck, there's real, tangible, objective proof that players can point to of a broken format. We can even look back and see that, before Felidar Guardian combo, no deck had been 45% of the meta since Caw Blade, which provides a very strong argument that something is wrong with Standard.

Now, Wizards has reset the Magic Online metagame to be five decks that each make up 20% of the metagame by publishing only five different lists each day—perfectly balanced and diverse, the ultimate Standard format! It doesn't matter that 60% of the players at your local game store are playing Marvel; Magic Online (the primary source of decklist data in the Magic community) will show that perfect 20% balance. More troublingly, Wizards will have a cache of secret, hidden data that it can use to support its arguments, which the players are helpless to dispute because Wizards can just yell "fake news" or "conspiracy theory" at anything it doesn't like.

When players say that Felidar Guardian decks are too good, Wizards can just say, "Nope, that's fake news. Our data shows that Dynavolt Tower will save the metagame and Standard is actually amazing, no matter how little fun you are having playing the format." How can we, as players, respond to that? Until last week, at the very least we could point to metagame percentages as some sort of evidence to back up our feelings and experiences, but now this option is diminished because Wizards is the gatekeeper of all the data and can release whatever bits it wants (and withhold what it doesn't want) to support its arguments.

Monday

This would be problematic in the best of times, but it's doubly troubling right now because Wizards' credibility on metagame issues is shot. Remember, the data showed them that the Standard metagame was fine on a Monday when they didn't ban Felidar Guardian but was broken on Wednesday when they emergency banned Felidar Guardian. In the best case, this shows that Wizards isn't very good at using data to make meaningful decisions about the metagame (since the sample between Monday and Wednesday during a prerelease period—when nobody has the new cards to play with and are trying out weird, random decks—is close to meaningless), and at worst, it shows that Wizards is willing to use its secret, hidden data (which just became much more plentiful) to manipulate the player base.

Wednesday

Messaging

The idea that the reasons for this change might not be (at least entirely) the ones stated in the article on the Magic Online website is further backed up by the confused messaging coming from Wizards itself. For instance, the original article states that lacking data analysis is problematic and actually hurts the Standard meta: "Since we have been presenting a random selection of top-performing decks, even if a deck doesn't have a particularly high win rate, it can appear to be extremely dominant if it's widely played. With only this information, it's not possible to disentangle win percentage and metagame percentage. This can lead, and at times has led, to feedback cycles where a deck appears more dominant than it would otherwise, which leads to an even greater percentage of play."



However, when someone asked Mark Rosewater on Blogatog: "Are you concerned with the loss of access of information is going to drive people to gather their own data (which is inherently more biased)? I totally understand about the concern about a 'solved state', but what happens when perception overtakes reality and people believe the metagame is X and it's actually Y? (I have seen a lot of very bad analysis of metagames.)," Maro responded, "False metagame reads result in more flux in the system over time and delays reaching the solved state."

So somehow, false metagame reads are a justification for the data ban because they cause decks to be more popular than they should be, but false metagame reads also diversify the format by keeping it in flux and preventing it from being solved too quickly? Regardless of whether either of these things is true or not, the messaging is clearly poor and contradictory, which is scary in a time when Wizards is reducing our ability to research these things for ourselves by cutting back on data.

The bottom line is that by tightening its grip over metagame data, Wizards is essentially asking players to trust the company. Wizards is basically saying, "Don't you worry about the metagame; let us worry about the metagame." This is a strange request for Wizards to make of players at this point in time, considering the mess of the past nine months, not just with the bannings but some of the twisted (and apparently data-based) justifications to go along with various ban / no ban decisions. If things were going well, the decision to reduce deck lists could look like "Things are great, and this change will make things even greater by keeping the metagame from getting solved so fast" and perhaps no one would blink an eye, but with the rockiness of the last few months, it pretty clearly comes across as, "We're tired of you talking about your mistakes, so we'll take away the only objective argument you have, so we can pooh-pooh your subjective complaints as silly and not backed up by data."

The Solution

All in all, it's hard to find many positive things to say about this change. It hurts people looking to brew new decks, and while it could work to slow down the metagame to some extent, it also makes it more difficult for new decks to develop and succeed, so this seems like a wash. It hurts less established competitive players, who now have an even bigger hill to climb to compete with more established pros with testing teams that can generate their own data with ease. It hurts the community at large because we have less ability to come to our own conclusions about a format and instead have to rely on the information Wizards chooses to give us. It hurts Magic Online by taking away a major incentive to play competitive leagues and potentially pushing former competitive players into friendly leagues for the sake of value, and it hurts older formats that rely on Magic Online deck lists as an almost exclusive source of decklist data. Since the community will have less meaningful data from which to draw conclusions, it greatly increases Wizards' ability to shoot down criticism by using its secret stash of data as ammunition or as a shield to deflect important questions from the community.

Maybe most importantly, it's a huge step backwards in terms of transparency. It was just over a year ago that Wizards was publishing articles declaring its commitment to "being as transparent as possible," and while strides have been made in this area, this change works directly against this goal. Not only is releasing fewer deck lists in and of itself a reduction in transparency, but as we talked about before, the fact that the community will have less objective data on which to base its arguments and discussion of various formats will also enable Wizards to be less transparent about bannings and problems in the format.

So, what's the solution to the problem? Ideally, Wizards would release all of the Magic Online data. We've had extremely high-data Standard formats (back during Return to Ravnica and Khans of Tarkir) before that were amazing, which seems to work against Wizards' argument that having too much data hurts Standard. While I can understand Wizards' argument that too much data solves Standard too quickly, it's hard to agree, since the other side of the coin shows that having more (and better) data actually allows the metagame to develop and evolve, which in turn keeps the metagame from becoming solved. Rather than forcing players to play the best deck, in-depth matchup data actually allows players to figure out how to beat the best decks in the format.

What seems to be happening here is that Wizards is confusing the cause of its problems with a symptom of its problems. Felidar Guardian wasn't 45% of the field because players had too many deck lists; it was 45% of the field because Wizards messed up and printed Splinter Twin in Standard. Aetherworks Marvel wasn't 45% of the field because of decklists; it was 45% of the field because Wizards decided to add an entirely new resource system to the game and made it more powerful than it should have been. Basically, Wizards is using data as a scapegoat for its failings over the past several sets, preferring to point its finger at an exterior cause rather than back at itself. This is the easy way out and a decision that comes with the additional upside of insulating Wizards from criticism in the future. (Here, I should note that we have seen several positive changes come out of the bannings, and although this decklist data reduction is troubling, it doesn't take away these changes.)

What Wizards doesn't seem to realize is that releasing all of the decklist data would actually help to save it from a lot of these problems. While it's possible that this wouldn't have worked with Felidar Guardian or Aetherworks Marvel because Wizards decided to not print reasonable answers, in most Standard formats, having the full picture of the format would allow brewers to find weaknesses in the best decks, create new tier archetypes, and keep the metagame fresh and exciting. If, as Wizards claims, the lack of good data on things like win percentages is causing players to (wrongly) gravitate toward certain decks, then releasing matchup data would be a much more logical solution to the problem (rather than reducing and releasing even worse data). Instead of destroying Standard, data has the power to save Standard, if Wizards will let it.

The bottom line is that for the good of the game, Wizards needs the strength to try something new. It's been trying the "cut back on deck lists" technique for years now, and while correlation is not causation, it's worth pointing out that Standard has been getting progressively worse during the same time frame. Instead of heading back to the same old well in the hopes that somehow it will be different this time, despite all the other times it didn't work, Wizards should go the opposite direction for once. Release all the data, and see what happens. I believe it would make Standard significantly better, more diverse, and more fun for everyone. What if I'm wrong? Well, Wizards can always take the data away again. At this point, what's the harm? We just went through three separate bannings of five different cards in Standard—can releasing the data really make things worse?

Conclusion

While I hate the quote because it's so overused, it has been said that insanity is doing the same thing over and over again while expecting different results, which would technically qualify Wizards' handling of decklists and data as insane. Thankfully, there's still time for a change, and Wizards can easily do the sane thing by releasing all of the Magic Online data for the good of the game and betterment of the community.

Anyway, that's all for today. As always, leave your thoughts, ideas, opinions, and suggestions in the comments, and you can reach me on Twitter @SaffronOlive or at SaffronOlive@MTGGoldfish.com.