In a podcast, the HAWPcast, 1.5 years ago the creators of Super Meat Boy discussed many things but spent a great deal of time discussing their experience with the Independent Games Festival (IGF) (~6:00-41:00) as both entrants and judges. (Thanks to user fwahfwah at Reddit for bringing this to my attention). This was an interesting podcast that introduced a lot of interesting themes about the IGF and how/why certain games are selected over another. The interest in what the Super Meat Boy creators had to say spiraled into a general interest into the IGF. Following is an account evidencing just how poorly constructed/maintained the IGF's systems are.

To begin, a quick summary and some quotes from Edward McMillen and Tommy Refenes will help to show how the IGF seems to function for judges. McMillen begins explaining that the IGF used to criticize many games for having a big budget - that having a decent amount of funding/exposure would act as almost a penalty before the competition even really began. He likened the IGF in the past to a scholarship system to give indie games good coverage/prize money because they lacked funds.

However, that changed and eventually big budgets and success didn't seem to matter as much to where everything leveled out with games like Minecraft winning the grand prize in 2011. But recently much of that has changed, as that penalty mentioned before seems to be back in a big way.

This quote, for McMillen, sums up what he sees the issue to be at the IGF:

... nobody's actually going by these rules that are set in place, they're just going by their own personal rule set, where they think, you know, that... there's arguments that are literally, 'hey, this person needs help! And I think letting them win will help!' They directly say, 'this game is better than this other game but, this game needs some help. Let's make them win.'

In this description, the IGF seems to be more of, like McMillen said, a poorly regulated, need-based scholarship system. It might not be a stretch to say that the judges treat the IGF like some kind of veiled funding group where part of the funding comes from massive exposure in addition to a cash prize.

Probably the most astounding thing that McMillen said was:

It pains me to have Phil Fish directly tell me that... he straight up just told us that 'I was one of the many people that voted against Super Meat Boy because I knew you guys were going to be fine.'

... right after we lost, Brian Crecente [former editor-in-chief at Kotaku, now a founder and news editor at Polygon] coming up to us and saying, 'oh you guys didn't really need it.'

This was not isolated either as Refenes relates another incident:So McMillen and Refenes' fears have been confirmed to them. Though let us all remember that McMillen and Refenes have submitted many games to the IGF, and their most profile game, Super Meat Boy, did not win. To say a little resentment may still be there would not be unfounded. This is just a reminder that both may be a little biased - but this is not dismissing any of what they said.

While the takeaway for that particular IGF seems to be that there was a focus on giving the awards to games the judges believed needed help, rather than giving merit the most weight, this evidences something much more important. Much of what McMillen and Refenes talked about shows how the IGF is susceptible to the mood swings/change in views of the judges. That what the judges take into consideration each year is likely different. That particular year it focused on games that needed more financial success, maybe in the future it will be about giving the award to a game that the judges agree with politically.

Who the judges are doesn't seem to matter (to an extent, more on that in a bit), as each year the IGF is going to be ruled by the ideology of the final judges. This also means the amount of corruption/negligence/lack of integrity will vary from year to year.

One such example comes from the developers behind a game called Kale in Dinoland. They detail their experience with the IGF here. You are welcome, and encouraged, to read more in-depth there but the gist of it is this: they found out that some of the judges that were assigned the game (judges are assigned games based on accessibility i.e. owning the console, etc.) hardly played the game and one of them didn't even install it.

The key quote from that blog post about the 8 assigned judges:

Of those judges, 1 didn't install the game... 3 judges didn't play the game. Of the remaining 5 judges that played the game, 3 played it very close to the IGF deadline... One judge, our outlier, played the game for 53.2 minutes. Excluding the outlier, on average each judge - including the 3 that didn't play it - played the game for almost 5 mintes' time. Back in that build, Kale's intro cutscene took about a minute's time. So we're talking almost 4 minutes for each judge of actual game time.

I was given 14 games to judge, and less than a month in which to do it.

This is not necessarily surprising when Alex May, an IGF judge in 2011, had this to say on the official IGF site That's all anyone really needs to know to explain the lack of judges playing the game. Obviously it is an issue that the judges didn't play the game, but the bigger issue seems to be that they are not required to play the game at all to vote for it, otherwise some kind of mechanism would be in place to prevent voting from judges who didn't play the assigned games.

This is a problem across all mediums in all awards. For example, recently at the Oscars one of the judges admitted to voting for a film they never watched. That still does not make this okay and one would hope that the IGF would want to do better in the future. Time constraints are a real problem everywhere and 14 games in one month is a lot for anyone to do - unless you have almost zero obligations and a ton of free time - which I would venture many of the judges do not.

One would have to assume that this is not an isolated incident. The developers of Kale in Dinoland just happened to have this information fall into their lap. While there is no further evidence (that I am aware of), it would probably be safe to assume that this has happened with at least a few more games at the IGF.

IGF has also had the issue of allowing games to enter (and win) multiple years. As of now there are only two examples of this happening, one being with the game Antichamber and the other being Fez. Fez was entered when it was really only a proof of concept at the 2008 IGF where it ended up winning "Excellence In Visual Art." Well, four years later Fez then won the 2012 Grand Prize at IGF. You can read Phil Fish's rationalization for why this is okay here (while discussing this with many indie developers directly interested in the IGF).

Basically, Fish says that because Fez was only a proof of concept game in 2008, the final product of Fez in 2012 was vastly different - basically he claims it was an entirely different game. Okay, maybe. Aside from the issue of having games submit multiple years (especially after already winning an award), shouldn't the IGF reject games that are only in their infancy? Or at least create a category for proof of concept games / games in progress?

The lack of oversight or care into what the rules and regulations should be, and whether or not they should be followed, is astounding. There are only three real explanations: incompetence, carelessness, or corruption. All of those are undesirable, but seem to have been rectified in some sense. Disallowing finalists to enter again in the future is a good change and does show that the IGF is willing to change in some respects. But, as seen above, the IGF has a real problem enforcing their own policies so we'll see how it goes. However, in this case, public scrutiny will be the oversight so the IGF won't need to provide any, thankfully.

The IGF has another significant issue in that they allow judges to "roam" about to check various titles that other judges have been assigned if they hear enough positive things about it. That judges could vote on things that they had not been assigned essentially. The IGF states as much here:

After judges finish recommending their assigned games, they are free to explore the rest of the entered games, and vote for any additional game they believe deserves a recommendation.

That is a huge problem. What's to stop judges voting just because of the "bandwagon" mentality? Maybe a judge heard from a friend of theirs (and/or fellow judge) that a game is good, convincing them to vote on it even if they haven't played the game. As we've already seen above, the IGF doesn't seem to do any checking on whether or not a judge has actually played a game.

Also, doesn't this give at least a slight advantage to more well-known/well-marketed titles? It is not unreasonable to say that someone would be more inclined to vote on something they have heard of, rather than something they haven't heard/seen before. Pair up something like Minecraft and Desktop Dungeons. I'm sure most haven't even heard of Desktop Dungeons, even though it was a finalist for the IGF grand prize in 2011.

Someone could argue that while those games do have a leg up, the good games without the same marketing should generate enough buzz from judges to level the playing field. Maybe, but then we would have to know just how much the various judges interact with one another and where this "buzz" comes from. It is more than likely created by judges who, while not friends maybe but certainly acquaintances, talking about the games. As mentioned before, is it because all of the judges are talking about a game (and where do they do this?) or is it because a friend/judge suggested the game causing another to vote on it?

We also have to consider what kind of time judges have to "roam" anyway. The quote from the judge, Alex May, earlier suggests that he had barely enough time to give the games he was assigned a good shot let alone look at what anyone else is assigned. Right now, ironically (when you consider what McMillen spoke about earlier), the IGF system seems to at least somewhat favor more well-known games. Likely games with a higher budget/more exposure.

If the IGF had a good track record in the past, allowing judges to "roam" may not be that bad of an idea (though there are still issues regardless as said above). This just shows further incompetence from the IGF as, while the rule may have had good intentions, there is absolutely nothing in place to enforce anything. If nobody holds the judges accountable, can we really blame them for breaking the rules? Obviously they hold some of the blame, but as of now they seem to have no deterrent to following them.

One thing to note for those that might be looking for information on the Indie Fund, Polytron, IndieCade, and IGF connection (as well as the separate Silverstring Media connection); you will find nothing here. That is something that was alleged a month ago. Some of the evidence and arguments are compelling, but when the true allegations begin to come out, all of the evidence is stretched at best. Most of it is just speculation at a certain point.

So to sum up the problems (known ones anyway) about the IGF:

the subjectivity of the judges is likely to change year to year depending on the ideals of that year's judges

there have been issues with judges not playing all of the assigned games or not playing them enough

the IGF had lax rules in terms of allowing games to enter multiple years

games do not need to be "complete" to enter

the judges are allowed to vote on games they were not assigned to play

there is an overall lack of oversight in ensuring the integrity/fairness of the many facets of the IGF, including things like ensuring judges actually played the game they vote on

As said earlier, it is hard to pin down the root cause of the many IGF problems. As of now it seems plausible that the root could stem from incompetence, carelessness, corruption, or a mixture of the three. What is certain though, is the fact that the IGF is horribly managed. They seem to have very few mechanisms, if any, in place to ensure the integrity of the judges and keep the competition as fair as possible.

The IGF will have to make a lot of changes for me to ever take them seriously.