Metrics are one of the most important and frustrating parts of game development. How do you really know when something is “done”? What do terms like “done” and “balanced” even mean? Is it balanced if a matchup is 4-6? What if it’s 45-55? What about 495-505? As you can see, there’s really no end to it. Even these numbers are suspect, if the skill level of the players is differential. Not all playtesters are going to be at the pro-player level. Nor should they be.

What we’re recording during playtest sessions, and by extension, what we are looking for during these sessions, needs to be far more clear. “Play some games and let us know when you think it’s good enough” just isn’t going to cut it when you’re trying to deliver multiple, highly-polished, competitively-balanced games per year. We have to have a standard for what “good enough” is, so that we can ship games to you with consistent quality and on a consistent schedule.

We’ve identified four metrics for playtesting that we look at to determine if a character is “complete”. By following these four points, we will have a complete definition of Completion in playtesting. The exact requirements and exact numbers change for each game. With Exceed, we require more plays for balance. With Millennium Blades, things are fairly balanced across the board.

Fun - "Content is played and enjoyed.”

Completion: X players have played and enjoyed the content.

Fun is a difficult thing to test, and means a lot of things to a lot of people. The important thing here is that we have a bottom limit of players who will stamp “Fun” on the content. We don’t need to please everyone with every piece of content, but if a certain number (not a certain percentage) of testers enjoy the content, we can be confident it will find an audience in the real world. The reason to avoid a percentage here is that playtesters are largely a self-selected pool, and tastes may become homogenous in the group. In order to give each voice equal weight and better reflect the demographics of the real world, we use a fixed number rather than a percentage of our playtest pool.

Balance - "Content is mathematically & heuristically balanced."

Completion: X experienced players have played the current version of the content and found no need for further updates.

For Balance, we specifically look for the word of experienced players. As with fun, player skill levels and experience levels will vary, and balance will need to be determined carefully. Where possible, mathematical and mechanical balance comes first. After this, the heuristic (or “learned”) balance of actual players is applied. This “gut check” is intended as a final step to balancing the system, not as the primary driver of balance. Like fun, we look for a certain number of sign-offs from experienced players, rather than a percentage consensus. Competitive game balance is a divisive subject, after all.

Function - "Designer intent is expressed in the content."

Completion: X new players have played content correctly & without confusion.

The Function portion of our assessment is what we would more colloquially term as “Blind Testing”. It’s there to guarantee that Designer Intent (that is, the way the game is played) is clearly transferred to new players by the game’s rules and the text on the cards. In this pass, we seek to eliminate any hidden or obscure interactions with the game contents. If a card doesn’t reveal its true value or correct playstyle immediately, then it needs to be reworked in order to improve its functionality.

We want players to be making strategic decisions, after all, not to be exploiting hidden interactions. This streamlining of function different than Depth. Depth of play which comes from the interaction of strategic decisions. It is our belief that making strategic play clearer and more accessible leads to an increase in total game depth, since more strategies can be accessed more easily by players.

Clarity - "Content is unambiguous and templated."

Completion: X experienced players have proofread and approved the content.

To pass the muster for Clarity, the interactions of the component are clear and unambiguous. This covers wording, symbols, templating, and consistent use of language. A templating guide is provided to ensure that the game’s text is clear and clean. As a rule, text is minimized in favor of symbols, diagrams, and (where appropriate) keywords. Developers should suggest these updates where appropriate.

3. Let’s Get Together