In a comment replying to The Craft of Game Design Cannot Be Measured By Any Metric, game designer and Chief Creative Officer at Spryfox Dan Cook gave such a sterling, thorough rebuttal that I’ve reposted it here in full.

Empiricism in Game Design

When I design I have a mental model of how I imagine my game will be played by players. This includes predictions about player emotions, learning, buying behaviors and a dozen other factors necessary to make a self-sustaining game in one of today’s various markets. I also make predictions about how markets will act. Platform desires, player designers, press desires.

Then we build the game, or at least we build an initial version of it.

Then we playtest the game to see if the my predictions worked out. Most of the time they don’t. In the best cases I’m only off by a factor or two. In the worse cases I’m off by several orders of magnitude. However, I may also find that players behaved in a manner that was actually more interesting than I predicted.

So we build another iteration of the game. Somehow, we need to connect the empirical reality of what the playtest suggests with what we predict will happen. This usually involves updating our models, sometimes radically. Often incrementally.

For some designers, this process can be frustrating. The reality of player behavior imposes constraints on their mostly imaginary vision. But I tend to see constraints as necessary to the process of design. And constraints based off observing real people playing the game tends to more often than not yield opportunities to impact the real shared world of many people vs the isolated imaginary world of a single person. We find new ways of playing that are more vibrant and interesting.

How are metrics useful when iterated on a game?

Game designers are information starved. With writing, we have an imperfect but competent mechanism for imagining how someone might feel reading a bit of text. In order to write, you must read. And thus you are forced to process a work in a somewhat similar fashion to how a potential reader might process. Game developers do not have this luxury. We build systems multiple times removed from a player’s experience. Write some code. Do a dozen other steps. Build an executable that someone somewhere runs. Knowing how people with react to what we make is hard.

So we use crutches. We create complex models of how players think. We use ‘proven’ patterns. We watch players and try to imagine what they are feeling. Then we try to backtrack all far removed information to whether or not a number in the bowels of a broken machine should be 2 or 4.

There are certainly classes of information we can extract more easily. Surface player emotions on individual playthroughs. Awesome. We can do that. But human behavior is broad. We see the need to sample behaviors across populations and discover central tendencies or outliers.

So metrics or analytics are that tool. They let us understand statistical patterns of behavior. Do they let us see inside the minds of our players? No. Nothing does yet. Do they replace in person playtests? No. Smart designers use multiple sources of insight.

But metrics do provide an amazing range of insight by allowing us to look at hard problems from a different direction. If players in an MMO are flooding forums with complaints about a change, how many people are impacted? How did playstyles change?

When balancing economies and progression systems, metrics are essential. You can’t do an in-person playtest of someone playing a game for 90 days. The old tools don’t work. And various forms of data collection do.

Maybe all this doesn’t need to be said. Maybe you are worried about something else entirely.

Are you worried about how metrics shines a light on bullshit design? Because a lot of design is unsubstantiated bullshit. We imagine people will play a game a certain way and then they don’t. Such an ego buster. Metrics beat us with bully numbers. They bluntly state our initial idea was flawed. Or even worse, the thing that people have been praising us for years doesn’t actually apply to anyone but some weird elite group of outliers that happens to give out chintzy feel good awards. Reality can be cruel when you live in a fantasy. But it also acts as a constraint that forces us to up our game and make something that works. Versus wandering blindly off a cliff in a feel good haze. Which I’ve done. (Lovely until you fall).

Are you worried that Bad Men use metrics in a reductive fashion to emphasize making money over art? Bad Men have been emphasizing making money over art for a very long time. For any golden era of games there were penny pinchers micromanaging creative decisions at a level that destroyed souls. Might I suggest that a new tool for getting data is not the actual problem. The team sets their goals. The tools just get them there.

Are you worried that we are using Dumb Metrics? That the dumb patterns dumbly followed by dumb practitioners result in dumb ideas and dumb games? Well it is true. And the solution is one that applies to all complex instruments used in the pursuit of art and beauty: Get Good.

I actually see metrics, competent design and building something positive that meets player needs as three complementary pursuits. I’ve asked “Well, what do players want and how does that align with business? And how does that align with art or craft?”

Here’s one answer. Many players want connection with meaning and community. They want mastery and agency. This leads to them enjoying an activity for a long period of time. That results in great retention metrics. And when deep needs are being met, people are willing to spend. Will I spend a buck on Pokemon lures to enhance a relaxing afternoon with my wife at the coffee shop? Yes. It makes for joyful light conversation. The game improves our relationship by creating a shared playful space.

Metrics track and tune all this. Is that evil? Just the opposite. I consider it doing great good for the world through competent design practices.

I have made minor edits to the text to make it read as a standalone post: the original comment is still available under the original post.