Understanding your game through data

Analytics for indies 101

Along with marketing & PR, market research and analytics are still considered to be something reserved exclusively for “the big guys” — either major publishers or free-to-play behemoths.

Some indie developers even go as far as to declare these tools “evil”, and stand by our ancestral ways of doing things, i.e. relying purely on instincts and luck.

In this article I’ll try to explain some of the basic research you can do yourself to improve your game without investing significant resources or buying expensive subscriptions and white-papers. I firmly believe that proper research and knowledge will help you make informed decisions and that is ultimately a good thing.

I won’t be talking about free-to-play stuff — there are many articles covering MAU, ARPPU, UAC, CPM, CCU and other scary acronyms. My article is for indies developing old-school paid single-player games.

But first let’s start with some arguments against doing your research.

”Don’t tell me how to make my game”

“I’m a creator, not a corporate slave, and I don’t need marketing research to make something truly unique. Like my new retro-looking platformer for Steam”.

“I know what I’m doing and I don’t want unwashed masses’ opinions on that”.

You’re not asking for advice when you’re doing research. You’re asking questions. A doctor doesn’t ask a thermometer for its opinion on treating the patient.

“I’m making something truly unique, there is no pre-existing market for it”.

Good for you, but how do you know that, if you didn’t do proper research first? Chances are there have been games with mechanics similar to yours. Don’t you want to learn from their mistakes?

“I don’t care about the numbers, my goals are qualitative, not quantitative”.

Qualitative goals can be measured too. Don’t you want to make sure you’ve hit your targets?

”I want my game to make people rethink their life choices, not to make a lot of money.”

Splendid! Did you know that you can measure this as well?

”What do you mean by ‘improving my game’?”

It’s for you to decide. You’re the one creating the game, you’re the one deciding what is considered an improvement.

Some developers might want people to stick around in their game longer, others might want to ensure the game doesn’t drag on for too long. You might want people to have fun or you might want them to cry after playing. It’s your call.

Some truisms

You’re doing research and analytics to make informed decisions about your game.

You’re the one making decisions. Not the audience, not Steam Spy, not Google Analytics.

Data doesn’t “tell” anything, nor it is “good” or “bad”. You’re the one interpreting it.

There are many tools to do the research. Not everything will be applicable to your game, but something will be.

A single day spent doing the research could save a month in development later.

While doing research is an actual job (like marketing or art), you can still do some of it yourself without hiring other people.

Imagine you found out that the audience for your game is pretty small but quite dedicated. Should it stop your from developing your game? Of course, not! It might stop you from overspending and will help you understand your audience better.

Before the development

”I don’t even have a game yet, why do I need to do analytics at this stage?”

Why?

Now is the best time to research the market, as you are not invested in your game too deep yet — both emotionally and financially. You have no code, no graphics, nothing to prevent you from re-imagining your game.

You might cancel or change the game after one week of market research — it’s cheaper and easier compared to doing the same after months or even years of development.

You might improve your game even before you start working on it by learning more about your theme, your core mechanics and your audience.

What do I need to research?

The game’s theme;

The game’s core mechanics (genre);

Audience;

Market & competition.

Theme, mechanics and visual style

Your game could be described as an intersection of a theme (sci-fi, fantasy, modern warfare), a set of core mechanics (FPS, RPG, side-scroller) and a visual style (80’s cartoon, surrealism, gritty, fake retro). Of course it’s not the whole picture, but that’s how an audience is perceiving your game before playing it.

Theme

The theme is the easiest part to research, as we can rely on massive amounts of data gathered by other, more open, industries. While you’ll have to pay north of tens of thousands of dollars for a simple gaming audience survey you can get results for similar movie audience research for free.

Look for movies, TV shows, comic books and books sales and audience engagement stats for your theme. Most of it could be obtained through Statista, Box Office Mojo and IMDB. Or even Google Trends. Steam Spy is not much of a help here.

E.g. it’s obvious that the interest in sci-fi and space opera is growing recently as we see more movies, books and comic books in this theme. It’s visible both in the box office for sci-fi movies, in the number of releases and it is reflected in the marketing.

It’s hard to breathe life into a trend — to create a game that will make everyone interested in sci-fi. Much easier it is to ride the wave — to create a sci-fi game right when people have an itch to fly space ships thanks to a movie or book they’ve enjoyed recently.

Remember when The Martian came out and everyone wanted to build a colony on Mars? That’s roughly when Planetbase launched on Steam and despite getting Score Rank of 48% it managed to get over 100K owners. I don’t know if the developers predicted the surge of interest in building cities on Mars, but they’ve certainly reaped the benefits from it. So could your game.

Predicting a trend is a bit harder, but it’s worth it. Sci-fi growth was visible right after the success of Avatar and quite a number of games managed to employ this observation to their benefit.

Look for announcements of big movies with your theme and chart them on a calendar to determine when you can expect their marketing campaigns to start and help you. You don’t have millions of dollars to get people interested in Martian city-building, so why not let rich guys from Hollywood do some work for you?

Though if your theme is too niche or too small for Hollywood or even books — it’s fine too. You’ll learn more when researching your audience.

Core mechanics

I prefer to think of games in terms of their core mechanics instead of the genres. Otherwise it gets too fuzzy — is Rocket League a racing game or a sports game? But when you think about it as a Racing + Sports + PvP + Team-Based, it gets a bit easier to understand and analyse.

Dissect your game into several core mechanics that are easy to explain and (preferably) match Steam’s tags — this way it will be easier to research your competition on Steam Spy.

Make a list of top games for each mechanics (or better for their combinations) and list their strengths and weaknesses (gameplay-wise). You will later use this list to avoid making mistakes other people made while developing similar games and to improve your own game.

It would also help to play most of them to see what works and what doesn’t.

It’s certainly possible that you won’t find a game similar to yours on Steam Spy. That’s because Steam Spy only covers games from Steam and there are many more platforms out there. Just google then — there is a good chance you’ll find a game with some of the mechanics you’re planning to use on Kongregate, AppStore or Google Play.

Again, as you’re (hopefully) not building a full clone, you won’t find an exact match. But if you’re creating a puzzle game about driving racing cars, it would be useful to see how racing cars are usually handled in other games.

Audience

This one is seriously tricky.

Your potential audience are people with access to the platform of your choice, who are interested in your game’s theme and its core mechanics, and not scared away by its visual style.

The problem is, there aren’t many free tools to research your audience. Steam Spy only covers Steam and that’s a very specific subset of gamers — 95% of them are male (vs roughly 50% of general audience), around 70% of them are buying games (vs roughly 25% of the audience), they tend to be from Europe and US.

So if you’re not developing a game for the Steam audience, don’t trust Steam Spy for audience research. And that’s coming from the guy who created Steam Spy.

You’ll have to use social media groups or forums. I’d start with the ones dedicated to your theme, but you can go with core mechanics as well.

Go, introduce yourself and just ask people if they would be interested in playing something like your game. Make a poll if the social media of choice allows it.

You probably won’t need too much socio-demographic data about your audience, as you should pay attention to their interest in your theme, not to them personally. Though knowing age brackets and gender distribution might still be useful.

If possible, try to understand where are these people coming from — it will help you decide on your localisation later.

There is a good possibility that there are no online communities dedicated to your core mechanics. As for the theme, it is almost guaranteed, as Internet is extremely diverse and there are groups and communities about almost anything.

Caveats

Don’t ask your friends and don’t promote that poll on your own social media. You want an opinion of your target audience.

Use several independent forums or groups, not just one. If you ask a question on Twitter, you’ll get opinions of your friends. I am sure you value their input, but it will skew the results if you’re asking your target audience at the same time.

Don’t pay too much attention to game ideas people will gladly provide you with. Not at this stage at least.

Be aware of a sampling error (both oversampling and undersampling). If, according to movies and books with the same theme, your game’s theme is more appealing to women over 40, don’t ask twenty-something males about it.

I cannot stress this enough — asking your target audience instead of “general gaming audience” is extremely important, especially if you’re truly creating something new and unique.

Competitors

Make a list of games that are using the same theme and some (or all) of your core mechanics. Both released and upcoming. Especially upcoming, because chances are you’ll be judged against them.

Make a basic SWOT analysis for every one of them, but also add an additional field: “How is our game different?” The key word here is “different”, not “better”, so you won’t get caught in wishful thinking “we’ll have better graphics and better balance”. Why your target audience should consider your game instead of another one? People only have so much time to play.

You can also check geographical distribution and stats for released games on Steam Spy or AppAnnie, but, frankly, it’s not that useful at this stage. You’ll look into it later, when deciding on focusing your marketing and localisation efforts.

If you’ll decide to check geo distribution for similar games, don’t trust it too much — an audience research you did previously will be more helpful. Other games might’ve done something specific to become popular in some countries, like partnering with a local publisher or getting a good video from a local YouTube celebrity.

For example, there aren’t many owners of The Witcher 3 from Poland on Steam despite the game being immensely popular in that country. That’s because most Poles bought the game from CDP.pl or GOG.com instead of going for much more expensive Steam version.

Summary

A good analysis will take you a week or two, full-time. It might seem like an overkill, but it’s nothing compared to doing it at a later phase of the game development and then changing the game accordingly.

Do it yourself, don’t outsource. You might come up with new ideas for your game at this stage by just researching relevant media and audience.

During the development

Your choices now are becoming more and more expensive.

Why?

It’s the time when you can get a reaction to your game, not just an idea of it.

You can change many things without angering your existing audience.

What do I need to research?

An audience reaction to your visual style;

Players behaviour in your prototype;

User interface.

Visual Style

If you’ve talked to people online about your game in the previous stage, you should have a pretty good understanding of who is the target audience for your game.

You can now ask them about your visual style. Usually it’s enough to have several art drafts/mockups. Don’t ask if the audience likes them or not, ask if they’d play a game that looks like this.

Focus on your target audience. Every successful game is considered ugly by someone on the Internet. Skyrim? Too dirty. Clash of Clans? Too childish. League of Legends? Too bright.

Don’t just rely on common sense, verify your assumptions with your audience. Games for older women were supposed to be bright and pink, focused around kittens and farming, until dark horror games about sexy vampires arrived and made a lot of money. If you’re not a part of your target audience, your assumptions about it are probably full of prejudices and could be plain wrong.

Player behaviour

By now you should be adding in-game analytics already. There are many options (Google Analytics, Unity Analytics, Flurry and so on), you can choose whatever you like. You can even create your own, as many people do — it’s not exactly a rocket science.

But even if it’s not there yet, you can learn a lot by watching people play your prototype.

The best way is to record a person’s face simultaneously with what’s going on the screen — think “let’s plays”. Every modern laptop with a front-facing camera is already equipped for this, you’ll just have to install the software.

Alternatively, you could just sit nearby and watch if the person is comfortable. Interestingly, several years ago people were more comfortable with someone watching them instead of being recorded. Nowadays, thanks to ubiquity of Twitch and YT streamers, they don’t mind as much.

Again, when inviting people to try your game, focus on the target audience, not your friends. You probably know many people in your target audience — either directly or through some other people. For example, I know a couple that develops games for children. They have two of their own, so it helps, but it’s obviously not enough. So they’ve got kids of their friends play testing their games, then classmates of their kids and so on.

It will get harder in the end, as you’ll be running out of relatives to test the game on. You’ll need fresh eyes for every iteration, so you can’t just ask the same people over an over again.

Things to look for when watching people play your game:

What catches their attention?

What do they do first?

How much time passes before the person gets to the first major point in the game?

Did you expect it to take this long?

Do they understand the game rules?

Do they understand effects of the game items or enemy strategies?

How do they handle controls?

When a person starts playing, don’t talk, don’t help, don’t defend or explain your game. You won’t be sitting near every actual gamer when people will start buying and playing your game. Shut up and watch.

Always do a short interview or a survey after the playtest. It helps to quantify and record the results. You’ll probably start with smaller samples (one or two people playing it) and non-formal interviews, but as you go bigger (and possibly invite people online to play it), you’ll want to opt for more or less formal surveys. You can use your knowledge from those first informal interviews to compile a questionnaire.

Determine a net promoter score. Ask “On a scale of 0 to 10, how likely are you to recommend this game to a friend?”. Then divide the answers into three groups: Detractors (0–6), Neutrals (7–8), Promoters (9–10). Your NPS score is ( Promoters — Detractors ) / Total.

So if 10 people voted 0–6, 20 voted 7–8 and 20 voted 9–10, your NPS would be (20–10) / 50 = 20%. Don’t stress out if it’s low (or even below zero), your goal is to measure net promoter score at all stages of your game development to ensure it is growing, not falling. And of course NPS is only useful if you’re asking your target audience.

By the way, note how it is similar in distribution to gaming press scores and Metaritic.

You can ask if the person liked the game or not, but NPS is more useful, as people tend to lie so they don’t hurt your feelings. At least in Western countries. In Russia they might say your game is bad and you should feel bad despite somewhat liking it, because they want you to improve.

Ask about what the person found most confusing, most fun, most difficult and most boring. Record the answers.

User Interface

Prototype and test your interface on paper. Get a notepad the size of an iPad or an iPhone (they’re available online), or just a standard A4 one and draw your interface.

If your game is for mobile devices, consider different screen sizes and different hands sizes. Test on different people.

You’ll be able to eliminate many problems by just handing a piece of paper to a person and asking if he/she can understand how the game is supposed to be played and what’s happening on the screen.

I have a friend who builds elaborate paper UI prototypes with moving parts, but it’s probably too much. Quite fun to use, though.

After the launch (or, more likely, Early Access)

The game is done, fire and forget. Why are we still analyzing it?

Why?

You have actual users playing the game!

You can still find and fix a lot of problems.

You’ll learn a lot for your next game.

What do I need to research?

In-game behavior;

In-game economics;

In-game balance.

In-game behavior

I hope that by the time you release, your game have some analytics system implemented — Google Analytics, Unity Analytics or even home-grown solution will do.

You should be tracking user path through the game. Where does he/she starts? Does he/she go for the first enemy or wanders around clicking on every NPC?

Cluster users based on the steps they’re taking in your game. Some will go straight to tutorial, some will go PvP, some will start exploring. They’re playing the same game, but it’s now their experience, not yours, so watch them and learn.

How much time a person spends in a game per sitting? How often does he play per week? Is it correlating with the path he chose earlier? If “explorers” are playing less than “fighters”, maybe exploring isn’t as fun. You might want to emphasize fighting more or create something interesting for people to explore.

In-game economy

Don’t go for averages, always go for brackets or clusters, as averages are deceiving. You want to look into outliers to see the whole picture.