(This short series is written to give some guidance on the basics of working with game data. It gives you some idea of the fundamentals, such as what data to track and when, how to figure out whether data is actionable or of good quality, as well as a final case study. Get in touch with us if you’d like to know more.)
Data is a great tool for any developer, and learning how to analyze it is invaluable. But data can be imposing to work with. Developers often struggle to know what they should be tracking, when they should be tracking it and whether the resulting data is actionable. Moreover ensuring that data is of consistent quality and accuracy is often a skill in and of itself, and this is one of the challenges faced by developers on a daily basis.
Any event or action in game can be a data hook. From the moment the player installs your game to the instant they unlock their first special weapon, make a purchase or reach level 1000, you can know that that happened. Every interaction, operation, movement of currency, score, win, loss, chat interaction and so much more can be data. But is that required?
Often, the problem that arises from being able to track everything is that a developer ends up overwhelmed, loses track of actionable insights vs. data that is informatory & adds to their analytics costs. As a result, a developer may lose focus on game improvement and data driven decision making and instead spend time focused on game data that does not help with decision making.
What To Track:
So what should you track? There are some pretty typical metrics that are worth knowing, such as:
- User data
- Device, OS version, game version
- Country, location
- Session beginnings, ends, average time spent in the game etc
- Demographics such as age and gender
User data gives the developer insight into who is playing their game, which often helps with design, and marketing/user acquisition (UA) decisions. This kind of data can sometimes lead developers into stereotypical, or even cynical thinking, about an audience. That kind of thinking is generally best avoided. But at its most best it can yield key insights about what an audience values. For example if your game is being played by an older audience that might indicate a need for gameplay that isn’t twitch-dependent, or stems from familiar themes already known to the audience. It might also, however, indicate a preference for a certain level of complexity, which can be very helpful.
- Core business performance data
- User installs
- Retention, typically measured across several timespans
- Lifetime value of players
- Revenue, often averaged per day/week, per user, per paying user
Core business data such as active users, retention rates and lifetime value (LTV) tend to be among the most commonly discussed KPIs that define the success of a business and are benchmarked across game genres on various communication forums like conferences, blogs, studios & discussion groups. Stakeholders often want to know these data as a way to frame their thinking about a game’s potential or its shortcomings. So they’re important to know, but rarely do they give deep-enough insight on what to do to improve or optimize a game. Like television ratings, core business data mostly tells us what is and isn’t working, but not why.
- Engagement data
- FTUE (“first time user experience”) completion rates
- Features used or played
- Player win or success rates, depending on the game design
- Spends, earns and wallets (amounts held) of game currencies and other resources
- Progression through levels, puzzles, maps or character levels, depending on the game design
- Response rates to notifications, prompts, referrals
- Monetization conversion, such as currency purchase or subscription rates
Engagement data, on the other hand, tends to give closer insight. This kind of data looks hard at what a player actually does when they play, giving designers clues about how to make the game more fun & challenging based on the average player’s purpose and what they value in the game. It also forms the bedrock of most meaningful experiments (such as A/B testing), building an understanding of common patterns and likely pitfalls, especially if it’s considered on a regular basis. There are limitations, however. Engagement data often doesn’t surface holistic issues well, for example,players not liking the material in a game for aesthetic or cultural reasons. Developers relying on engagement data also tend to assume that what’s presented in a game is structurally sound but not optimized, meaning they get stuck in local maxima – ignoring big issues that are preventing them from reaching new plateaus of success.
- Qualitative data
- Survey results
- Interviews, focus groups etc
- Playstore & Appstore Reviews
That’s where qualitative assessments can sometimes help. While subjective and reliant on the ability of respondents to explain their thinking (which is not always guaranteed), sometimes qualitative data helps a developer understand the truth underneath their engagement data. Such data needs to be treated coolly, however, as it can turn into a form of confirmation bias (in essence the developer only hears what they want to hear). This kind of data should be gathered from neutral parties rather than friends and family, where possible, such sources are invariably biased by wanting to be kind. But with open eyes and a clear head, qualitative can help a developer understand that they have been looking at a problem in the wrong way, or allowing their engagement data to railroad their thinking.
These are fairly standard sets of metrics to monitor, but they should be considered a starting point. Most games have special mechanics or systems that drive how they are played or define their core loops. This in turn tends to make every genre of game somewhat different from every other genre, and also necessitates some bespoke thinking regarding what to measure. Those mechanics and systems sometimes need to be measured in unique ways, to help balance or refine them.
For example, if you wanted to understand why your day-seven (“D7”) retention was poor, the answer might not be apparent from standard metrics. It might be that a particular level introduces a mechanic that’s fun to play but whose score system doesn’t give quite enough reward for the player to progress. It might be that this lack of score is demotivating even when players complete the level, necessitating retries that are boring. That might not be apparent from ordinary metrics of engagement and need a specialized event track to uncover, and the insight to realize that that metric is needed in the first place.
Actually adding such event tracking into a game is usually not that difficult. The difficult part is in clearly defining what is it that you want to measure, and what your baseline expectations are to understand whether the resulting data is typical or atypical. Doing this well usually takes practice. You should always be thinking about what to measure, and whether what you’re measuring can improve your understanding of the gameplay & lead to data driven decision making, and this will help build your skill in doing so over time – a topic I will get into in much greater detail in the next part of my Data Challenges blog post series. Check back here for a link to that article when it goes up in a few weeks.