Telemetry Data, Heat Maps and Unlearning what you have Learned
In a continuing effort to share stories about the development of Lucidity, here’s a piece written by Level Designer Joe Ching. Joe also worked really closely with Dave & Andrea on story development for the game, which you can hear all about in the Story podcast. In this piece, he talks about a subject that has come up a fair amount in the comments and questions – the art of difficulty ramping. Over to Joe…
As a designer, it’s definitely hard to take an objective approach when evaluating your own work. Sure, you can play your game over and over, hoping to pickup all the subtleties that need some tweaking or fine tuning. But how do you unlearn what you have learned to experience your design through the eyes of a first time player? Well… the truth is you can’t. And that’s where the power of Telemetry Data and Heat Maps comes into play.
We designers on Lucidity were heads-down designing, planning and then executing on the game for quite some time. With each new build, we played through to get a feel for where things were at, and for what still needed to be done. All the while, we felt the game was certainly challenging, but also extremely fun. With each round of new levels, we often found a new benchmark for the title of toughest level in the game. But often as I walked through the team area, I’d overhear conversations about how hard the game was, and I would think to myself, “Really? It doesn’t seem that hard to me.”
When the production phase of Lucidity came to a close, our lead designer Jens & I were left to see design through to Alpha and beyond. During this period, one of my primary responsibilities was to tune the difficulty of the game.
During several previous play-test sessions, we made some observations on which levels were giving players a particularly hard time. Keying off what I had seen from the play-testers, I went in and started adjusting level geometry, hazards, and enemies to make these levels easier. Or so I thought.
At the same time, Jens began opening up discussions with our QA & Compatibility departments to see what telemetry data we could get from their playthroughs. It wasn’t long before Tzuen, Matt, and Seth from QA sent us some detailed PDF files containing charts, graphs, and all-important heat maps showing us frequency and location of player deaths within the levels.
The information we received was a revelation. For many of the levels I had thought were easy enough, the data showed the exact opposite.
This is when I realized that my supposed “objective” evaluation of the game was anything but. Being submerged in the game’s development had raised my skill level to a point where, whether I liked it or not, I could no longer form an accurate opinion of the game’s difficulty. I needed hard data, and lots of it.
The “Before” picture shows, a level that had already been through a few tuning passes, but was still very difficult for players. Deaths (represented by the red dots) were still occurring all over the level
… in places I never expected.
With this and several other very clear heat maps to work from, I could easily identify problem areas within each level. Rather than using my “gone native” judgment to make changes to geometry, enemies or hazards, the heat maps showed me exactly where and what to target.
As the “After” picture shows, the second round of telemetry data shows that we made the substantial improvements we were aiming for . One might argue that this went too far in making the game less challenging, but what we found was that players were far happier. Deaths still occurred, but not at a level where frustration set in.
Before receiving the telemetry data, I was pretty much shooting in the dark as far as tuning was concerned. But with the telemetry data and heat maps in hand, I had a road map to successful tuning. I was able to unlearn what I had learned about the game by using this data to see the game through the eyes of the player.
One of our primary goals is to make sure the player has an opportunity to experience all the story based levels – to see the amazing work of our artists, to hear the great music and sound effects from our audio department, and to see how Sofi’s story unfolds. My hope is that with the final tuning work done in the game, everyone who plays it will have a fun, challenging, and beautiful experience. I hope you like it.