Skip to content

Telemetry Data, Heat Maps and Unlearning what you have Learned

October 9, 2009

In a continuing effort to share stories about the development of Lucidity, here’s a piece written by Level Designer Joe Ching.  Joe also worked really closely with Dave & Andrea on story development for the game, which you can hear all about  in the Story podcast.  In this piece, he talks about a subject that has come up a fair amount in the comments and questions  – the art of difficulty ramping. Over to Joe…

As a designer, it’s definitely hard to take an objective approach when evaluating your own work. Sure, you can play your game over and over, hoping to pickup all the subtleties that need some tweaking or fine tuning.  But how do you unlearn what you have learned to experience your design through the eyes of a first time player? Well… the truth is you can’t. And that’s where the power of Telemetry Data and Heat Maps comes into play.

We designers on Lucidity were heads-down designing, planning and then executing on the game for quite some time. With each new build, we played through to get a feel for where things were at, and for what still needed to be done. All the while, we felt the game was certainly challenging, but also extremely fun. With each round of new levels, we often found a new benchmark for the title of toughest level in the game. But often as I walked through the team area, I’d overhear conversations about how hard the game was, and I would think to myself, “Really? It doesn’t seem that hard to me.”

When the production phase of Lucidity came to a close, our lead designer Jens & I were left to see design through to Alpha and beyond. During this period, one of my primary responsibilities was to tune the difficulty of the game.

During several previous play-test sessions, we made some observations on which levels were giving players a particularly hard time. Keying off what I had seen from the play-testers, I went in and started adjusting level geometry, hazards, and enemies to make these levels easier.  Or so I thought.

Reality Bites

At the same time, Jens began opening up discussions with our QA & Compatibility departments to see what telemetry data we could get from their playthroughs. It wasn’t long before Tzuen, Matt, and Seth from QA sent us some detailed PDF files containing charts, graphs, and all-important heat maps showing us frequency and location of player deaths within the levels.

The information we received was a revelation. For many of the levels I had thought were easy enough, the data showed the exact opposite.

This is when I realized that my supposed “objective” evaluation of the game was anything but.  Being submerged in the game’s development had raised my skill level to a point where, whether I liked it or not, I could no longer form an accurate opinion of the game’s difficulty.  I needed hard data, and lots of it.

This heat map shows that difficulty in this level spiked too soon - and there were no easy paths through the level.

This heat map shows that difficulty in this level spiked too soon - and there were no easy paths through the level.

The “Before” picture shows, a level that had already been through a few tuning passes, but was still very difficult for players. Deaths (represented by the red dots) were still occurring all over the level… in places I never expected.

With this and several other very clear heat maps to work from, I could easily identify problem areas within each level. Rather than using my “gone native” judgment to make changes to geometry, enemies or hazards, the heat maps showed me exactly where and what to target.

Here, you can see how the hot spots have been greatly reduced. And the player gets an easier ride if they propel Sofi upwards.

Here, you can see how the hot spots have been greatly reduced. And the player gets an easier ride if they propel Sofi upwards.

As the “After” picture shows, the second round of telemetry data shows that we made  the substantial improvements we were aiming for . One might argue that this went too far in making the game less challenging, but what we found was that players were far happier. Deaths still occurred, but not at a level where frustration set in.

Before receiving the telemetry data, I was pretty much shooting in the dark as far as tuning was concerned. But with the telemetry data and heat maps in hand, I had a road map to successful tuning. I was able to unlearn what I had learned about the game by using this data to see the game through the eyes of the player.

One of our primary goals is to make sure the player has an opportunity to experience all the story based levels – to see the amazing work of our artists, to hear the great music and sound effects from our audio department, and to see how Sofi’s story unfolds. My hope is that with the final tuning work done in the game, everyone who plays it will have a fun, challenging, and beautiful experience.  I hope you like it.

Advertisements
13 Comments leave one →
  1. jaybroni permalink
    October 9, 2009 2:29 pm

    Wow, very cool. Are these “heat maps” something that is common in the gaming industry, or something you created just for this game?

  2. Robert V permalink
    October 9, 2009 2:58 pm

    Another game that posts their heat maps is Valve for the game Team Fortress 2.

    At the bottom of this link is some for their various maps.
    http://steampowered.com/status/tf2/tf2_stats.php

    A direct link to one of the heat maps is below.

    Doubtless they use it for their redesigns and patches. Also, since they post it players can see how these develop.

  3. Cedge permalink
    October 9, 2009 5:53 pm

    @jaybroni:
    Lots of games use “heat” mapping. Epic and Bungie have shared heat maps showing which areas of Gears of War and Halo multiplayer maps experience the most player traffic, for instance.

  4. Jibece permalink
    October 10, 2009 1:45 pm

    Cool article 🙂

    But.. can we hope get the SDK.. one day? ^^’

  5. Telixion permalink
    October 11, 2009 7:17 pm

    You think you could get some dlc in this?
    I would really like to get some time trial maps. With the items in a certain order every time you play.
    You have to get to the end as fast as possible. But for every item you use it adds more time. So do it the fastest but with the least amount of items.

  6. October 15, 2009 1:01 am

    Hey, guys?

    Is anyone from Lucasarts reading these comments at all?
    Who can I talk to regarding press stuff and reviewing Lucidity?

  7. October 15, 2009 1:51 am

    BTW, this kind of information is very interesting. keep going!

  8. Jaybroni permalink
    October 15, 2009 12:19 pm

    @zipdrive

    The people posting all the articles and responding to comments are all LucasArts employees, so yes, they are reading the comments.

    To contact LucasArts regarding evaluation copies of software, see here:

    http://bit.ly/8OUOJ

    • October 15, 2009 3:16 pm

      Thanks for the link, Jaybroni. I did not see any LA eployee responses in the posts I’ve read, so I was unsure about them reading the comments.

      • Shara permalink
        October 15, 2009 4:44 pm

        Hi zipdrive: yes, we’re definitely reading the comments! We can get caught up in work sometimes, but it’s great to see that all of you in the Workshop community have also taken part in helping answer questions 🙂

  9. November 18, 2009 2:21 am

    Yes indeed, very interesting information. I never knew about these heat maps. I love to learn new things 🙂
    Keep it up

    Regards
    Tom

  10. January 18, 2010 6:18 am

    This was a really interesting read! These articles on the development of the game have been a fascinating read for me.

    Using heat maps would have been very useful to me when designing levels for a platform game at work this month, as I experienced very similar problems when attempting to judge the difficulty of levels. It got to a point where I became so fluent at playing the game that I could no longer effectively judge their difficulty.

    The playtest results certainly helped, and I was able to see on a broad scale which levels players were dying too frequently on, or restarting the level constantly – but not specific areas/puzzles within that level. I, too, felt that it was a bit of a stab in the dark at times!

    Nonetheless, I look forward to playing Lucidity on PC, I’ve yet to get hold of a copy 🙂

Trackbacks

  1. Telemetry Data, Heat Maps and Unlearning what you have Learned | Dave-I-Y

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: