AL16
Raiders of the Lost ROM: Epilogue
I was not planning on writing another article in this series, but then something unexpected happened: after all these months, I found a bug in Quadromania. Worse, it only took 10 minutes of playtesting to uncover! What was the bug, how did I miss it until now, and what could be done to avoid this type of bug? Those questions and so much more will be discussed in this new episode entitled “Epilogue” (really the full name should be “Epilogue: Never Say Done With Testing Again”).
What was the bug?
When starting a new game of Quadromania, you can select levels 1-9. If you defeat level 9, you advance to level 10. Level 10 is then unlocked as a future starting level. One of the features I added to the game was saving your unlocked level to EEPROM. This feature has been tested extensively and is known to work with EEPROM. However, the bug I found was while using a version of the Handy emulator that does not support EEPROMs (same would be true with Mednafen). Once you defeat level 9, there was a bug that caused the max_level variable to get reset to the default value, which then unlocked all levels above 9 and in fact even unlocked invalid levels too! Let me describe in more detail.
I added a check for EEPROM support and saving/resetting certain variables every time the player enters the title screen. I even added some checks to avoid writing the EEPROM repeatedly when no data has changed. Here is the very simple pseudo-code for my EEPROM routine:
- Read first bytes from EEPROM and check for unique game ID string (for example, “QM” for Quadromania).
- If string does not match, set max_level to default which is level 9.
- If string does match, set max_level to value stored in EEPROM.
This is a fine set of pseudo-code, but it does not function as expected if invoked every time the player enters the title screen. The first time at game boot it works properly, but subsequent times it will keep resetting max_level if it thinks no EEPROM is attached. This results in the game being naughty (to be fair, it’s really me the developer who is naughty!) by allowing the player to select any level above 9 at this point, including levels that don’t exist! The very simple fix was to move this entire check so it only happens once on game boot and never again; now I only touch the EEPROM during the game if a new max_level is awarded.
Why didn’t I find it during my earlier testing?
I was so focused on adding EEPROM support that I spent 99% of my time testing ROM builds either with Handy 0.98 (which has EEPROM support) or on actual Lynx carts with EEPROMs on board. It never occurred to me to think about how the game would behave if no EEPROM was present, either due to lack of emulator support or if a physical EEPROM failed on the cartridge at a later date. Fortunately, by putting Quadromania on the shelf for a couple months while I worked on packaging for it as well as tested other games, I was able to revisit Quadromania with a fresh set of eyes.
What could be done to avoid this type of bug?
Once I saw how bad this particular bug was, and how easy it was to hit via the simple transition from level 9 to level 10, I knew I had to expand my test configurations. I now have this set of tests established for my games:
- Handy without EEPROM support
- Handy with EEPROM support
- Mednafen without EEPROM support
- Lynx cartridge with EEPROM support
The only test configuration I’m missing from that list is “Lynx cartridge without EEPROM support” which I could do by physically cutting a line to an EEPROM on one cartridge, but for now I am using the emulators to validate the “no EEPROM” configs and also hacking the ROMs to pretend like no EEPROM is present.
Really this issue highlights the need for rigorous, and preferably independent, testing. You have to know your matrix of test platforms where users may enjoy your game. You have to know what code you have added or changed. You also have to know that you may not have physically changed a part of the code but you could still break the behavior of that code due to other supposedly isolated code changes. And finally, you have to know that you, as the coder, may be blind to to some of the above interactions and resultant bugs.
Historically I have done most of my game testing myself. Over the past year, I have found myself testing a variety of games for other developers too. There is legitimate value in having an independent party test your game as they may uncover things you never thought to try. I guess the lesson here for me is I should consider doing the same! 🙂
