Starcraft AI blog

Early on in Steamhammer’s history, one version introduced a bug that caused small delays in starting the construction of buildings. You couldn’t really tell by watching games unless you knew what to look for, but results were worse and it was clear that something was wrong. It was a stealth bug: The visible symptom was that play was worse, and the cause was hard to discern. I was only able to narrow it down and solve it by knowing what I had changed recently.

Another version had a problem with zergling micro that dragged down results for months. I did not see the cause. I had to watch a great many games to train my eyes to see that zerglings were not as effective as they had been earlier, to tell which fights were going the wrong way, and then I could finally ferret it out. Zerglings are numerous; I remember focusing on the actions of a few specific lings in a fight and being satisfied, but it was an illusion that didn’t hold up when I learned to better follow the movement of the whole fight.

Most recently the latency compensation issue that caused Steamhammer to try to re-use larvas was hard to notice. I noted that Steamhammer was opening with 11 hatchery strangely often, but it can do that so I didn’t smell a bug. And the BASIL ranking was a little disappointing, but that might be because of Monster and other improving opponents. It was only when I was writing a new opening and watching production closely that I saw the blip and could start tracing it. After the fix and its friends were uploaded, Steamhammer’s ranking quickly climbed back to the range I had originally expected.

How many stealth bugs have I never noticed? I have no shortage of known bugs and severe weaknesses on my to-do list, so in a way it doesn’t matter. And yet surely I could do better if I knew more.

What is a stealth bug to one person might be an obvious problem to another. A game has more going on than a mere human can absorb in one viewing, and people pay attention to different aspects. (Flash is presumably not a mere human.) I remember a lurker target-fixation bug that Steamhammer shared with Microwave. I thought the bug was noticeable, since I pay attention to lurker micro, while Microwave’s author MicroDK had not realized it was happening.

In the end I guess there’s no moral to the story, other than to test rigorously, vigorously, and variously, which is standard software advice anyway. Pay attention to performance regressions and dig up their roots. It can never hurt to run tournaments at home.