Starcraft AI blog

I ran test games of the new SparCraft version versus the old one, using Steamhammer versions that are otherwise identical. So far I ran a 15-game test match with a fast mutalisk strategy, plus a few scattered test games with other strategies.

The bad news: The new SparCraft is slower. Both versions have spiky, unpredictable CPU usage; small differences in the input can make large differences in the runtime. It’s common to see most runs taking microseconds and occasional runs taking milliseconds or tens of milliseconds. New SparCraft seems to spike up more often and several times higher. I often saw it spike up 4 times higher. I’m worried that in a complex situation it could bust the time limit and lose. If so, it will need an annoying mitigation like being run in a separate thread, or cutting the search short when it threatens to run too long.

The good news: The new SparCraft is sometimes much more accurate, and I haven’t found a case where it is worse. I tested by fighting the 2 versions against each other, both playing the same opening. It’s only good for testing restricted sets of units, but it is the most sensitive real game test I can do. Other tests would require running more games to make differences show over the noise. The new version scored:

• 11-4 with 11 gas 10 pool opening, zerglings into fast mutalisks
• stalemated games that take too long to finish with a mass ling strategy
• 2-1 with a mass hydra-ling strategy
• 2-1 with a dragoon strategy

Only the first result is meaningful in itself. The new SparCraft is much, much more successful with air-to-air battles between mutalisks. New SparCraft claims to support scourge (the function to test whether a unit type is supported returns true), but I can’t tell from the results. Muta vs scourge fights are as confused as ever; mutas don’t know when to attack and when to run. Also muta versus spore colony is as useless as before, to my surprise; the air units still suicide into static defense. Only muta versus muta is improved, but the improvement is giant. The new version pulled out some wins from behind, which was impressive to see.

The other 3 results only mean that the new SparCraft is not glaringly worse in those cases. It might be better, it might be the same, it’s less likely to be downright awful. I didn’t collect enough data to say more.

Of course I tested only a few unit mixes. There may be many more unit combinations it loves like muta-vs-muta, and many it hates like muta-vs-spore. I hope that by looking at melee units, ranged ground units, and ranged air units, I covered everything at least lightly, but who knows what quirks may be lurking?

My test with identical players playing identically except for the SparCraft version was designed to amplify differences between the SparCraft versions. If I didn’t restrict both sides to play the same opening, the results would not be 11-4, but probably closer. Differences in the openings would add their own effect. The difference in real world muta-vs-muta fights is probably not as big as the 3:1 that the match result suggests, because of all the other differences between bots.

To find out the real world difference, I’ll have to try it on the server.