Starcraft AI blog

The Starcraft AI Ladder does not display crosstables or per-map results. I wanted to see the charts to know Steamhammer’s strengths and weaknesses, so I calculated them myself. I modified the script I use to analyze the CoG and AIIDE tournament results every year. The tournament manager’s results file is now in CSV format, a change, but of course it was no trouble to parse. The pop-up table legend explains how to interpret the results to know whether to count each game, only referring to a “Duration” column which the file itself names “Game Time” (to distinguish it from “Wall Time”), and which has value “0:00” on an unstarted game rather than “00:00:00”. My script skipped a total of 3 games out of 2793, all of them with PurpleWave as one player and all due to GAME_STATE_NOT_UPDATED_60S_BOTH_BOTS.

I found that the “Download Search Results” did not behave quite as its name suggests. It seemed to perhaps remember a previous search rather than the current setting, or anyway something unexpected. But after a couple tries I was able to get the complete record of games played since the last reset on 17 April (just a couple days ago). I trimmed off the incomplete round 133, so the file I analyzed includes all games of rounds 0 through 132. 2793 games in 2 days is a great number, far more than BASIL plays.

The ladder would be more valuable if it had more participants. As it is, I am learning from it, because nowhere else runs so many games so quickly.

crosstable

BananaBrain is on top in this small field. Steamhammer is doing well; it scores nearly a third versus BananaBrain, wins most games versus Microwave, and is about equal with Halo by Hao Pan. Thanks to the huge number of games, Steamhammer’s learning is saturated so this should be its peak performance. Steamhammer eked out a slight overall lead over Microwave only due to its dominating head-to-head results; against every other bot, Microwave scored better.

The version of PurpleWave is broken; it crashes or oversteps a frame time limit most games. I have to imagine that fixes are progressing in the workshop. I suspect that this version of ZZZKBot may not be working perfectly either, but I didn’t look into it.

each bot’s results per map

BananaBrain barely noticed opponents other than Steamhammer and Microwave. Against Steamhammer it had trouble on the map Tau Cross, and against Microwave on Benzene. Checking the mix of strategies played on those maps would probably explain the cause.

Steamhammer’s results vary strongly from map to map. I think it is a sign that the opening selection is not paying enough attention to the map. I should have gone with a proper Bayesian calculation rather than an ad hoc algorithm.

Microwave also shows a lot of variation from map to map. That’s harder for me to interpret, even though it is the same evidence: Microwave has fewer openings overall than Steamhammer, so it is possible that poor results on some maps are due to not having an appropriate strategy available. Of course it could also be that the opponent’s play is much stronger on some maps.

Halo by Hao Pan seems to have consistent trouble on Aztec, the only map in the pool with a low-ground main and a ramp up to the natural. That could be the cause. Most bots underestimate the difficulty of defending the main from enemies on high ground.

For Steamhammer, Ecgberht is a tricky opponent that can sometimes pull surprise wins. Other bots don’t seem to have the same experience.

PurpleWave crashes every game on Aztec, and frequently on other maps. :-(