Starcraft AI blog

Dan Gant let me know yesterday that the CoG 2019 (formerly CIG 2019) full results are out. They finally got a new web site up. I grabbed everything, but found that replays_04.zip is corrupt, so we are missing replays from the final 10 rounds. The SOURCE_CODE download does not contain source code.

There were 27 participants, a good number, but only 9 were new entrants, not such a good number. The remaining 18 were holdovers from previous years (this assumes that LetaBot was a new submission as registered, not a holdover as stated in the slide show; I don’t know which is correct). 40 rounds were played, numbered 0 to 39. 27*26*40/2 gives 14,040 games ideally, and they claim that 14,027 successfully made it into the results. The five maps were a version of Heartbreak Ridge (2 starting locations), Alchemist and Great Barrier Reef (a version of El Niño) (3 starts), and Neo Sniper Ridge and Python (4 starts). Alchemist is badly designed, but the others are good choices. Heartbreak Ridge and Sniper Ridge have layout similarities; I would not have included both with only 5 maps total (of course, they chose randomly). I still maintain that 5 maps are not enough to smooth out differences; if a bot does particularly well or poorly on one map, it introduces an element of luck into the results.

The result chart in the slide show does not agree with the result crosstable. The table gives #4 Iron 75.96% #5 BananaBrain 74.81%, #6 XiaoYi 72.21%. The slide show gives #5 BananaBrain 72.21% from the next entry down in the table, #6 XiaoYi 70.38% from its next entry down in the table, and so on. The error seems to be in the slide show; all the values are assigned to bots which are off by one from #5 BananaBrain until #21 Ziabot, which in the slide show shares the same 29.33% win rate as #20 Bonjwa. I’ll be careful to use the win rates from the crosstable.

I get the impression that the organizers are overburdened. Running a tournament is a ton of work. They do not seem to have the resources to verify details and get everything right. I hope they have time to go back and clean up the errors.

The participants fall into fairly neat score groups. The protoss leaders #1 PurpleWave, #2 McRave, and #3 Locutus (all independently written, by the way, with no shared code history) are at 88.56-84.9%. Then a gap, and the next group is #4 Iron to #9 BetaStar at 75.96-67.41%. Another gap, and the next group is #10 MetaBot to #13 TitanIron at 59.04-56.35%.

I can verify from its learning files that terran #6 XiaoYi at 72.21% wins is a fork of SAIDA. By the way, it is given as registering under the name XiaoYiAI, but played under the name XIAOYI. It was brand new, so probably no bot had special preparation for it by name. Nevertheless, other bots seem to have played under their names as registered, so XiaoYi was potentially given an advantage of anonymity.

#4 Iron at 75.96% win rate is the top terran and the top holdover bot from the previous year. #7 Microwave did well at 70.38%, making it the top zerg. (#22 Steamhammer is the buggy holdover from the previous year and performed miserably, as expected.)

The biggest upset by far is #24 OpprimoBot, 23.22% win rate, 28-12 versus #1 PurpleWave with 88.56% win rate. I watched a few replays and found that, in those games, PurpleWave made one probe and then stopped all production. I can only guess that OpprimoBot tickled a bug, and the bug must be triggered by the name “OpprimoBot”, since PurpleWave went wrong before learning anything else about its opponent. I imagine that the famously thorough Purple tournament preparation hit a glitch in this one case. Maybe it is related to the fact that OpprimoBot plays random on SSCAIT, but played terran here. The object file Opponents.class does mention OpprimoBot by name, along with many other potential opponents.

I will analyze the results as usual, with the colorful crosstables and stuff, but may be slower than in the past.