The CoG 2020 results are out today. Overall results, a crosstable, a win rate per map table, and zip files of replays for each bot were released. Source code will be delayed until after AIIDE 2020. It’s not mentioned, but I expect that the tournament manager’s log of detailed results and the bot write directories will appear eventually, and then I’ll put up my colorful crosstables and other analyses. There is also an unspecific apology “We conflicted with several problems while running the competition. Sorry that I didn’t smoothly handle them. However, I hope to meet again with better competition in the future.” Something must have gone wrong.
The results give only 8 participants of the expected 10. Of the announced entrants, newcomer random Mikhail Golovach and old-timer zerg ZZZKBot do not appear in the results. Are they related to what went wrong? The effect is to tilt the tournament further toward protoss, 5 of the 8 bots.
As I predicted (probably along with most), the top winners were #1 Stardust 88%, #2 PurpleWave 71%, #3 BananaBrain 70%. PurpleWave was last year’s version (see the 89% frame timeout rate of a recent version on the Starcraft AI Ladder), which explains why it barely nosed out BananaBrain.
The three holdover bots scored much worse this year than last, and ended up in a different order. Progress has been strong, though it’s hard to compare because the field was much smaller and less varied this year, and had no low-end bots. Tail-ender MetaBot scored over 50% last year, so the field was stronger from the get-go. (The year-old version of PurpleWave scored 88.56% last year to 70.82% this year, but I didn’t include it in the table because I don’t know whether it’s the same year-old version.)
|bot||last year||this year|
There were exactly 3 upsets: #3 BananaBrain upset #2 PurpleWave, #6 XiaoYi beat #4 BetaStar, and #7 McRave zerg overcame #5 Microwave. Another surprising result is that #7 McRave zerg scored only 1 win out of 200 versus #6 XiaoYi, even though XiaoYi is a holdover that McRave could have tuned against. By comparison, #5 Microwave scored 163/200 versus XiaoYi.
The per-map table shows that #1 Stardust performed about equally well on all maps. I guess its fast-mass strategy is not sensitive to map layout. All other bots were more sensitive to the map. Most strikingly, Microwave scored well on the tricky two-entrance map Alchemist, while #8 MetaBot utterly collapsed on Blue Storm.