CoG 2020 results are out
The CoG 2020 results are out today. Overall results, a crosstable, a win rate per map table, and zip files of replays for each bot were released. Source code will be delayed until after AIIDE 2020. It’s not mentioned, but I expect that the tournament manager’s log of detailed results and the bot write directories will appear eventually, and then I’ll put up my colorful crosstables and other analyses. There is also an unspecific apology “We conflicted with several problems while running the competition. Sorry that I didn’t smoothly handle them. However, I hope to meet again with better competition in the future.” Something must have gone wrong.
The results give only 8 participants of the expected 10. Of the announced entrants, newcomer random Mikhail Golovach and old-timer zerg ZZZKBot do not appear in the results. Are they related to what went wrong? The effect is to tilt the tournament further toward protoss, 5 of the 8 bots.
As I predicted (probably along with most), the top winners were #1 Stardust 88%, #2 PurpleWave 71%, #3 BananaBrain 70%. PurpleWave was last year’s version (see the 89% frame timeout rate of a recent version on the Starcraft AI Ladder), which explains why it barely nosed out BananaBrain.
The three holdover bots scored much worse this year than last, and ended up in a different order. Progress has been strong, though it’s hard to compare because the field was much smaller and less varied this year, and had no low-end bots. Tail-ender MetaBot scored over 50% last year, so the field was stronger from the get-go. (The year-old version of PurpleWave scored 88.56% last year to 70.82% this year, but I didn’t include it in the table because I don’t know whether it’s the same year-old version.)
| bot | last year | this year |
|---|---|---|
| #4 BetaStar | 67.41% | 51.73% |
| #6 XiaoYi | 72.21% | 36.57% |
| #8 MetaBot | 59.04% | 11.02% |
There were exactly 3 upsets: #3 BananaBrain upset #2 PurpleWave, #6 XiaoYi beat #4 BetaStar, and #7 McRave zerg overcame #5 Microwave. Another surprising result is that #7 McRave zerg scored only 1 win out of 200 versus #6 XiaoYi, even though XiaoYi is a holdover that McRave could have tuned against. By comparison, #5 Microwave scored 163/200 versus XiaoYi.
The per-map table shows that #1 Stardust performed about equally well on all maps. I guess its fast-mass strategy is not sensitive to map layout. All other bots were more sensitive to the map. Most strikingly, Microwave scored well on the tricky two-entrance map Alchemist, while #8 MetaBot utterly collapsed on Blue Storm.
Trackbacks
No Trackbacks
Comments
Quatari on :
PurpleWave was the CoG 2019 version.
I didn't get around to working on ZZZKBot so I didn't submit it this year, which is why it's missing. Congrats Stardust!
Jay Scott on :
Bruce on :
MicroDK on :
MicroDK on :
For this tournament I updated the learning system to take maps and detected opponent strategy into consideration inspired by Steamhammer. Also I added pathing for lurkers because of Blue Storm but I am not sure that this gave Microwave any advantages on Alchemist.
McRave on :
MarcoDBAA on :
All competitions seem to become more elitist, progressively removing comparably weaker bots and new authors either don´t even try or don´t want to get completely obliterated by the strong bots and do not enter at all.
Some of the established authors stop to work on their bots and the competition will slowly dry up. Think that COG definitely shows this already. Just the same strong field, slowly getting smaller.
You can have some elite tournaments, but if everything runs (more or less) like that, it will get problematic soon enough.
Dan on :
Bytekeeper on :
I think the best approaches are:
* Good libraries
* Good bot skeletons
Almost all bot authors are trying to reinvent "all" the wheel every time (which is really a lot of fun if you do like that).
Libraries as BWEM, BWEB, FAP, ASS are a good start but people tend to use and modify but not really share (ie. FAP is modified a lot but authors keep that in their code, although a repositiory exists.).
Bot skeletons: Steamhammer basically is the only bot intended to be used as a base. But it's also competitive, and modifiying it might be overwhelming for newcomers at first.
One thing to lower the hurdles might be doing it like https://www.codingame.com/start or sport leagues do it: Have multiple tiers.
That way you can write a basic bot and "win" a lower tier competition. Currently no such sense of winning can be achieved since the top bots obliterate the field.
Jay Scott on :
CoG first, but later I want to post about this discussion.
MarcoDBAA on :
I think, that it is totally possible to run a single league with a large field (low and high tier).
You could make it more probable for bots to play vs bots with similar elo (to not have that many lopsided, and probably boring games). Just modify the random generator somewhat.
jtolmar on :
You could even leave old versions of bots that are getting updates, so the ladder permanently has, say, Steamhammer 2017 as a possible opponent. I think the ability to beat old versions of top bots would give newbie bot authors a lot of encouragement - it's a pretty clear signal that they can get to the top of they keep working at it.
There's a limit to how many versions of a fast-updating bot are okay to include (the proposal falls over if versions are added faster than it can fill out its match tables), so this would work better for a yearly or quarterly tournament than for a permanent ladder.
MarcoDBAA on :
I also suggested before, that the random version of a top bot might go full random, using the whole range of its strats, including normally inactive crazy ones. Would be an easier, but interesting opponent. Not really applicable for Randomhammer, because there are no active independent terran and protoss versions.
Old weak bots could get less games indeed, except if they are chosen as an opponent for a new one. Would not make certain matchups completely impossible however. The idea could be to first pick a new or strong bot with a higher probability, and in a second step choose a bot with similar elo as an opponent with higher probability. Most games would be "new or good" vs "similar elo", and very few games would be "old and weak" vs "dissimilar elo".
Anyway, if someone creates a tournament/ladder he has the freedom to do what he wants of course... I just think, that making all tournaments more and more "elite" will be suicidal in the long run.
MarcoDBAA on :
A significant selection of comparably weaker bots, that are stable (no/few crashes and timeouts, can finish games) and interesting enough (expand, different strategies maybe, use a variety of units of that race) need to be active in this competition/league. Just bots, that would be a good choice being an easy/medium AI in Starcraft, and create fun games for the new authors.
New authors bots can compete vs them, defeat them and slowly work towards beating the better bots.
If there is no land in sight however, you probably won´t even try, you won´t upload your new bot, that gets obliterated by the elite bots, and you also may get discouraged prematurely and discontinue working on it...
Well, I talked about this here already (and it might be somewhat annoying xD), and yes, I am sure to be correct. And I think, that it is clearly visible already.
Don´t make new authors feel unwanted by removing all bots with similar strength, and don´t make them feel hopeless losing nearly every game vs a restricted field of good and great bots.
If there is no ladder below, few will (be able to) climb to the higher parts of the ladder. Some may use a strong bot template like SH instead to be competitive, but even this will reduce bot variety over time.