archive by month
Skip to content

CoG 2020 results are out

The CoG 2020 results are out today. Overall results, a crosstable, a win rate per map table, and zip files of replays for each bot were released. Source code will be delayed until after AIIDE 2020. It’s not mentioned, but I expect that the tournament manager’s log of detailed results and the bot write directories will appear eventually, and then I’ll put up my colorful crosstables and other analyses. There is also an unspecific apology “We conflicted with several problems while running the competition. Sorry that I didn’t smoothly handle them. However, I hope to meet again with better competition in the future.” Something must have gone wrong.

The results give only 8 participants of the expected 10. Of the announced entrants, newcomer random Mikhail Golovach and old-timer zerg ZZZKBot do not appear in the results. Are they related to what went wrong? The effect is to tilt the tournament further toward protoss, 5 of the 8 bots.

As I predicted (probably along with most), the top winners were #1 Stardust 88%, #2 PurpleWave 71%, #3 BananaBrain 70%. PurpleWave was last year’s version (see the 89% frame timeout rate of a recent version on the Starcraft AI Ladder), which explains why it barely nosed out BananaBrain.

The three holdover bots scored much worse this year than last, and ended up in a different order. Progress has been strong, though it’s hard to compare because the field was much smaller and less varied this year, and had no low-end bots. Tail-ender MetaBot scored over 50% last year, so the field was stronger from the get-go. (The year-old version of PurpleWave scored 88.56% last year to 70.82% this year, but I didn’t include it in the table because I don’t know whether it’s the same year-old version.)

botlast yearthis year
#4 BetaStar67.41%51.73%
#6 XiaoYi72.21%36.57%
#8 MetaBot59.04%11.02%

There were exactly 3 upsets: #3 BananaBrain upset #2 PurpleWave, #6 XiaoYi beat #4 BetaStar, and #7 McRave zerg overcame #5 Microwave. Another surprising result is that #7 McRave zerg scored only 1 win out of 200 versus #6 XiaoYi, even though XiaoYi is a holdover that McRave could have tuned against. By comparison, #5 Microwave scored 163/200 versus XiaoYi.

The per-map table shows that #1 Stardust performed about equally well on all maps. I guess its fast-mass strategy is not sensitive to map layout. All other bots were more sensitive to the map. Most strikingly, Microwave scored well on the tricky two-entrance map Alchemist, while #8 MetaBot utterly collapsed on Blue Storm.

Trackbacks

No Trackbacks

Comments

Quatari on :

There are links for the html table and plaintext Detailed Game Results at: https://cilab.gist.ac.kr/sc_competition_result/2020/html/index.html

PurpleWave was the CoG 2019 version.

I didn't get around to working on ZZZKBot so I didn't submit it this year, which is why it's missing. Congrats Stardust!

Jay Scott on :

So there are! Stand by for more tables.

Bruce on :

I understood the apology as referring to the issues up to the submission deadline, like announcing the maps early, getting them wrong initially, and confusion about the registered participants. They haven’t mentioned anything about problems during the tournament, so we likely won’t know about any potential issues there until they release the rest of the data files.

MicroDK on :

I would like to know which problems they had.... it seems Microwave did worse than in my local testing against BetaStar. My local tests showed around 50% winrate and only 25% in the tournament and I provided training data with my submission. I wonder if they had problems with read/write folders again. :(

MicroDK on :

I am not surprised by the small setup of McRave zerg. In local testing we were around 55/45% in favor of Microwave but that is vs the previous version of McRaveZ available on SSCAIT.

For this tournament I updated the learning system to take maps and detected opponent strategy into consideration inspired by Steamhammer. Also I added pathing for lurkers because of Blue Storm but I am not sure that this gave Microwave any advantages on Alchemist.

McRave on :

Proud of my lil potato for not being the worst, goal accomplished.

MarcoDBAA on :

"Progress has been strong, though it’s hard to compare because the field was much smaller and less varied this year, and had no low-end bots."

All competitions seem to become more elitist, progressively removing comparably weaker bots and new authors either don´t even try or don´t want to get completely obliterated by the strong bots and do not enter at all.

Some of the established authors stop to work on their bots and the competition will slowly dry up. Think that COG definitely shows this already. Just the same strong field, slowly getting smaller.

You can have some elite tournaments, but if everything runs (more or less) like that, it will get problematic soon enough.

Dan on :

What do you think would help encourage new authors?

Bytekeeper on :

The hurdles are huge.
I think the best approaches are:
* Good libraries
* Good bot skeletons

Almost all bot authors are trying to reinvent "all" the wheel every time (which is really a lot of fun if you do like that).

Libraries as BWEM, BWEB, FAP, ASS are a good start but people tend to use and modify but not really share (ie. FAP is modified a lot but authors keep that in their code, although a repositiory exists.).

Bot skeletons: Steamhammer basically is the only bot intended to be used as a base. But it's also competitive, and modifiying it might be overwhelming for newcomers at first.

One thing to lower the hurdles might be doing it like https://www.codingame.com/start or sport leagues do it: Have multiple tiers.

That way you can write a basic bot and "win" a lower tier competition. Currently no such sense of winning can be achieved since the top bots obliterate the field.

Jay Scott on :

Excellent points. Steamhammer truly has become too complex for a beginner to easily start with (just look at the config file). There are ways to make it easier. Also I like the idea of a novice league (especially if we can think of a better name for it). Roman Danielis is a fascinating bot and we should be able to see it play.

CoG first, but later I want to post about this discussion.

MarcoDBAA on :

Sure, a challenger league could work too. Someone would have to create a second tier league however, that would generate less interest than a normal league.

I think, that it is totally possible to run a single league with a large field (low and high tier).

You could make it more probable for bots to play vs bots with similar elo (to not have that many lopsided, and probably boring games). Just modify the random generator somewhat.

jtolmar on :

Any given version of a bot doesn't really improve (bot learning is quite limited), so there's no reason to keep playing a matchup that's already been done 100 times. If you never replay old vs old matchups, that leaves a lot more room for new vs new and new vs old. And it gives a lot less reason to remove old bots that have already had most of their matchup table filled out.

You could even leave old versions of bots that are getting updates, so the ladder permanently has, say, Steamhammer 2017 as a possible opponent. I think the ability to beat old versions of top bots would give newbie bot authors a lot of encouragement - it's a pretty clear signal that they can get to the top of they keep working at it.

There's a limit to how many versions of a fast-updating bot are okay to include (the proposal falls over if versions are added faster than it can fill out its match tables), so this would work better for a yearly or quarterly tournament than for a permanent ladder.

MarcoDBAA on :

Personally, I would go for variety, if possible, using independent AIs, instead of old versions. An exception could be, if these old versions of bots are rather different from the current version (this is true for Killerbot/Devecka for example), not just weaker.

I also suggested before, that the random version of a top bot might go full random, using the whole range of its strats, including normally inactive crazy ones. Would be an easier, but interesting opponent. Not really applicable for Randomhammer, because there are no active independent terran and protoss versions.

Old weak bots could get less games indeed, except if they are chosen as an opponent for a new one. Would not make certain matchups completely impossible however. The idea could be to first pick a new or strong bot with a higher probability, and in a second step choose a bot with similar elo as an opponent with higher probability. Most games would be "new or good" vs "similar elo", and very few games would be "old and weak" vs "dissimilar elo".

Anyway, if someone creates a tournament/ladder he has the freedom to do what he wants of course... I just think, that making all tournaments more and more "elite" will be suicidal in the long run.

MarcoDBAA on :

At least one/some competition/s need/s to keep/reenable some weaker bots. This probably needs to be a year around competition/league I think, to make it viable.

A significant selection of comparably weaker bots, that are stable (no/few crashes and timeouts, can finish games) and interesting enough (expand, different strategies maybe, use a variety of units of that race) need to be active in this competition/league. Just bots, that would be a good choice being an easy/medium AI in Starcraft, and create fun games for the new authors.

New authors bots can compete vs them, defeat them and slowly work towards beating the better bots.

If there is no land in sight however, you probably won´t even try, you won´t upload your new bot, that gets obliterated by the elite bots, and you also may get discouraged prematurely and discontinue working on it...

Well, I talked about this here already (and it might be somewhat annoying xD), and yes, I am sure to be correct. And I think, that it is clearly visible already.

Don´t make new authors feel unwanted by removing all bots with similar strength, and don´t make them feel hopeless losing nearly every game vs a restricted field of good and great bots.
If there is no ladder below, few will (be able to) climb to the higher parts of the ladder. Some may use a strong bot template like SH instead to be competitive, but even this will reduce bot variety over time.

Add Comment

E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

Form options

Submitted comments will be subject to moderation before being displayed.