archive by month
Skip to content

AIIDE 2016 results discussion

The AIIDE 2016 results are out. The top finishers, in order, are Iron, ZZZKBot, Tscmoo zerg, Letabot—half terran and half zerg. As usual, Martin Rooijackers sent me a few observations. Here are my thoughts so far.

• The tournament was played on the same maps as last year. It doesn’t call for that in the rules. We know that they’re OK maps, though.

• Martin Rooijackers says that all four top finishers were updated in between the CIG 2016 entry deadline and the AIIDE deadline a month later. [This turned out to be wrong. See the comments. I misread Martin Rooijacker’s message—he wrote that #1 Iron and #2 ZZZKBot had been updated.] You need continual work to stay at the top. It’s fierce up there.

• In particular, ZZZKBot moved up to #2, with minus scores only against fellow zergs Overkill and JiaBot. I’m curious to know what the improvements are.

• Tscmoo zerg scored even against ZZZKBot (a poor performance; zerg should easily beat the 4 pool), and worse than even against the rest of the top 5. Tscmoo owed its #3 finish to terrorizing its inferiors. Martin Rooijackers took this to mean that terran dominated. But with 2 of the top 4, I have to say that zerg is hanging in there.

• In last year’s AIIDE, UAlbertaBot and Overkill were neck-and-neck. The Bayesian Elo calculation said that UAlbertaBot was about 60% likely to be stronger, a narrow margin. Neither has been updated since. [This was also wrong! See comments.] In CIG 2016, Overkill was ahead. In this tournament, UAlbertaBot finished well ahead. These two can’t make up their minds! Apparently this time UAlbertaBot with its rush builds was better able to cope with the changing field.

• XIMP made it to #6. On SSCAIT its last update was February 2015, and I believe it was a minor adjustment to timings. The carrier strategy seems resilient. By contrast IceBot, AIIDE and CIG champion in 2014 ahead of XIMP, was in the middle of the pack. All its smarts did not make it resilient against improvements in its enemies.

• “JiaBot” in AIIDE is probably the same as “Ziabot” in CIG and “Zia bot” in SSCAIT. In some languages, the same sound can transliterate into English as either J or Z. Looking at the Ziabot source from CIG 2016, I think that the author is Korean based on variable names.

• Xelnaga finished ahead of Skynet, which surprised me. In CIG, XelnagaII was near the bottom. It seems a surprisingly sudden change.

• Tyr finished dead last by a mile, with a 1% win rate which seems so low that it must be due to bugs. Tyr had major updates this year which left it fairly strong. It finished in the middle of the pack in CIG 2016. I imagine it was updated at the last minute with a grievous error.

Tomorrow: The Elo table. I’ve done the calculations, but I have to draw up the table.

Trackbacks

No Trackbacks

Comments

krasi0 on :

Has ZZZKbot really been updated between both competitions? At least on SSCAIT, the last updated was in February. Perhaps the timestamps of some files in the source code could give us more insight?

Jay Scott on :

I don’t know. I was only repeating what Martin Rooijackers wrote to me. Presumably the source will be released soon and we’ll find out.

LetaBot on :

If you look at the DLL from the CIG 2016 compared to the AIIDE 2016, there is an increase in size.


I also got the information from the excel that dave churchill put up. But it could ofc be that someone registered but didn't submit.



https://docs.google.com/spreadsheets/d/18bOpQx2iNTb2pYrVFy3dnSh8g0KWWlqKBc3M-I7iVDg/edit#gid=0

Jay Scott on :

This is the first time I’ve seen that spreadsheet—thanks! The February ZZZKBot that I got from SSCAIT uses the same 4.1.2 BWAPI version that this ZZZKBot shows on the spreadsheet, so I don’t see evidence there either way. It does say “2016 registered,” but that’s also not strong evidence. Here’s another version mystery: Overkill is listed on the spreadsheet as the 2015 version, which is what I assumed, but when I looked at its I/O files there were a ton of new ones full of different-looking data and named to suggest reinforcement learning. If those files were actually used, then this was surely a new version of Overkill. I’m looking forward to the source release more than ever!

Jay Scott on :

... and Overkill’s github repository https://github.com/sijiaxu/Overkill has a huge update commented “2016AIIDE”. The README says “shorten the opening strategy building length and entirely rely on reinforcement learning model to select unit to build”. Not remotely the same version!

krasi0 on :

Nice! We need to somehow ask the authors to submit to SSCAIT, too! I guess many fans would like to watch improved versions on the stream :)

tscmoo on :

Unfortunately I did not have time to make a submission to AIIDE this year, so they used my entry from last year. With that in mind, it was a pleasant surprise to make 3rd.

Jay Scott on :

Aha. Thanks, I’ll add a note about the error!

Jay Scott on :

The spreadsheet reminds me that JiaBot aka Zia is written by Sungguk Cha, which is indeed a Korean name.

Add Comment

E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

Form options

Submitted comments will be subject to moderation before being displayed.