AIIDE 2018 - 2 locutusoids dropped
Dave Churchill decided to drop the locutusoid bots BlueBlueSky and ISAMind from the AIIDE 2018 tournament results. They were, he must have concluded, too similar to Locutus. The locutusoids #3 CSE and #5 DaQin were kept. The change switches the order of 2 pairs of participants who had close scores: Iron moved ahead of McRave, and Steamhammer moved ahead of ZZZKBot. Steamhammer is now officially #8 out of 25 (no longer #11 out of 27). #12 Tyr, with a 48% win rate, is now in the upper half of the rankings, which reinforces how much stronger the top bots were.
I only noticed today. No doubt I’m behind the times.
The change puts my analysis out of sync with the official results. You may have to refer to my result tables instead of the official tables, to see the ranking numbers I used.
Trackbacks
No Trackbacks
Comments
McRave on :
2018 AIIDE Competition Update:
After receiving feedback from several members of the SCAI community about their concerns over Locutus forks, I've made the following ruling about the 2018 official results:
ISAMind and BlueBlueSky will be disqualified from the 2018 competition for not making enough changes to the underlying Locutus bot to be considered a unique entry. This is of course a subjective decision, but was based on several factors such as code diffs, results similarity, strategic changes, etc. Other bots in the competition that were based on Locutus had significant changes, and definitely are not trivial clones.
Luckily due to the round robin format of the tournament we are easily able to remove those two bots from the official results list and re-parse the results of the tournament as if they had never entered in the first place. The only way this could change any overall results unfairly is if someone used learned data about those two bots in matches against other bots, which we don't think happened.
Here are the updated results:
http://www.cs.mun.ca/~dchurchill/starcraftaicomp/2018/
This is a hard balance to achieve: we want to continue to encourage open source development and sharing of code as we feel this is the most important reason the AIIDE competition exists in its current form. However, we do not want a bunch of trivial clones entering the competition since they do not advance the state of the art in any meaningful way.
Next year's competition will also feature more strict / concise rules about disclosure of underlying bot source code, and declaring affiliations.