AIIDE 2017 unattributed crashes
In AIIDE 2017, the tournament manager launched some games that did not start. These games were recorded with duration 0 and score 0 for both sides, and were ignored in the official tally. In the detailed results HTML page, the games are listed as crashes with the crashed player being “unknown”. I think of these games as unattributed crashes: If one bot identifiably crashed, then that bot lost the game. But some games failed without either bot crashing in a way that the tournament manager recognized and attributed to the bot, and those games had to be skipped.
And yet, looking at how often bots appeared in “unknown” crash games, there is one obvious conclusion. The % column here is the percentage of unattributed crash games that the bot participated in. Each unattributed crash game has 2 participants, so the percentages add up to 200% before rounding (even though the column total says 100%).
| bot | crashes | % |
|---|---|---|
| ZZZKBot | 4 | 2.20% |
| PurpleWave | 7 | 3.85% |
| Iron | 5 | 2.75% |
| cpac | 7 | 3.85% |
| Microwave | 8 | 4.40% |
| CherryPi | 4 | 2.20% |
| McRave | 6 | 3.30% |
| Arrakhammer | 7 | 3.85% |
| Tyr | 4 | 2.20% |
| Steamhammer | 6 | 3.30% |
| AILien | 4 | 2.20% |
| LetaBot | 15 | 8.24% |
| Ximp | 8 | 4.40% |
| UAlbertaBot | 2 | 1.10% |
| Aiur | 5 | 2.75% |
| IceBot | 15 | 8.24% |
| Skynet | 12 | 6.59% |
| KillAll | 5 | 2.75% |
| MegaBot | 168 | 92.31% |
| Xelnaga | 8 | 4.40% |
| Overkill | 12 | 6.59% |
| Juno | 8 | 4.40% |
| GarmBot | 9 | 4.95% |
| Myscbot | 6 | 3.30% |
| HannesBredberg | 6 | 3.30% |
| Sling | 7 | 3.85% |
| ForceBot | 10 | 5.49% |
| Ziabot | 6 | 3.30% |
| total | 182 | 100% |
With these numbers in hand, the great majority of unattributed crashes can be attributed after the fact to MegaBot. MegaBot may have a bug that sometimes breaks the tournament infrastructure. Likely the bug is in the infrastructure itself, and MegaBot happens to tickle it—and other bots do too, though less often.
As a side effect, MegaBot’s official score could be considered too high. If we see the unattributed crashes with MegaBot as “MegaBot’s fault,” then the games should not be skipped in the results, but counted as wins for the opponent and losses for MegaBot. The change is unfair, though: Even if the bug is in MegaBot, which we do not know, then surely not all of the unattributed crashes are due to MegaBot. Other bots or the infrastructure must be responsible for some.
Running a big tournament is hard....
Trackbacks
No Trackbacks
Comments
Dave Churchill on :