SSCAIT scores - crashes
In the SSCAIT data, some scores are recorded as 0 or -100. I think 0 means that the bot scored no kills and no razings, so the game finished very early. Bots might legitimately lose with no kills against a rush, but I have to guess that in many cases it means the bot failed to start up or failed to do anything. I think -100 means the bot crashed. I also think that exceeding the time limit per frame ends the game without changing the scores, so -100 only means crashes, not time infractions. I don’t promise that my interpretation of the numbers is correct! I could be wrong!
Anyway, assuming that my interpretation is right, this table of -100 scores should tell us about reliability. Like yesterday, I include only games after 17 August, so that distant versions of the same bot are not lumped together. The game counts are different from yesterday, because yesterday’s table excludes games in which either side has a score of 0 or -100.
“Crashes” are games in which the bot had -100 score. “Double crashes” are games in which both sides had -100 score. “Fast crashes” are games in which the opponent had 0 score, so the crash must have happened very early or when the opponent failed to achieve anything. “Max opp score” is the largest score that the opponent had in any game where the bot crashed. “Other crashes” are crashes where the opponent has a positive score (all crashes except double crashes and fast crashes). “Mean pos opp score” is the average of the opponent’s score in these other crashes. Large opponent scores mean that the game went on for a long time before the crash. Since a crash scores as -100, there is no way to tell whether the bot was ahead or behind when it crashed.
| bot | games | crashes | crash % | double crash % | fast crash % | max opp score | other crash % | mean pos opp score |
|---|---|---|---|---|---|---|---|---|
| krasi0 | 394 | 1 | 0.25% | 0.25% | 0.00% | -100 | 0.00% | - |
| Iron bot | 331 | 8 | 2.42% | 1.51% | 0.00% | 7225 | 0.91% | 4367 |
| Marian Devecka | 302 | 30 | 9.93% | 0.33% | 0.00% | 129110 | 9.60% | 52320 |
| Martin Rooijackers | 336 | 2 | 0.60% | 0.60% | 0.00% | -100 | 0.00% | - |
| tscmooz | 330 | 3 | 0.91% | 0.91% | 0.00% | -100 | 0.00% | - |
| tscmoo | 332 | 1 | 0.30% | 0.30% | 0.00% | -100 | 0.00% | - |
| LetaBot CIG 2016 | 318 | 1 | 0.31% | 0.31% | 0.00% | -100 | 0.00% | - |
| WuliBot | 304 | 3 | 0.99% | 0.99% | 0.00% | -100 | 0.00% | - |
| Simon Prins | 318 | 19 | 5.97% | 2.52% | 3.46% | 0 | 0.00% | - |
| ICELab | 334 | 12 | 3.59% | 0.30% | 0.30% | 3500 | 2.99% | 2960 |
| Sijia Xu | 285 | 1 | 0.35% | 0.35% | 0.00% | -100 | 0.00% | - |
| LetaBot SSCAI 2015 Final | 338 | 28 | 8.28% | 0.89% | 0.59% | 60850 | 6.80% | 16196 |
| Dave Churchill | 323 | 0.00% | 0.00% | 0.00% | - | - | ||
| Chris Coxe | 223 | 1 | 0.45% | 0.45% | 0.00% | -100 | 0.00% | - |
| Tomas Vajda | 317 | 16 | 5.05% | 0.63% | 0.32% | 118435 | 4.10% | 44220 |
| Flash | 306 | 2 | 0.65% | 0.65% | 0.00% | -100 | 0.00% | - |
| Zia bot | 327 | 33 | 10.09% | 0.92% | 1.22% | 45095 | 7.95% | 23120 |
| PeregrineBot | 157 | 0.00% | 0.00% | 0.00% | - | - | ||
| tscmoop | 328 | 2 | 0.61% | 0.61% | 0.00% | -100 | 0.00% | - |
| Andrew Smith | 334 | 5 | 1.50% | 1.50% | 0.00% | -100 | 0.00% | - |
| Florian Richoux | 308 | 31 | 10.06% | 0.97% | 0.32% | 57405 | 8.77% | 14962 |
| Carsten Nielsen | 347 | 4 | 1.15% | 1.15% | 0.00% | -100 | 0.00% | - |
| Soeren Klett | 284 | 2 | 0.70% | 0.35% | 0.35% | -100 | 0.00% | - |
| Jakub Trancik | 318 | 0.00% | 0.00% | 0.00% | - | - | ||
| Tomas Cere | 345 | 3 | 0.87% | 0.00% | 0.58% | 81825 | 0.29% | 81825 |
| MegaBot | 305 | 54 | 17.70% | 1.31% | 2.95% | 92450 | 13.44% | 16394 |
| Aurelien Lermant | 349 | 4 | 1.15% | 1.15% | 0.00% | -100 | 0.00% | - |
| Odin2014 | 183 | 17 | 9.29% | 0.55% | 1.64% | 38600 | 7.10% | 22077 |
| Gaoyuan Chen | 329 | 0.00% | 0.00% | 0.00% | - | - | ||
| DAIDOES | 136 | 8 | 5.88% | 0.74% | 0.74% | 1750 | 4.41% | 4742 |
| Igor Lacik | 144 | 13 | 9.03% | 0.69% | 0.00% | 54100 | 8.33% | 13421 |
| Matej Istenik | 323 | 1 | 0.31% | 0.31% | 0.00% | -100 | 0.00% | - |
| NUS Bot | 137 | 3 | 2.19% | 0.00% | 0.00% | 34175 | 2.19% | 22400 |
| Roman Danielis | 304 | 3 | 0.99% | 0.99% | 0.00% | -100 | 0.00% | - |
| ZerGreenBot | 36 | 25 | 69.44% | 2.78% | 8.33% | 37025 | 58.33% | 22087 |
| Ian Nicholas DaCosta | 150 | 1 | 0.67% | 0.67% | 0.00% | -100 | 0.00% | - |
| AwesomeBot | 164 | 16 | 9.76% | 1.22% | 1.83% | 27850 | 6.71% | 7968 |
| Johan Kayser | 306 | 1 | 0.33% | 0.00% | 0.33% | 0 | 0.00% | - |
| Martin Vlcak | 151 | 19 | 12.58% | 0.66% | 11.92% | 0 | 0.00% | - |
| Rob Bogie | 135 | 0.00% | 0.00% | 0.00% | - | - | ||
| Christoffer Artmann | 343 | 52 | 15.16% | 0.58% | 11.95% | 700 | 2.62% | 300 |
| Marek Gajdos | 175 | 68 | 38.86% | 26.29% | 12.00% | 100 | 0.57% | 100 |
| Travis Shelton | 155 | 0.00% | 0.00% | 0.00% | - | - | ||
| Bjorn P Mattsson | 288 | 36 | 12.50% | 0.00% | 12.50% | 0 | 0.00% | - |
| Vladimir Jurenka | 349 | 37 | 10.60% | 1.43% | 2.87% | 400 | 6.30% | 264 |
| neverdieTRX | 170 | 1 | 0.59% | 0.59% | 0.00% | -100 | 0.00% | - |
| OpprimoBot | 351 | 51 | 14.53% | 0.57% | 1.42% | 38825 | 12.54% | 17424 |
| Sungguk Cha | 320 | 168 | 52.50% | 1.25% | 4.69% | 60675 | 46.56% | 17463 |
| Jacob Knudsen | 177 | 56 | 31.64% | 0.56% | 3.39% | 17750 | 27.68% | 5957 |
| HoangPhuc | 151 | 115 | 76.16% | 0.66% | 5.96% | 38050 | 69.54% | 10765 |
| ButcherBoy | 142 | 35 | 24.65% | 0.70% | 7.04% | 10000 | 16.90% | 1364 |
The top bots are mostly reliable—except Marian Devecka’s Killerbot. My impression from watching games is that Killerbot may crash when losing. It makes sense that bots should crash less often when winning, because bot authors have more reason to fix crashes that lose winning games. The bottommost bots are all crash-prone. Not a coincidence, is it? Step 1 to a winning bot: Fix your crashing bugs!
We might imagine that double crashes are caused primarily by the game or some other part of the runtime system, but the high rate of double crashes by Marek Gajdos makes me question that. What is that bot breaking?
Tomorrow: I’ll try to correlate scores with ratings, one way or another.
Trackbacks
No Trackbacks
Comments
krasi0 on :
Anyway, my point being, if a .DLL module crashes, that means that Starcraft crashes, too. That's probably the case where the score for that bot is recorded as -100. OTOH, if a Client based AI crashes, this only results in the bot player not executing any more commands but the actual game goes on until that inactive player gets eliminated or a timeout occurs.
Now, there are two cases with the crashes of a client based AI:
1) at the start of the game, which means a score of 0;
2) at some point during the game itself;
I doubt that the tournament manager can detect the second case at all due to the many types of client bots: java based, C++ based, python based, etc., so it's likely that case 2) is recorded as a regular loss (if the opponent somehow managed to finish off an inactive player),
while case 1) is probably logged as a crash with score of 0.
What are your thoughts?
krasi0 on :
Jay Scott on :
ButhcerBoy Author on :
We know when it crashes, it happens whenever it makes a gas mining station on a windows 7 machine, however it does not happen on a windows 10 machine. Because our testing environment was all windows 10 we did not know about this bug but found it after the bot started crashing on the tournament (we can see that it runs on a windows 7 machine). The project has ended so the bot will not be updated, also because we cannot very easily reproduce and test this crash, so that is a reason for that bot crashing.
Great article though, really enjoying what you're writing about the bots :)
Jay Scott on :