AIIDE 2022 hardware version - results comparison
The alternate AIIDE 2022 tournament finished. As expected, the top beneficiary was PurpleWave. Well, McRave was the biggest beneficiary in terms of wins, because its frame timeouts fell so much, but it was only enough to pull it up one rank. I thought Steamhammer and Dragon would fall down the ranks, relative to the bots that were no longer timing out, but in fact Steamhammer held its place at #4 and Dragon dropped from #3 to #6. Huh! Steamhammer’s learning does insulate it from its opponents to some extent.
• virtual machine tournament (original)
• hardware tournament (alternate)
Here’s a table comparing the results. The numbers are changes in percentages (not percentage changes). In the vm tournament BananaBrain scored 85.53%, in the hardware tournament it scored 85.03%. The difference is -0.5%.
| # | #’ | bot | win diff | crash diff | timeout diff |
|---|---|---|---|---|---|
| 1 | 1 | bananabrain | -0.5% | 0 | -3.5% |
| 2 | 2 | stardust | -4.86% | -0.8% | -0.2% |
| 5 | 3 | purplewave | +6.27% | +0.04% | -9.88% |
| 4 | 4 | steamhammer | -1.42% | -0.04% | -0.14% |
| 6 | 5 | mcrave | +11.31% | 0 | -26.55% |
| 3 | 6 | dragon | -9.22% | +3.2% | 0 |
| 7 | 7 | microwave | -2.62% | +0.18% | -0.9% |
| 8 | 8 | ualbertabot | -1.84% | -0.04% | 0 |
| 9 | 9 | pylonpuller | +2.86% | 0 | -0.5% |
| 11 | 10 | cunybot | +2.86% | 0 | -6.01% |
| 10 | 11 | styx | -2.83% | 0 | +0.14% |
PylonPuller and CUNYBot have identical numbers in the win diff column. It’s a coincidence. Styx somehow had 3 frame timeouts running directly on hardware versus 0 on a vm. It could be bad luck. All other bots had fewer frame timeouts on hardware, unless they already had 0.
Here’s a table comparing totals and averages between the tournaments.
| column | vm | hardware |
|---|---|---|
| games | 12167 | 12043 |
| game time | 12:04 | 12:56 |
| wall time | 4:05 | 2:42 |
| game timeouts | 115 | 229 |
| crashes | 127 | 117 |
| frame timeouts | 1144 | 90 |
The hardware tournament ran games much, much faster, and yet had far fewer frame timeouts. And that is even though games ran slightly longer on average in terms of game time, so that more frames were needed.
The hardware tournament had twice as many game timeouts, where the game was adjudicated on points after 60 minutes of game time. I imagine those long games were a disproportionate source of frame timeouts. Longer games are also likely why Dragon had more crashes; it was the only bot to have an important change in crash rate.
Trackbacks
No Trackbacks
Comments
Dan on :
On top of your observations I'll note that McRave gained a lot of wins on PurpleWave, due to our games running long (like our games against Dragon) and McRave timing out more readily than PurpleWave in those games.
A lot of the long game timeouts in the second run were likely PurpleWave vs. Stardust, where for reasons I haven't attempted to debug yet, PurpleWave would reach a winning position but not attempt to clear out the main base, which in the bare metal tournament would probably result in many timeouts.