AIIDE 2022 - first look at results
AIIDE 2022 results are out today, complete with the detailed results file. The carryovers from last year are #3 Dragon and #8 UAlbertaBot. The others are updated for this year.
My version of the crosstable. It’s identical to the official crosstable except for the presentation.
| # | bot | overall | bana | star | drag | stea | purp | mcra | micr | ualb | pylo | styx | cuny |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | bananabrain | 85.53% | 52% | 78% | 89% | 69% | 94% | 91% | 97% | 92% | 94% | 100% | |
| 2 | stardust | 81.48% | 48% | 92% | 83% | 52% | 89% | 93% | 83% | 95% | 84% | 97% | |
| 3 | dragon | 66.46% | 22% | 8% | 21% | 97% | 95% | 56% | 77% | 98% | 94% | 95% | |
| 4 | steamhammer | 62.71% | 11% | 17% | 79% | 43% | 43% | 73% | 95% | 80% | 90% | 97% | |
| 5 | purplewave | 61.17% | 31% | 48% | 3% | 57% | 84% | 50% | 66% | 87% | 96% | 89% | |
| 6 | mcrave | 46.79% | 6% | 11% | 5% | 57% | 16% | 92% | 29% | 62% | 91% | 100% | |
| 7 | microwave | 46.62% | 9% | 7% | 44% | 27% | 50% | 8% | 57% | 67% | 99% | 99% | |
| 8 | ualbertabot | 45.74% | 3% | 17% | 23% | 5% | 34% | 71% | 43% | 66% | 95% | 98% | |
| 9 | pylonpuller | 28.91% | 8% | 5% | 2% | 20% | 13% | 38% | 33% | 34% | 62% | 74% | |
| 10 | styx | 13.92% | 6% | 16% | 6% | 10% | 4% | 9% | 1% | 5% | 38% | 44% | |
| 11 | cunybot | 10.69% | 0% | 3% | 5% | 3% | 11% | 0% | 1% | 2% | 26% | 56% |
The top four finishers are the same as last year, except that #1 BananaBrain and #2 Stardust are reversed. In CoG this year #5 PurpleWave made it to second, but not in AIIDE. #2 Stardust did not overtake BananaBrain, but came closer. The top two were not far apart from each other and dominated the rest.
I’m pleased that Steamhammer was able to hold its rank, because it is only slightly improved over last year’s version. I expected to be behind #5 PurpleWave and hoped to pass #3 Dragon, since I knew Steamhammer would score well head-to-head.
Stardust had around a hundred crashes. PurpleWave, McRave, and CUNYBot had hundreds of frame timeouts each. All these bots had a chance to move up in the rankings if they hadn’t lost so often for non-play-related reasons. Bots seem to be having increasing trouble with the time limits.
Thanks to an influx of weaker opponents, #8 UAlbertaBot finished above the bottom of the table, unlike last year. I wasn’t afraid of Styx finishing high, but I’m surprised it did so poorly. In the BASIL rankings, new bot #9 PylonPuller (which has been improving fast) and #10 Styx have almost the same elo. In fact, all the tail enders have curiously low win rates. Last year UAlbertaBot scored 27%—much worse than this year—and the fairly weak FreshMeat one rank up scored 34%. This year, the weak tail enders pushed everybody else’s wins up and made the tournament seem easy for them.
by race
The table of how each bot did by opponent race. Since there is only one terran and only one random bot, it’s less informative than we might like.
| # | bot | overall | vT | vP | vZ | vR |
|---|---|---|---|---|---|---|
| 1 | bananabrain | 85.53% | 78% | 71% | 94% | 97% |
| 2 | stardust | 81.48% | 92% | 65% | 89% | 83% |
| 3 | dragon | 66.46% | - | 56% | 73% | 77% |
| 4 | steamhammer | 62.71% | 79% | 38% | 76% | 95% |
| 5 | purplewave | 61.17% | 3% | 55% | 75% | 66% |
| 6 | mcrave | 46.79% | 5% | 24% | 85% | 29% |
| 7 | microwave | 46.62% | 44% | 33% | 58% | 57% |
| 8 | ualbertabot | 45.74% | 23% | 30% | 63% | - |
| 9 | pylonpuller | 28.91% | 2% | 9% | 45% | 34% |
| 10 | styx | 13.92% | 6% | 16% | 16% | 5% |
| 11 | cunybot | 10.69% | 5% | 10% | 15% | 2% |
Every bot scored better versus zerg than versus protoss, except for Styx which was about the same. That’s the important message in the table.
Next: Map tables.
Trackbacks
No Trackbacks
Comments
Bruce on :
I expected PW and Stardust to swap places compared to CoG, since CoG is played on beefier hardware and PW greatly benefits from more processing time. Unfortunately it fell even further down the rankings because of frame timeouts despite all of the work Dan has done to make PW immune to that issue :(
There is some active discussion of the timeout issue on Discord. Some of us are advocating a loosening of the rules, as we feel the current rules set a too-strict performance envelope, but there are no conclusions as of yet.
Dan on :
In practice, it didn't at AIIDE. This is hardware-dependent, as Bruce observes; PurpleWave timed out exactly once at COG, and isn't doing anything meaningfully different that would affect performance. The most likely cause is CPU contention causing a "sleep(1ms)" to take 10+ms to return. Without access to the hardware I can't test that hypothesis, and if it's true there's frankly no way for a bot to ensure that it meets the timing rules, just to use arbitrarily little time and pray it's sufficient.
The COG/AIIDE timing rules demand a 99.6% performance SLA on hardware you can't test against; and in AIIDE's case, on a machine with 8 VMs running simultaneously. The timing rules need to change or this will continue to be an issue.
There's really nothing else for me to do as an author to meet the timing rules besides cutting features from the bot, retiring from adding new ones, and praying it's enough. But playing performance chicken isn't fun or interesting, so I'll continue to make PurpleWave as strong as I can and if tournaments want to see bots play at their best, their operators can reconsider the timing rules or hosting conditions.
Please note that none of this is criticism of tournament operators, all of whom I appreciate greatly. Hosting is a lot of work and without platforms for competition the bots would barely exist. I intend only to describe the constraints on authors and argue for a change to timing rules or hosting conditions.
Bytekeeper on :
I was expecting around 18% due to my tests, but ~14% was still ok.
I did not have much time and had to rewrite a lot of stuff. I had fun (which was the main point) and I'll try to improve it for the next tournament.
Jay Scott on :