CIG 2018 - bots that wrote data
The CIG organizers have released the final read/write folders for the 2018 tournament. I looked through all the folders to see if each bot recorded information. If it saved nothing, it did not learn. If it saved some data, it may have used it for learning (or it might be log files or whatever). I also added a curve “↕” column, whether the bot’s win rate moves up or down (or is approximately flat) between round 40 and the end of the tournament—is the bot improving until late in the tournament? Win curves early in the tournament are noisy, so they’re hard to compare. (If anybody can’t see the Unicode up and down arrows, let me know and I can change them.)
Some bots have files in their AI folder, which may be prepared data or pre-learned data for specific opponents. I note that too. Prepared data could be kept elsewhere, including in the binary, so I didn’t see all of it. We know that PurpleWave had extensive preparations for specific opponents.
As has been mentioned, bots in Java or Scala (bots which run on the jvm) were unable to write learning data. Those that depend on their learning data were playing at a severe disadvantage. #2 PurpleWave lost narrowly to #1 Locutus and was one of the affected bots. It’s a serious problem for a tournament that wants to be taken seriously.
| # | bot | ↕ | info |
|---|---|---|---|
| 1 | Locutus | ↑ | Prepared data for 11 opponents. Learning data very similar to Steamhammer’s. |
| 2 | PurpleWave | - | jvm :-( |
| 3 | McRave | - | Looks like wins and losses of each of 16 available strategies for the previous 8 games. Perhaps a sliding window? |
| 4 | tscmoo | ↑ | Looks like strategy and win/loss info for each opponent, in a hard-to-read structured format. Past years have had more elaborate data. |
| 5 | ISAMind | ↓ | Prepared file that looks like neural network learning data. Per-opponent learned data that looks like Steamhammer data. ISAMind is based on Locutus, so that makes sense. |
| 6 | Iron | ↓ | Nothing. |
| 7 | ZZZKBot | - | Game records, one game per line, in an opaque format that looks about the same as last year. |
| 8 | Microwave | ↑ | For each opponent, wins and losses for 8 different strategies. |
| 9 | LetaBot | ↑ | Information about a few recent games against ZiaBot, probably not used for learning. |
| 10 | MegaBot | ↓ | Extensive log data. The apparent learning files are MegaBot-vs-[opponent].xml and give scores for NUSBot, Skynet, Xelnaga (MegaBot’s three heads). |
| 11 | UAlbertaBot | ↑ | Win/loss numbers for 4 protoss, 4 terran, and 5 zerg strategies. But the same strategy was always chosen for each race, so learning was turned off. |
| 12 | Tyr | ↓ | jvm :-( |
| 13 | Ecgberht | ↓ | jvm :-( |
| 14 | Aiur | ↑ | The familiar lists of numbers for each opponent. |
| 15 | TitanIron | ↓ | Nothing. |
| 16 | ZiaBot | ↓ | One file with data for TerranUAB, UAlbertaBot, and 3 lines for SRbotOne. Zia’s learning looks broken or disabled. |
| 17 | Steamhammer | ↑ | Steamhammer saved data when it did not crash, and successfully learned a little bit. |
| 18 | Overkill | ↑ | A file for each opponent, game records with opponent/opening/score. |
| 19 | TerranUAB | - | Nothing. |
| 20 | CUNYbot | ↓ | One file output.txt with strategy information and numbers, naming a few opponents but not most. A prepared file in the AI folder has the same format. It’s mysterious. |
| 21 | OpprimoBot | ↓ | Nothing. |
| 22 | Sling | ↓ | Nothing. |
| 23 | SRbotOne | ↑ | A large number of “stats” files named with date and time, apparently game records. For each opponent, another file giving the strategy “Terran_Attrition” and win/loss numbers. I’m not sure whether this could be learning data, but the bot did earn an up arrow. |
| 24 | Bonjwa | ↑ | Nothing. |
| 25 | Stormbreaker | ↑ | Prepared data NN_model_policy and NN_model_stateValue, apparently neural network learning data. For each opponent, game records with 4 numbers per game. The format is like Overkill’s but records more information. |
| 26 | Korean | - | Nothing. |
| 27 | Salsa | - | Nothing. |
Most striking is that the “nothing” bots cluster toward the bottom. If you don’t even try to record data, either you are Iron or you performed weakly. The jvm bots, which in fact recorded nothing due to no fault of their own, still placed higher than all the nothing bots other than Iron. Perhaps recording data is a proxy for how much effort has gone into the play.
Some bots had a rising win rate (an up arrow) despite doing no learning, most distractingly UAlbertaBot. I think that since UAlbertaBot plays random, its opponents can easily get confused about it. In general, I think that playing unpredictably (either being random or choosing varied openings randomly) can mess up the learning of some other bots.
I will be analyzing what certain bots learned. It will shed light on their opponents.
Trackbacks
No Trackbacks
Comments
Dan on :
I expect against most opponents the original settings were optimal or nearly optimal. The three opponents where it probably had the biggest impact were:
* Locutus (It picked proxy gates every game on maps where they're enabled, or something random otherwise).
* McRave (It picked random strategies every game)
* tscmoo (It picked the same possibly-arbitrary strategy every game; not actually sure which one)
Against everyone else, it used the fixed (or initially suggested) strategy I picked for each opponent. Generally that was intended to be a fixed build vs. bots that were returning without updates, and a "try this for 5 games" vs. opponents who I expected to be similar.
---
This bug impacted the Java bots last year, too. PurpleWave probe rushed every game in CIG 2017 because that was the "try this first" strategy. At the time I thought the bug was on my end; it was only because they released the run_proxy.bat files (and due to Chris' diligent investigation) that we now understand the problem.
---
I have no hard feelings about any of it, by the way. In sports, when you lose a game due to referee calls, the real takeaway is that you should've been so far ahead that the referee calls don't matter :)
Running tournaments is hard, getting all the bots working is hard, and when you're in a sport built on automating a series of hacks memory-injected into a 20 year old game it's expected that things will go awry. It's a miracle any of it works at all and I appreciate the efforts of everyone putting these events together.
LetaBot on :
McRave on :
There's nothing hardcoded in my bot to choose only 8 records of data, so that means plenty of my games didn't choose other strategies.
Jay Scott on :
Jay Scott on :
Edmund nelson on :
JVM bots performed an average placing of 9th
Nothing bots performed an average of 17.89th
Bots that do write data performed an average of 12th
now this isn't great evidence because the JVM sample is literally N=3 but still, I think this might be a case of better bots record data moreso than recorded data makes bots a lot better.
(though n=3 is terrible and one shouldn't base their beliefs aroudn evidence that weak)
Ecgberht on :
I dont think It would have done much better but because of its variety of builds It would have played way more interesting games and maybe have taken way more games against the top bots.
Lets see how AIIDE ends, maybe It will do better than I expected.
MicroDK on :