Starcraft AI blog

We don’t get many tournaments of bots versus humans. I don’t think there have been any with conditions controlled well enough that we can judge how strong bots are and how they are improving: Enough human participants, of known strength, with known levels of familiarity with computer play, finishing enough games. Then hold events across years so we can compare. We have to make do with seeing how bots are improving against other bots. Here is my best idea so far for comparing strength across tournaments.

1. We need 2 tournaments, preferably round robin, that share some participants—exactly identical bots, the more the better. We can’t do it with humans, because we can’t get exactly identical people across time. Ideally the maps should be the same too. AIIDE has more games, and SSCAIT has more shared participants; either should work, but I think SSCAIT may work better for this purpose despite being short by comparison. You could also compare between AIIDE and SSCAIT, but it would not work as well. It would take extra effort to make sure you know which players are exactly identical, and the different lengths of the tournaments means each provides a different amount of evidence to support the ratings, plus you could get confusing results for learning bots.

2. Pool all the games from both tournaments and compute elo ratings. If some participants which are not identical have the same names, distinguish them somehow—Steamhammer 2017 versus Steamhammer 2018, or whatever.

3. The identical players have identical strength in both tournaments, so consider their elo ratings as fixed. For each tournament separately, compute the elo ratings of the remaining players while keeping the ratings of the identical players fixed. The fixed ratings are benchmarks that keep the elo comparison stable for the remaining players (the idea has been used before).

It’s the best way I’ve thought of to get strength comparisons across time. We can get a pretty accurate measure of how individual bots have improved—Steamhammer 2018 is this much above Steamhammer 2017. We can treat elo as a linear measure of strength (a given elo interval always represents the same win rate difference), so we can simply average together the ratings of any set of bots to compare: The top 16 are x points stronger this year, the protoss are y points stronger, the spread between best and worst has widened to....

I may do this analysis for SSCAIT once it finishes. It’s a bit elaborate, but I’m interested.

A few unrelated notes about Steamhammer:

The bug that causes Steamhammer to drop commands is due to a missing & in an inconspicuous declaration, causing a data structure to be copied instead of referred to. Updates are made to the copy instead of the correct data, then the copy is thrown away. Even after I deduced that something was being copied behind the scenes, it was tricky to nail the exact mistake.

I developed an opening that I feel I can properly call Fried Liverpool. Like the Fried Liver Attack from chess, mentioned in a comment, it’s crazy sharp and can put on tremendous pressure. Steamhammer can’t play it yet; it needs a couple new features. I tried it by hand and found it is effective against unprepared opponents. Maybe I’ll get it working in time for version 2.2.1 or thereabouts (the one after the upcoming 2.2).

Steamhammer just lost a game against the cannon bot Jakub Trancik. I don’t remember another loss against Jakub Trancik since early last year (maybe I have a bad memory). During the tournament I can’t log in to fetch the game records, but I assume it is the first game against this opponent since version 2.0. It takes a long time to collect enough game records. I’m glad I fixed the proxy recognition bug, or Steamhammer might have lost the next game too.

The SSCAIT round robin phase is halfway over, a good time to take stock. Plenty of places will yet change hands, but the ranks are starting to firm up. #1 SAIDA is on top with less than half the loss rate of #2 Locutus. #3 Iron and #4 PurpleWave follow, close to each other. Then comes a gap, and the rest of the possible top 16 are more closely spaced.

#3 Iron, #7 KrasioP, and #8 Skynet by Andrew Smith are performing better than I expected. Iron is still reliable at rolling up the lower end. KrasioP’s game plan of cannon push into dark templar into carriers is more effective than I anticipated—each step sets the opponent a different problem. Overall, it’s striking how much stronger the field is this year than last. #13 Bereaver and #14 Tscmoo protoss have been pushed well down the rankings. When the round robin is over, I want to do an analysis to compare the win rates of unchanged bots; SSCAIT has many more unchanged entrants than other tournaments.

#9 Steamhammer looks likely to finish at 9 or 10 (the middle of my forecast range of 7 to 13 or one slot above). Most of Steamhammer’s losses are unexpected; it’s annoying, but the same happened last year so it’s not a surprise. In upsets, #2 Locutus lost a game to #36 Ecgberht, and #3 Iron lost a game to #49 AIUR by Florian Richoux—everybody has surprise results, except for SAIDA whose only 2 losses are to high-ranked opponents. Even tail-ender #72 FergalOGrady, which fails to start most games and plays awkwardly when it does start, has a win over #25 ICEbot (though it’s a win by crash as its last buildings were being destroyed).

Getting into the top 16 of 72 bots is not easy. Most of the finalists can be predicted now, though not with certainty. Place 16, just below #15 Microwave, has high odds of changing hands; those above range from likely to nearly sure to be in the finals. Pairings in the finals have historically pitted the top finisher against the last finalist and so on toward the middle. Even once you’ve made it into the top 16, you can benefit from climbing up a place (except that climbing from 9 to 8 makes no difference).