Starcraft AI blog

The annual AIIDE, CIG, and SSCAIT tournaments give us snapshots of bot strength at particular moments in time. The SSCAIT ladder used to let us gauge strength at other times as well, but then voting was added, and voting distorts the ratings so that we only get a general idea. To add voting made sense, because it helps keep people engaged, and that (as I see it) is SSCAIT’s main goal. Still, we lost a useful ability.

To me, following Steamhammer’s results, the elo distortion caused by voting is unmissable. Since November or earlier, and except during the tournament when voting wasn’t allowed, every time Steamhammer’s rating rose to around 2200, the voters soon took interest and fed it a sequence of opponents that it would lose to, hammering the rating back down. Then voters lost interest and its rating gradually rose again. Watching this process helped me make my correct prediction that Steamhammer would finish in the SSCAIT tournament between places #4 to #8, even though the bot could not maintain such a high rank outside the tournament.

BWAPI bots ladder fills the gap. (It doesn’t seem to have an official name.) There is no voting or streaming; it only plays games. Since it doesn’t slow down games to make them watchable, it also plays more games. The larger amount of data should also improve the ratings (though it depends on the K factor in the elo calculation).

The UI is sparse. Presumably the project is in an early stage of development, just finished enough to make public. The web page doesn’t provide information about plans. I’ve sent a list of questions to the contact address.

A few points:

• The ladder seems to have been pre-populated with SSCAIT bots. It is a decision that can be questioned: The bot authors did not give permission for this public use. An author who doesn’t like it can write to the contact address to have their bot removed.

• The maps are the SSCAIT maps.

• Minimal public information. The bot names and game results and replays are made public; nothing else is revealed. It is an intentional choice. I don’t even see a way to find out the upload time of a bot, so that you can try to distinguish versions.

• Random bots are allowed, but the tournament manager chooses the race so that the opponent knows it when the game starts. This is different from playing random. It reflects the opinion that playing random gives an unfair advantage.

I personally disagree with the treatment of random players. I think that decisions about fair balance should be made on the basis of data, not argument, and that we don’t have the data. One issue is that a random bot is more difficult to create (no matter whether it is taught its knowledge by hand or by machine learning), which you could take as counterbalancing any advantage it might gain by playing random. Another issue is that a bot which wants to play against random opponents is not able to on this ladder. Of course, in the end the organizers are doing the work, and they get to make the decisions.

Overall, the appearance of a new ladder is a good sign of the health of the community. It fills a gap: It provides a better continuous measure of how well different bots are doing than we have had before. Those who like the design decisions may prefer it over other competitions, and those who don’t also gain by living in a richer world.