Starcraft AI blog

Here are the graphs I promised. There is one for each bot in AIIDE 2020. Opponents are not labeled, but are arranged along the x-axis in order of strength. The green line shows the expected win rates against the opponents, based on the elos of the two bots (from yesterday). The blue line shows the actual win rate in the tournament. For purposes of charting, each bot has a fictional win rate of 50% against itself, on both the green and blue lines. So every chart shows green and blue crossing at 50%.

The green line has fundamentally the same shape in every graph, since it is based on fixed elo ratings. It’s just stretched a little differently each time. The blue line must roughly follow the green line; by construction, it can’t deviate in one direction without also deviating in the other. Notice that the scale is different on a couple of the later graphs. In particular, EggBot’s graph only goes to 50%.

It’s not easy to eyeball the upset rate as such. You have to align your eyes to the 50% point where the green and blue cross. I should have drawn vertical lines on the graphs there, but my software was not fun. Nevertheless, the general goodness of fit is easy to see, and I guess that might be just as informative. For example, you can easily tell by eye that Dragon tends to upset the strong and suffer against the weaker. The strongest players want to be solid to avoid losing, since most losses are upsets, and the weaker players want to be daring to win more, and some of that shows too. To me, it’s telling that Microwave is visibly more consistent than Steamhammer, since MicroDK aims for defensive play that avoids risk while Steamhammer aims for aggressive play and seeks risk. That is exactly the kind of difference that a solidity metric is supposed to measure.

I judge the experiment a success so far.

Next: I’ll look for a good way to turn the data into single numbers, completing the solidity metric.