Starcraft AI blog

Yesterday I claimed that the cannon bot Jakub Trancik “has been falling slowly in the rankings year by year, even as bots that began above it fall further.” Is it true? I see room for argument, but there is something to it.

Here is a graph of the SSCAIT finishing ranks of 5 bots over 6 years, from the 2014 through 2019 editions of the round robin phase of the annual tournament. The bots were selected to have no updates over the time period; it is the same code every year, according to the info on SSCAIT’s website. (UAlbertaBot by Dave Churchill was updated in 2015, so I didn’t include its 2014 finish.) The finishing ranks are normalized so that finishing first is 100 and finishing last is 0, so that the ranks can be compared over time even though each year had a different number of participants. The graph shows old bots falling in relative performance as new and updated bots grew stronger over the years.

Jakub Trancik’s finishes were nearly flat from 2014 through 2017, and it fell in 2018. It did not participate in 2019, though it has been allowed back this year. The other non-updated bots showed declines over the period, but not always steep declines. Each bot has a visible knee in the curve, where it bent more sharply down. The year of the knee, the last year of relatively stable performance, ranges from 2016 for Tomas Cere to 2018 for Skynet by Andrew Smith. That might be because performance gains have accelerated in the last few years, or it might be because it takes that long for enough new and updated bots to be tuned against the unchanging old ones. Maybe the knees occur when flashy newcomers start to exploit specific weaknesses of the old guard.

Of these 5 bots, Jakub Trancik has the flattest curve, though it doesn’t look exceptional. It is not a statistical outlier, and Skynet’s curve is almost as level. Jakub Trancik is also the least sophisticated bot, and it has the most extreme and unconventional strategy. The facts might be related.

For comparison, here’s another chart with 4 more non-updated bots. Roman Danielis missed 2016. These curves also seem to have knees, though less sharp, and the shape of Roman Danielis’s curve is not clear to the eye.