Starcraft AI blog

Here is a list of the AIIDE 2016 players with upsets, cases in which a lower-ranked bot defeated a higher opponent. 15 of the 21 participants scored a total of 27 upsets out of 210 pairings. In most cases the upset was of an opponent only slightly ahead—the table doesn’t worry about the size of the upset. Xelnaga, well-positioned near the end of a run of bots with close scores, is the runaway upset champion; it also upset LetaBot 8 places ahead. To earn an upset you have to score worse overall, so another way to say it is that Xelnaga is the most inconsistent performer, with strengths and weaknesses that don’t always offset each other. The deepest upset is JiaBot’s upset of ZZZKBot, ranked 9 places ahead.

Upsets are interesting because they suggest weaknesses in the stronger bot. They’re clues that may point toward what’s important to fix. For example, the upsets of #2 ZZZKBot by much less successful zergs remind us that 4 pool is easy for zerg to counter; protoss and terran need specific knowledge to counter the rush, but zerg can play a standard 9 pool (which in ZvZ is also fine against other openings) as a hard counter.