Starcraft AI blog

#3 BananaBrain had pre-trained data, 3000 (!) games versus Stardust and 100 each against PurpleWave, Dragon, McRave, Microwave, and DaQin. Other opponents did not rate. The main tables here include only tournament games, not pre-training games. BananaBrain scored higher in training games than in tournament games against every opponent except the carryover DaQin (training 61% of 100 games, tournament 66% of 150 games, statistically close enough). It looks as though BananaBrain might have won the tournament if it had played against the same versions of opponents that it trained against. I take it as a sign that secret tournament improvements may be worth it.

A couple conclusions: 1. Enemy strategy recognition seemed to have some misfires with terran opponents. 2. BananaBrain would have scored slightly better if it had never played the Stove. It’s a for-fun build more than a for-real build—though bots are often poor at adaptation, so perhaps on BASIL it is more effective than here.

BananaBrain was the only bot to leave a dent in Stardust. Its plan recognition was not precise enough to pin down Stardust’s 4 gate goon strategy consistently, likely because Stardust ejected the bananascout before the build was complete. Still, BananaBrain’s 4 gate goon opener defeated Stardust’s 4 gate goons 65% of the time in 26 games. BananaBrain is deliberately unpredictable, but against this opponent consistency may have been better: Stardust played without learning, so its opponents should have sought the single best opening as an answer rather than the best mix of openings. It’s possible that #3 BananaBrain could have upset Stardust if it had done that, though it would not have gained enough wins to pass #2 PurpleWave.

opening	games	wins	first	last
PvP_10/12gate	60	3%	11	2934
PvP_12nexus	56	0%	4	2915
PvP_2gatedt	112	31%	8	2978
PvP_2gatedtexpo	278	53%	5	2921
PvP_2gatereaver	1110	66%	6	2988
PvP_3gaterobo	60	3%	2	2935
PvP_3gatespeedzeal	116	26%	7	2916
PvP_4gategoon	56	0%	3	2917
PvP_9/9gate	136	36%	12	2995
PvP_9/9proxygate	437	61%	13	2936
PvP_nzcore	56	0%	1	2918
PvP_zcore	102	21%	0	2989
PvP_zcorez	72	12%	9	2986
PvP_zzcore	349	67%	10	2999
14 openings	3000	51%

This is the table of pre-trained games. It looks different from the tournament table; the overall score and the individual results by strategy do not match up. The training may have been against an older version of Stardust on the Starcraft AI Ladder, or it may have been against Locutus, which was wrapped around an encrypted Stardust binary on SSCAIT and played instead of Stardust if it didn’t have its encryption key.

The misleading training cannot have helped BananaBrain’s results. It did adapt, but notice that BananaBrain’s most effective counter of 4 gate goon was only played 26 times in 150 games, less often than P_2gatereaver and P_zzcore which scored highest in training.

Again, 4 gate goon was BananaBrain’s best, though zealot-zealot-core was statistically equal. PurpleWave learns, so mixing it up was likely correct. Judging by the names, the last 4 builds here, the ones that mention “core”, are nonspecific tech builds that might aim for any tech and unit mix in the midgame. Do the names accurately describe them?

Again the training data was somewhat misleading (52% win rate in 100 training games and entirely different best counters), but BananaBrain tried its strategies in a fairly even distribution so I think it made little difference in this case. Presumably 100 games of training provide a smaller bias than 3000 games.

BananaBrain could not recognize most of Dragon’s builds. It looks like Dragon was vulnerable to mass zealots, an important weakness, and to hidden dark templar, which any terran might die to if unscouted. But what is the Stove doing there? Does it work against decent terran bots? Maybe the key is that there are few mid-rank terran bots, they are mostly in the upper and lower tiers.

67% overall win rate in training, but this time BananaBrain’s best counters matched between training and the tournament. The training helped, even though it was against a weaker version of Dragon.

According to McRave’s tables, McRave played overpool every game against BananaBrain. BananaBrain was able to correctly recognize that most of the time, but strangely won at a higher rate when recognition failed. Does BananaBrain perhaps have a reaction to overpool which was detrimental in this case? Or was BananaBrain perfectly right, and McRave sometimes slips up in its build order and goes 9 pool instead of overpool? I think it’s more likely that BananaBrain misrecognized it.

BananaBrain’s openings scored roughly similarly against Microwave, only a couple below 50% and only one above 75%. I think that argues that Microwave is well balanced in its skills.

The 2 gate zealot openings whomped Steamhammer. Steamhammer survives the zealot attack in many cases, but usually by spending more than it can afford so that it falls far behind in economy.

ZZZKBot is the first opponent that BananaBrain outclassed. It did not much matter what protoss played.

It’s interesting that BananaBrain recognized a terran wall in one game. UAlbertaBot does not know how to build a wall, and in fact places its early buildings near the command center. Terran is also the opponent race with the highest number of unrecognized (“unknown”) builds. BananaBrain suffered recognition trouble against Dragon too.

The Stove did not work well against WillyT.

The Stove did as well as anything against Ecgberht. But then, only one opening had any losses (it had 2 losses).

Steamhammer’s tables are fuller than others, partly because Steamhammer records more information than most bots and partly because I put more effort into analyzing my own bot’s results. Even so, the game records contain far more information than is summarized here. I have many features in mind that I’d like to add to my analysis scripts.

Some bottom line findings: 1. My preparation for specific opponents was largely successful, as far as it went. All prepared openings had at least a fair win rate, and most were among the most successful openings Steamhammer discovered throughout the tournament. 2. Steamhammer’s gas steal skill had value against many opponents, even though my first analysis of the skill suggested that it might not. (I have known for a long time that my first analysis had missed the truth.) It was able to judge pretty accurately whether stealing gas was effective versus a given opponent, so it could exploit it when successful and reduce the cost of trying it when unsuccessful.

Steamhammer managed 3 wins with 2 different openings, but basically nothing worked despite trying the full range from 4 pool to hive rush. Stealing gas did not help, but the situation was desperate so Steamhammer tried it 10% of the time. An interesting number is the enemy scout timing: The scout probe arrived at the zerg base with more consistent timing (arriving in a narrower time window) than against any other opponent. Part of Stardust’s recipe is good scouting.

Steamhammer again desperately tried every tech and timing known to zerg, and this time found a little success with a few of them—a sunken build, a lurker build, and two 2-hatch zergling wave builds. PurpleWave largely stuck with 2 gates. Stealing gas barely delayed Protoss from taking gas, but Steamhammer still tried 10% of the time—if you’re losing almost all games, it doesn’t hurt.

BananaBrain was the first opponent weak enough that Steamhammer was able to find answers and did not feel a need to throw in the kitchen sink. 2 hatch muta builds were the most successful (and the best was one specialized for play against terran), but zergling builds and hydra builds were also OK. I love that the defiler rush won a game (it’s not at all a practical opening, but against a bot...). Unlike Stardust and PurpleWave, unpredictable BananaBrain played a variety of builds—for example, the appearance of an enemy cloaked unit at 3:26 in one game suggests a dark templar rush. The gas steal was successful and delayed protoss from taking gas for nearly an entire minute, so Steamhammer went with it frequently.

Steamhammer upset Dragon. The success of 5 hatcheries before spawning pool means that Dragon was not aggressive early, to its detriment—or that Steamhammer recognized any early aggression as a rush and reacted correctly. 3 hatchery mutalisk builds were also good. Even 3 hatchery lurker was not bad, though on the face it appears unsuited to counter Dragon’s play style. The gas steal was not successful in terms of win rate, but Steamhammer noticed that it delayed Dragon’s gas for a long time so it tried anyway. Fewer than half the attempts to steal gas ended with the extractor successfully made, though. The 37 minute game (here 37:21, officially 37:22) was Steamhammer’s longest game of the tournament, except for a couple games versus EggBot that went to the 60 minute limit and had to be adjudicated on points.

Steamhammer failed to recognize McRave’s build in 2/3 of games, though we know from yeterday’s post exactly what they were, and that they should not have been hard to understand. I put more effort into recognizing rushes and proxies that change the whole course of the game than into regular builds, and the result is that Steamhammer’s understanding and prediction are sometimes spectacularly unhelpful. The 8DroneGas build is actually a 9 pool which makes a second hatchery; it’s called that because it ends up with 8 drones while flooding zerglings (it’s not the Styx build, but related). 5 games had cloaked units; I think that means that McRave researched burrow, because it does use burrow and does not seem to ever make lurkers. Steamhammer was unable to steal gas in 10 attempts, so it gave up trying.

Steamhammer found a variety of overpool openings to be best. Overpool is flexible; those openings play out quite differently. Microwave didn’t play 5 pool often because it didn’t work: Steamhammer’s rush recognition is improved this year, and it reacted well. The burrow openings and 8-8 dawn hydra rush are not appropriate for ZvZ; those choices could be improved. Unlike McRave, Microwave did not use burrow. Stealing gas failed, so Steamhammer gave it up quickly.

DaQin upset Steamhammer badly, and zerg fell back on exploring widely. One ling bust and a few macro builds were able to save some games against DaQin’s consistent forge expand (which Steamhammer often misrecognized as Turtle because it did not actively look for the expansion nexus). The gas steal is measured to increase the overall win rate from 18% to 22%, not bad.

ZZZKBot switches between its famous 4 pool (recognized as Fast rush; see the 100% win rate when the strategy was recognized) and a build with sunkens, two hatcheries, and 6 sudden mutalisks (recognized as Turtle). It seems that part of Steamhammer’s difficulty with ZZZKBot (I consider a 75% win rate to be low in this case) is due to poor scouting: See the rate of Unknown enemy strategies. A bigger part of it may be a problem with the exploration policy. ZvZ_OverpoolTurtle scored 97% and was discovered early, but was played in only 59 games out of 150.

Steamhammer does not use enemy scout timing as a clue to the enemy strategy, and it should. When a scout appears at your base in 39 seconds it must have been sent almost immediately. Either the enemy is rushing, or is terrified that you might. A comparison of past scout timings versus recognized strategies for this opponent could be strong evidence.

ZZZKBot takes gas only in its muta strategy, and then it has sunkens so that a gas steal cannot succeed. Steamhammer figured that out fairly quickly.

Steamhammer recorded only 149 games against UAlbertaBot, because it crashed one game, its only crash in the tournament. Against every other opponent, Steamhammer recorded all 150 games.

There was no need to test the gas steal. If you’re winning nearly all games, spending a drone to vary your play is not likely to gain anything. Another point is that UAlbertaBot often takes gas quite late, so stealing gas early is unlikely to pay off.

WillyT was a tough opponent for Steamhammer. The choice of zerg builds does not seem strong. I have no idea how the guardian rush won 2 games out of 5.

Ecgberht is a tricky opponent. If it knew how to defeat fast lurker builds, it would score a lot higher against Steamhammer.

EggBot is a cannon bot. Its builds were recognized as Proxy for close cannons, Contain for cannons farther away from zerg bases, or Turtle for distant cannons. Steamhammer tried a variety of openings, discarding one as soon as it had lost a single game because there were others that still scored 100%. Steamhammer understood that EggBot never took gas, so gas steal would gain nothing.

This is not all of Steamhammer’s zerg openings! The tournament wasn’t long enough for it to try everything.

I got distracted, but I’m back. These tables aren’t what McRave learned exactly, they are what it recorded in its Info files, more like a summary of how McRave played given what it had learned. History files like these are rich with information and I wanted to extract more from them, but this will do for now.

The “opening” tables give McRave’s strategy as it represents it, build-opener-transition (see last post). The “enemy” tables represent the recognized enemy strategy in the same format. I like that the strategies of both sides are represented the same way, it’s elegant. The hierarchical representation has advantages for reacting to enemy strategies: McRave may be able to react to an aspect of its enemy’s plan, or to an enemy strategy that it only partially recognized.

Here’s a sample line from the file ZvZ Microwave Info.txt, the first game of 150, to give a taste of how much information there is.

Won,Destination,28:43,HatchPool,9Pool,LingRush,PoolLair,9Pool,1HatchMuta,4:07,2:45,3:31,1,Zerg_Larva,92,Zerg_Zergling,105,Zerg_Drone,18,Zerg_Overlord,73,Zerg_Mutalisk,36,Zerg_Scourge,30,Zerg_Larva,114,Zerg_Zergling,102,Zerg_Drone,12,Zerg_Overlord,34,Zerg_Mutalisk,14,Zerg_Scourge,4,Zerg_Cocoon,11,Zerg_Devourer

I think Stardust does not make dark templar, so the one DT game may be a strategy inference miss. It’s interesting that most of McRave’s wins came when Stardust was recognized as opening with gate-cybercore instead of with two gates.

PurpleWave chose to play similarly to Stardust, not quite as successfully.

McRave’s plans were about equally successful, but it chose to go with mutalisks much more often than zerglings. I think that’s strategically correct, because BananaBrain is susceptible to the occasional ling bust but will play safely if zerg repeats it too often. Otherwise, BananaBrain is unpredictable as always. As I understand it, McRave does not try to directly predict the enemy strategy but only reacts to what it scouts, so unpredictability can confuse UCB but nothing worse.

Dragon is the first opponent to push McRave into exploring most of its available strategies. Notice McRave only played PoolHatch,Overpool,2HatchSpeedling once (about a third of the way through the tournament) even though it won, and even though PoolHatch and Overpool was its preferred stem. It shows a strong preference for mutalisk play.

McRave’s play was similar against Microwave and Steamhammer: It chose the same builds in roughly similar proportions, and it found the opponent’s strategies to be highly diverse and difficult to recognize (notice all the Unknown values).

McRave never varied against DaQin. Why is that? It doesn’t seem to be due to elitism (that is, always choosing a plan that has shown itself “elite” aka good enough), because McRave tried two plans versus UAlbertaBot and both had a higher win rate.

Ecgberht is a tricky opponent. I think McRave tried every plan that was enabled—mostly with success, to be sure.