archive by month
Skip to content

fixing a lurker micro bug

Change of plans. I solved a bug that others should know about, so Steamhammer-Tscmoo games will wait until next time.

Steamhammer has a bug that shows up when a lurker starts attacking a building: The lurker becomes unable to switch targets until the building is destroyed. So, for example, if a lurker doesn’t see any other immediate target, it will go after a supply depot. Once it has started in on the depot, marines can wander up behind the lurker in perfect safety, and perhaps scan and kill it. Until the depot is destroyed, all the marines have to do is stay out of the line of fire between the lurker and its target.

The bug is not in targeting. MicroLurkers correctly recognizes that the marines are higher priority targets and issues a command to target them. The bug is that Micro::SmartAttackUnit() does not act on the command. The code is inherited from UAlbertaBot, so probably many UAlbertaBot and Steamhammer forks have the same problem. Here is the inherited code:

// if we have issued a command to this unit already this frame, ignore this one
if (attacker->getLastCommandFrame() >= BWAPI::Broodwar->getFrameCount() || attacker->isAttackFrame())
{
    return;
}

I think the isAttackFrame() test may be a good idea if you are controlling, say, a dragoon. Retargeting a dragoon while it is in the middle of its animation will at best cause extra delay before the dragoon can fire again. But when I looked closely, it turned out that for a lurker attacking a fixed target every frame is an attack frame. Steamhammer drops the command because the lurker is constantly attacking.

Instead, it should accept the slight delay of retargeting lurkers. I rewrote it like this:

// Do nothing if we've already issued a command this frame, or the unit is busy attacking.
// NOTE A lurker attacking a fixed target is ALWAYS on an attack frame.
if (attacker->getLastCommandFrame() >= BWAPI::Broodwar->getFrameCount() ||
    (attacker->isAttackFrame() && attacker->getType() != BWAPI::UnitTypes::Zerg_Lurker))
{
    return;
}

Lurkers behave better, and other unit types are not affected. I see improved lurker micro in test games.

In Starcraft, every unit type behaves differently. UAlbertaBot, and therefore Steamhammer, tries to treat different unit types the same, which is sure to be causing other weaknesses. At some point I will analyze micro thoroughly and make sure that every unit type realizes its potential. For now, I’m happy that lurkers are a little smarter.

Steamhammer doing well

Apparently I fixed enough bugs and added the right features to Steamhammer, because the bot is doing well on SSCAIT. It’s ranked #5 right now, high up in the mass of bots in the 2200 range. This Steamhammer version has negative scores against Krasi0 and Iron, and against several protoss bots—and that’s all. It is equal or winning in head-to-head score against every other opponent it has faced on SSCAIT, including all other zergs.

I don’t expect to do as well in AIIDE, because there are unfamiliar opponents that I can’t prepare for. But I hope to finish fairly high.

The upcoming 1.4 version has changes to improve results against protoss. The following version 1.5 plans mutalisk skills beyond other bots, which will be especially dangerous to terran. It should be able to notch some wins against Krasi0 and Iron. But it will take time, and everybody else is improving too....

Next: Games versus Tscmoo.

Steamhammer is getting better at scouting

I have improved Steamhammer’s early game scouting and added midgame scouting. It’s far short of human scouting skill, but much improved. The first overlord and the drone scout cooperate to locate the enemy base as quickly as possible, and the overlord moves there to watch as long as it can. I also taught it to notice the enemy’s overlord and try to infer the enemy’s base location (ZZZKBot is the only other bot that I know has that skill). The effect is that Steamhammer finds the enemy sooner and collects more information. The midgame scouting ensures that enemy expansions are found, and in my testing often destroyed quickly.

I already find it hard to watch the release Steamhammer play on SSCAIT, since the new one is so much less clumsy at finding things. I think that Steamhammer 1.4 will be stronger than the current Steamhammer even with opponent modeling turned off, because of the improved scouting, slightly more flexible tactics to take advantage of the improved scouting, and a couple of key bug fixes.

It still doesn’t overlord scout versus terran, though. It has no concept that marines are dangerous or that cliffs are useful.

Overlord scouting, by the way, is a deep topic. Pros often send their first overlord straight to the closest possible enemy natural base. But they sometimes move the overlord on a complex path, trying to catch proxies, or avoid the enemy scout, or spot activity without being seen, or just to stay flexible so the overlord can move to its choice of locations depending on the situation. An overlord that always follows the most efficient scouting path is predictable, and the enemy may be able to save time by scouting for your overlord instead of directly for your base.

Overkill’s CIG 2017 replays

Well, I looked at Overkill’s replays from CIG 2017 as I said I would, but I was disappointed. Not only was the CIG 2017 Overkill carried over from the previous year (which I forgot), but the CIG 2016 was also carried over from the year before (which I also forgot). Overkill from CIG 2017 is the same as Overkill from AIIDE 2015, from before the big learning feature was added. It plays the same as the Overkill version on SSCAIT.

Nothing to see here. :-(

Of the tournaments in yesterday’s table, AIIDE 2016 has the least-old version of Overkill. Last year’s version is the first version with unit choice learning, not only opening learning. The other tournaments all played the AIIDE 2015 version which has only simple opening learning like many bots. The AIIDE 2016 result is not far out of line with the other results, so we can guess that the new learning feature worked adequately but not great.

Overkill AIIDE 2017 updates

PurpleWaveJadian pointed out in a comment that Sijia Xu had updated Overkill’s repository, from last year’s version to bring it up to date with the AIIDE 2017 version of Overkill. There are extensive changes. The bot looks substantially more capable.

Here’s what caught my eye. This is based on a quick read through a long list of diffs, so I may have missed or misunderstood a lot.

• New machine learning library with a couple different learning methods.

• Support for lurkers, scourge, guardians, devourers, ultralisks—all the regular zerg combat units. Only queens and defilers are omitted, the spell units. Last year it supported only zerglings, hydralisks, and mutalisks.

• Major changes in tactical calculations. Well, there would have to be to support new units.

• Changes to building construction. I’m not sure what the idea is here.

• It detects production deadlocks, to avoid freezes. The method is written to be simple and general-purpose rather than extensive and tweakable like Steamhammer’s, so it may be worth a look. See ProductionManager::checkProductionDeadLock().

• Scouting seems to be completely rewritten. Overlords scout around using an influence map. It looks like 1 zergling can be singled out to be a scout zergling.

• Overkill 2016 learned a model of when to make zerglings, hydralisks, or mutalisks. The actions in this Overkill version include the combat units except for guardians and devourers, sunken colonies, tech buildings to construct, upgrades and tech to research, expanding, attacking versus harrassing with mutalisks, and doing nothing. So it not only has a wider scope for strategy with more unit types, it looks as though it learns more of its strategy. I see declarations for 4 neural networks, which apparently cooperate to do different parts of the job.

• It chooses from among the same 3 openings as before: 9 pool, 10 hatch, 12 hatch. The details of the build orders have been revised.

How does it play? Well, I only looked at the source, and not too closely. Overkill first got a learning model in the CIG 2016 tournament. Here are its historical tournament results:

tournament#%
CIG 2015381%
AIIDE 2015381%
CIG 2016 qualifier471%
CIG 2016 final551%
AIIDE 2016762%
CIG 2017760%

The CIG 2016 qualifier is a better comparison to the other tournaments than the final, since it includes all entrants. The CIG 2017 entry is probably not much different from the AIIDE 2017 one. [Update: Not so. CIG says that the CIG 2017 was the same as the CIG 2016 version. See comments.] It seems that Overkill’s win rate has been falling as it learns more on its own. Is that a sign of how hard it is to do strategy learning? I would like to see it play before I draw conclusions!

Next: I’ll look at Overkill’s CIG 2017 replays and see what I can see.

an awful game Steamhammer-Microwave

Steamhammer and Microwave are both rated close to 2200 at the moment, putting them in the top ranks on SSCAIT. There is a big pileup at that skill level; only Krasi0 and Iron are significantly stronger. And yet a few days ago, Steamhammer and Microwave struggled against each other in a game with so much nonsense play that I was raging at them the whole time, sitting on the sofa with a laptop and shouting “Why are you doing that? DON’T DO THAT! Aargh!” Facepalm. There is so far to go.

It had better be true that you learn the most from mistakes, because there are a lot of painful mistakes here that are hard to look at.

The game is Steamhammer vs Microwave on Icarus. Steamhammer opened overlord, then hatchery, spawning pool, and gas all on 9/17. Microwave opened spawning pool on 9 and gas. Steamhammer’s worker defense is poor and the 9 pool will win if Microwave sends its zerglings immediately in the right direction. Microwave scouted with a drone and found Steamhammer’s base on the second try, which was too late. But Steamhammer's zerglings were distracted by the scouting drone, and Steamhammer did well to survive with 2 drones lost. Microwave could have won on the spot if it had scouted with an overlord, or if it had inferred Steamhammer’s base location when it saw Steamhammer’s overlord.

pulling drones while zerglings are distracted

Steamhammer continued to chase the scouting drone. Urgh. The misbehavior is still there because I haven’t found an easy way to prevent it that doesn’t cause other bad behaviors—if it ignores the scout, that’s bad too. My plan to fix it requires time and care, so I’m putting it off for now. Anyway, moving the zerglings out of position offered opportunities to Microwave to stab into the mineral line, but Microwave played overcautiously and did not. Aggressive play would be to rush all zerglings in to wreak havoc, then try to escape when Steamhammer reacts, so that you don’t lose to the counter. Safe play would be to detach 2 to 4 zerglings to try to catch a drone, while keeping the others back for defense; if the raiders are surrounded and lost it is no terrible setback, and if they kill a drone that is an edge.

the zerglings are even more distracted

Steamhammer finally killed the scout and started to realize the advantages of its build. With 2 hatcheries from the start, it was far ahead in zerglings, and Microwave was forced into static defense. Microwave added 2 sunkens in its main... and expanded, so then it added 2 more sunkens at its natural. 4 sunkens are a fatal expense, and Steamhammer was miles ahead. In the picture, Steamhammer has so many more zerglings that it should have attacked immediately.

four sunkens in Microwave’s bases

There are a lot more mistakes in the rest of the game, but the story is getting tiresome. Steamhammer refused chance after chance to widen its lead or win outright. It held back when it should have attacked. It built an ultralisk cavern and researched the expensive ultra upgrades, then did not build a single ultralisk until they lost their value. It made guardians and devourers and used them wastefully (partly because it hit a bug which assigned units to the wrong squads). It became so lax in applying pressure that Microwave caught up in economy. You can watch the end of the game yourself, if you can stomach it.

units behaving unhelpfully

It was much smoother than past Steamhammer games that I have blogged as disastrously bad. The bot is slowly getting more solid.

upgrade prices in Steamhammer

Today I solved a major cause of production freezes. Steamhammer did not understand that repeated upgrades increase in cost, and I didn’t register that BWAPI requires you to specify the level of upgrade to get the prices of upgrades above level 1. So if Steamhammer wanted to upgrade zerg carapace +2 (gas cost 225) and had 200 gas, then if gas collection was turned off it would never turn gas collection on. It concluded “the upgrade costs 150 gas so I have enough.” The upgrade sat in the queue waiting for enough gas to appear, and the gas never appeared.

I think it’s the most common permanent production freeze remaining in the live version 1.3.3. Most long-lasting production freezes happen when gas is turned off and stays off.

The fix is to add code to MacroAct::mineralPrice() and MacroAct::gasPrice(). I verified that all upgrade prices are fetched via MacroAct; prices that it fetches directly are always for units or buildings, which are constant price. I added this code to MacroAct::gasPrice() (and similarly for the mineral price):

	if (isUpgrade())
	{
		if (_upgradeType.maxRepeats() > 1 && BWAPI::Broodwar->self()->getUpgradeLevel(_upgradeType) > 0)
		{
			return _upgradeType.gasPrice(1 + BWAPI::Broodwar->self()->getUpgradeLevel(_upgradeType));
		}
		return _upgradeType.gasPrice();
	}

Finally! I didn’t trace all the consequences, but the fix probably corrects other bugs too.

breaking scouting assumptions

Working on scouting reminded me: There is a trick that terran can play to mightily confuse the opponent’s early scout. At the start of the game, make SCVs as usual, then when it nears time for the opponent’s scout to approach, lift off the command center and move it out of sight. Of course you have to build any supply depots or other buildings out of sight too, or away from your base (you could proxy 8 rax, or build your first depot in an empty main base). It can only conceivably be a good idea if there is another base close by to land the command center at, so the trick should be saved for maps like Electric Circuit which have an inside “backdoor” base near the main.

The opponent will scout every main base and see you at none of them.

This trick has been used in one pro game to my knowledge, Boxer versus Yellow from 2003, where the CC move was meant to keep the proxy barracks unknown; a base without barracks would have given it away. If it were a good trick it would have been used since then. It’s objectively bad, and human players who see that all bases are empty are able to figure out what happened. But in bot play, you might fool your opponent into doing something nonsensical or even cause it to crash.

When UAlbertaBot’s scout is looking for the enemy base and finds an enemy building in a starting base region, is assumes that that marks the location of the enemy base. Steamhammer inherited the heuristic and still uses it. So a low-cost way to fool these bots is to build a supply depot or pylon next to the ramp of an empty main base before it is scouted. Zerg could build an extractor at the empty base, if the gas is toward the ramp. 50% of the time, the scouting worker will find the decoy building first and will not scout other bases. Nobody has tried this trick yet, as far as I have seen.

OpenBW’s freeze bug only affects protoss

OpenBW has a freeze bug that occasionally stops replays when units are selected, so that you have to reload and try again. The bug only affects protoss units. My experience is that you can click on and select terran and zerg units all you like with no freezes.

I post this because apparently some people haven’t realized. If you’re overcautious because of the bug, you aren’t getting everything from OpenBW. Though I wish they’d fix the bug; it has been there for months.

Steamhammer 1.4 status

I’m working on version 1.4 now, and the major feature in version 1.4 will be opponent modeling. But after the AIIDE rush I felt tired of opponent modeling, and discouraged by the bugs, so I’m improving scouting first. Good scouting is important for opponent modeling to work well, so I’m happy to do scouting in the same version. No more rush; I will take my time and do it right.

I added a Recon squad that scouts around with ground units in the middle game, as soon as there are enough ground units to spare a few for scouting. It doesn’t search efficiently, but it does catch enemy expansions sooner or later, and sends eyes around the map to places Steamhammer used to ignore. I also made a small tactical change so that the attack squads are less maniacally focused on the enemy main base and more willing to attack other bases. Besides improving opponent modeling, I think it should help directly against most opponents that try to mass expand or to hide expansions.

I also refactored code so that I can work on overlord scouting. The first overlord will coordinate its movement with the scouting drone. Currently, if Steamhammer sends out both its first overlord and a scouting drone, they both head for the same base. It’s disgusting, but it has been that way since December when I first hacked in crude overlord scouting. Overlord scouting in 1.4 won’t be a polished gem, but it shouldn’t be a dirt clod any more.

I’m also making the usual random improvements here and there to things I come across. I finally realized why Steamhammer keeps making guardians against XIMP by Tomas Vajda: It doesn’t recognize that corsairs and carriers counter guardians. Oops. I left out that bit of knowledge and only now caught the mistake.

biggest weaknesses

The worst news is Steamhammer is doing poorly against the top protoss bots lately, for two reasons. First, some are using multiple strategies, and the current Steamhammer’s random openings cannot counter all of them. Opponent modeling should help. Second, protoss is often playing forge expand or other big macro openings, which Steamhammer only partially understands. Opponent modeling will again help, because then Steamhammer should usually be able to find good strategies in response, instead of only occasionally. But opponent modeling won’t be enough (unless it sticks with timing attacks like hydra bust), because Steamhammer’s tactics tend to break down in a big macro game. To have the right units at the right time is no help if you throw them away. Fixing tactical play, the way I intend to do it, will be another major feature.

In other words, I expect to have to finish 2 more major features, opponent modeling and tactical analysis, before Steamhammer can score consistently against top protoss bots. It won’t happen soon. Opponent modeling is underway, but tactical analysis likely won’t be until next year.

I will be able to make progress on small features. The 3 biggest weaknesses that can be corrected with minor features are:

1. Better worker micro. Drills would help in a lot of defensive situations. Even without drills, better choices about engaging and running away would reduce losses.

2. Run workers away from enemy attacks. It’s a basic skill that top bots have to some degree and Steamhammer doesn’t. Steamhammer not only loses workers in a base under enemy attack, it transfers more to the base to try to keep mining gas. Keeping workers alive would be a big boost to resilience. I will likely stop the transfers as one fix, and run workers away as a later fix.

3. Pathing in the presence of map blocks. Units too often get stuck trying to take a path that is blocked. The work to fix pathing is started, and I see it as related to dropping BWTA. I’ll continue step by step.

I’m not working on any of these 3 at the moment. One goal at a time. In 1.4 I will probably take at least one more step toward removing BWTA.

CIG 2017 race balance

I thought of another CIG 2017 analysis I wanted to do. Here is how bots of each race scored in the tournament.

racescore
terran43%
protoss55%
zerg48%
random68%

The table is not interesting in itself. It reflects the participants more than anything else. Random bots scored well because the 2 random players both finished high; terran scored poorly because 5 of the 7 terrans were in the bottom half. But the numbers make it easier to interpret the next table, which is how each bot scored against opponents of each race.

botraceoverallvTvPvZvR
ZZZKBotzerg75.43%80%75%74%67%
tscmoorandom73.50%81%74%67%54%
PurpleWaveprotoss66.51%69%74%69%36%
LetaBotterran62.75%63%55%76%40%
UAlbertaBotrandom61.67%69%51%64%46%
MegaBotprotoss61.06%75%45%59%52%
Overkillzerg59.65%74%45%63%37%
CasiaBotzerg58.32%56%60%65%47%
Ziabotzerg58.49%70%51%60%32%
Ironterran58.11%65%54%57%50%
AIURprotoss56.73%71%53%50%35%
McRaveprotoss47.20%53%42%52%23%
Tyrprotoss45.32%48%36%49%43%
SRbotOneterran45.24%54%35%50%29%
TerranUABterran38.58%55%27%42%10%
Bonjwaterran33.04%42%30%35%10%
Bigeyesterran30.90%31%32%33%24%
OpprimoBotterran31.90%41%24%37%10%
Slingzerg26.07%28%26%28%15%
Salsazerg9.52%9%10%10%7%

The “versus random” column means versus Tscmoo and UAlbertaBot, so the other columns tell us more about strategy. We can see that Tscmoo was a little weak versus zerg, but LetaBot was strong. MegaBot and Overkill struggled versus protoss, while CasiaBot was strong against protoss and had some trouble against terran.

I don’t think it’s necessary to normalize the table entries to draw conclusions from them. But if people are interested, I could produce a version of the table with the score of each race (in the first table above) effectively subtracted out so that we can see how each bot did against each race relative to how well it “should” have.

CIG Elo ratings

Elo as calculated by Remi Coulom’s bayeselo program. The # column gives the official ranking, so you can see how it differs from the rank by Elo. The bayeselo ranking should be slightly more accurate because it takes into account all the information in the tournament results, but unfortunately there are missing games so the Elo is computed from slightly less data than the official results. The “better?” column tells how likely each bot is to be superior to the one below it.

#botscoreElobetter?
1ZZZKBot75%174998.8%
2tscmoo74%1731100%
3PurpleWave67%166099.9%
4LetaBot63%162697.9%
5UAlbertaBot62%161185.1%
6MegaBot61%160496.7%
7Overkill60%159189.3%
8CasiaBot59%158255.2%
9Ziabot59%158158.8%
10Iron58%157997.0%
11AIUR57%1566100%
12McRave47%147697.6%
13Tyr45%146279.9%
14SRbotOne45%145699.9%
15TerranUAB38%139799.9%
16Bonjwa33%134794.9%
18OpprimoBot32%133569.0%
17Bigeyes32%133199.9%
19Sling26%1275100%
20Salsa9%1041-

Looking at the better? column, we see that the top 3 are well separated; the places are virtually sure to be accurate. ZZZKBot and Tscmoo are close, but bayeselo thinks they are separated enough. Farther down, CasiaBot, Ziabot, and Iron are statistically hard to distinguish; there is not strong evidence that they finished in the correct order. Also OpprimoBot and Bigeyes are not well separated—as you might guess since their results are reversed from the official results.

Is this all the analysis we want of CIG 2017? I also have a script for the map balance, to check whether any race is favored. But it tells more about who competed than about the maps or bot skills.

CIG 2017 bots x maps

How well did each bot do on each map? This table is nice and compact and easy to read conclusions out of. Hitchhiker is a 2 player map and a complex one. Alchemist and Tau Cross are 3 players. Andromeda and Python are 4 players.

overallHitchhAlchemTauCroAndromPython
ZZZKBot75.43%81%76%73%73%75%
tscmoo73.50%78%69%73%74%73%
PurpleWave66.51%82%70%56%57%67%
LetaBot62.75%62%58%63%68%62%
UAlbertaBot61.67%63%65%60%61%59%
MegaBot61.06%55%66%64%60%61%
Overkill59.65%66%59%59%57%57%
CasiaBot58.32%62%61%59%55%55%
Ziabot58.49%66%59%55%57%56%
Iron58.11%5%60%77%75%74%
AIUR56.73%61%56%57%55%55%
McRave47.20%9%49%62%62%54%
Tyr45.32%52%42%44%46%43%
SRbotOne45.24%54%41%43%44%44%
TerranUAB38.58%47%40%33%39%33%
Bonjwa33.04%44%33%32%27%30%
Bigeyes30.90%35%30%29%28%33%
OpprimoBot31.90%37%33%30%30%29%
Sling26.07%23%27%24%25%32%
Salsa9.52%19%7%6%7%8%

If one bot loses on a given map, then another bot has to win. The BWEM blowouts on Hitchhiker lift the other numbers in that column about equally. It’s striking that the top 3 finishers all liked Hitchhiker above the other maps. Maybe it’s a sign that map smarts make a difference? Well, ZZZKBot presumably liked Hitchhiker because the map has a short rush distance, and PurpleWave may have gotten a similar effect if it used its probe rush on the map. But UAlbertaBot got no boost from the map, although it plays rushes.

I noted earlier that PurpleWave struggled surprisingly on Tau Cross and Andromeda, and that MegaBot did not like Hitchhiker. It’s also notable that LetaBot did not like Alchemist and McRave did not like Python.

Next: The bayeselo ratings.

CIG 2017 without the problem maps

What if CIG 2017 had included only maps that Iron plays properly on? Then Iron would have won, at least according to its results on the remaining maps. Leaving out Hitchhiker and Alchemist leaves only 3 other maps, Andromeda, Python, and Tau Cross. A tournament with only 3 maps is hardly satisfying, but it’s the best estimate of how Iron might have done if it had been prepared for the map pool.

overallIronZZZKtscmLetaMegaPurpUAlbMcRaOverCasiZiabAIURTyrSRboTerrBigeOpprBonjSlinSals
Iron75.37%55%57%28%80%63%80%51%85%84%77%84%55%85%91%95%95%87%91%91%
ZZZKBot73.46%45%45%41%77%47%79%84%93%29%64%91%81%81%91%84%95%88%89%91%
tscmoo73.33%43%55%83%69%88%55%69%64%53%71%77%68%69%89%77%96%89%83%95%
LetaBot64.49%72%59%17%60%60%71%47%81%85%73%63%57%43%69%77%40%72%84%95%
MegaBot61.40%20%23%31%40%35%75%52%51%40%59%36%57%75%89%91%97%100%97%100%
PurpleWave60.07%37%53%12%40%65%52%25%89%44%35%76%84%71%87%92%63%60%64%92%
UAlbertaBot60.07%20%21%45%29%25%48%80%57%63%69%55%44%71%89%75%83%89%84%93%
McRave59.44%49%16%31%53%48%75%20%39%49%72%35%41%88%93%55%89%84%97%95%
Overkill57.89%15%7%36%19%49%11%43%61%45%85%61%43%84%95%93%93%89%80%91%
CasiaBot56.28%16%71%47%15%60%56%37%51%55%35%60%73%32%51%95%68%75%85%89%
Ziabot56.04%23%36%29%27%41%65%31%28%15%65%56%57%72%81%79%91%91%83%95%
AIUR55.72%16%9%23%37%64%24%45%65%39%40%44%71%79%77%69%89%93%80%93%
Tyr44.28%45%19%32%43%43%16%56%59%57%27%43%29%28%45%36%65%57%48%93%
SRbotOne43.65%15%19%31%57%25%29%29%12%16%68%28%21%72%39%100%43%59%72%95%
TerranUAB35.09%9%9%11%31%11%13%11%7%5%49%19%23%55%61%57%73%77%55%91%
Bigeyes29.96%5%16%23%23%9%8%25%45%7%5%21%31%64%0%43%68%40%48%88%
OpprimoBot29.68%5%5%4%60%3%37%17%11%7%32%9%11%35%57%27%32%40%80%92%
Bonjwa29.47%13%12%11%28%0%40%11%16%11%25%9%7%43%41%23%60%60%57%93%
Sling27.09%9%11%17%16%3%36%16%3%20%15%17%20%52%28%45%52%20%43%92%
Salsa7.23%9%9%5%5%0%8%7%5%9%11%5%7%7%5%9%12%8%7%8%

Iron narrowly edges out ZZZKBot and Tscmoo. PurpleWave falls back and MegaBot, which also did poorly on Hitchhiker, jumps upward. McRave moves from the lower half to the upper half but does not break into the top ranks.

Here is another version of the same chart, with bots in the official finishing order. It makes some comparisons easier.

overallZZZKtscmPurpLetaUAlbMegaOverCasiZiabIronAIURMcRaTyrSRboTerrBonjBigeOpprSlinSals
ZZZKBot73.46%45%47%41%79%77%93%29%64%45%91%84%81%81%91%88%84%95%89%91%
tscmoo73.33%55%88%83%55%69%64%53%71%43%77%69%68%69%89%89%77%96%83%95%
PurpleWave60.07%53%12%40%52%65%89%44%35%37%76%25%84%71%87%60%92%63%64%92%
LetaBot64.49%59%17%60%71%60%81%85%73%72%63%47%57%43%69%72%77%40%84%95%
UAlbertaBot60.07%21%45%48%29%25%57%63%69%20%55%80%44%71%89%89%75%83%84%93%
MegaBot61.40%23%31%35%40%75%51%40%59%20%36%52%57%75%89%100%91%97%97%100%
Overkill57.89%7%36%11%19%43%49%45%85%15%61%61%43%84%95%89%93%93%80%91%
CasiaBot56.28%71%47%56%15%37%60%55%35%16%60%51%73%32%51%75%95%68%85%89%
Ziabot56.04%36%29%65%27%31%41%15%65%23%56%28%57%72%81%91%79%91%83%95%
Iron75.37%55%57%63%28%80%80%85%84%77%84%51%55%85%91%87%95%95%91%91%
AIUR55.72%9%23%24%37%45%64%39%40%44%16%65%71%79%77%93%69%89%80%93%
McRave59.44%16%31%75%53%20%48%39%49%72%49%35%41%88%93%84%55%89%97%95%
Tyr44.28%19%32%16%43%56%43%57%27%43%45%29%59%28%45%57%36%65%48%93%
SRbotOne43.65%19%31%29%57%29%25%16%68%28%15%21%12%72%39%59%100%43%72%95%
TerranUAB35.09%9%11%13%31%11%11%5%49%19%9%23%7%55%61%77%57%73%55%91%
Bonjwa29.47%12%11%40%28%11%0%11%25%9%13%7%16%43%41%23%60%60%57%93%
Bigeyes29.96%16%23%8%23%25%9%7%5%21%5%31%45%64%0%43%40%68%48%88%
OpprimoBot29.68%5%4%37%60%17%3%7%32%9%5%11%11%35%57%27%40%32%80%92%
Sling27.09%11%17%36%16%16%3%20%15%17%9%20%3%52%28%45%43%52%20%92%
Salsa7.23%9%5%8%5%7%0%9%11%5%9%7%5%7%5%9%7%12%8%8%

Next: The chart of bot x map.

CIG 2017 crosstables per map

CIG 2017 was played on only 5 maps. In my view, that’s not enough maps for fairness, but the advantage is that each map is played enough times that the crosstables have ample data. Each cell in these tables represents 25 games (minus the missing games, of course).

(2)Hitchhiker

overallZZZKtscmPurpLetaUAlbMegaOverCasiZiabIronAIURMcRaTyrSRboTerrBonjBigeOpprSlinSals
ZZZKBot80.80%76%0%96%88%68%92%40%68%92%100%96%92%92%84%92%84%92%96%88%
tscmoo78.27%24%96%80%56%76%64%48%52%100%71%96%76%92%96%92%96%88%92%92%
PurpleWave82.07%100%4%52%80%96%88%84%60%100%96%100%96%84%88%68%80%92%96%96%
LetaBot62.37%4%20%48%60%32%88%88%72%96%33%92%52%12%76%84%96%44%92%96%
UAlbertaBot63.08%12%44%20%40%28%56%40%68%92%62%100%52%68%96%92%60%100%80%88%
MegaBot54.97%32%24%4%68%72%72%28%72%100%42%100%32%88%88%12%17%96%16%80%
Overkill66.03%8%36%12%12%44%28%68%80%96%96%92%36%80%92%96%92%96%92%100%
CasiaBot61.95%60%52%16%12%60%72%32%32%96%67%92%64%16%44%92%96%84%92%100%
Ziabot65.61%32%48%40%28%32%28%20%68%96%71%96%68%76%76%92%92%96%88%100%
Iron4.65%8%0%0%4%8%0%4%4%4%0%32%4%0%4%4%0%4%4%4%
AIUR60.88%0%29%4%67%38%58%4%33%29%100%92%75%83%88%92%91%96%88%92%
McRave8.88%4%4%0%8%0%0%8%8%4%68%8%12%0%12%20%0%8%0%4%
Tyr51.90%8%24%4%48%48%68%64%36%32%96%25%88%16%44%64%72%64%88%96%
SRbotOne54.22%8%8%16%88%32%12%20%84%24%100%17%100%84%28%48%92%76%100%92%
TerranUAB47.47%16%4%12%24%4%12%8%56%24%96%12%88%56%72%76%88%56%96%100%
Bonjwa43.76%8%8%32%16%8%88%4%8%8%96%8%80%36%52%24%83%76%96%100%
Bigeyes34.76%16%4%20%4%40%83%8%4%8%100%9%100%28%8%12%17%17%88%96%
OpprimoBot37.21%8%12%8%56%0%4%4%16%4%96%4%92%36%24%44%24%83%92%100%
Sling22.57%4%8%4%8%20%84%8%8%12%96%12%100%12%0%4%4%12%8%24%
Salsa18.57%12%8%4%4%12%20%0%0%0%96%8%96%4%8%0%0%4%0%76%

(3)Alchemist

overallZZZKtscmPurpLetaUAlbMegaOverCasiZiabIronAIURMcRaTyrSRboTerrBonjBigeOpprSlinSals
ZZZKBot76.00%48%44%44%88%84%80%28%76%100%92%68%84%72%84%88%92%96%84%92%
tscmoo69.26%52%92%84%48%52%68%24%64%64%40%68%64%76%96%84%68%92%84%96%
PurpleWave70.32%56%8%40%72%76%88%56%40%64%88%80%96%72%84%56%96%92%76%96%
LetaBot57.89%56%16%60%36%32%48%68%56%64%32%96%48%72%64%60%64%40%92%96%
UAlbertaBot65.05%12%52%28%64%44%80%72%72%56%76%56%44%56%76%92%80%88%96%92%
MegaBot66.11%16%48%24%68%56%52%36%56%60%72%44%44%92%96%100%96%96%100%100%
Overkill58.53%20%32%12%52%20%48%72%44%76%44%32%44%88%92%92%92%80%84%88%
CasiaBot60.84%72%76%44%32%28%64%28%32%88%64%52%68%32%40%92%84%88%72%100%
Ziabot58.74%24%36%60%44%28%44%56%68%72%52%16%60%72%76%80%84%80%68%96%
Iron59.58%0%36%36%36%44%40%24%12%28%92%96%56%88%96%84%92%92%88%92%
AIUR55.79%8%60%12%68%24%28%56%36%48%8%32%84%88%92%88%60%88%88%92%
McRave48.63%32%32%20%4%44%56%68%48%84%4%68%84%60%12%24%96%4%92%92%
Tyr41.89%16%36%4%52%56%56%56%32%40%44%16%16%20%44%52%64%56%56%80%
SRbotOne41.05%28%24%28%28%44%8%12%68%28%12%12%40%80%32%80%92%12%60%92%
TerranUAB40.21%16%4%16%36%24%4%8%60%24%4%8%88%56%68%76%56%60%64%92%
Bonjwa33.05%12%16%44%40%8%0%8%8%20%16%12%76%48%20%24%44%80%56%96%
Bigeyes29.89%8%32%4%36%20%4%8%16%16%8%40%4%36%8%44%56%76%56%96%
OpprimoBot33.26%4%8%8%60%12%4%20%12%20%8%12%96%44%88%40%20%24%64%88%
Sling26.53%16%16%24%8%4%0%16%28%32%12%12%8%44%40%36%44%44%36%84%
Salsa7.37%8%4%4%4%8%0%12%0%4%8%8%8%20%8%8%4%4%12%16%

(3)Tau Cross

overallZZZKtscmPurpLetaUAlbMegaOverCasiZiabIronAIURMcRaTyrSRboTerrBonjBigeOpprSlinSals
ZZZKBot72.84%28%60%28%84%80%96%12%72%28%84%92%92%92%92%92%88%92%88%84%
tscmoo72.84%72%88%88%64%64%52%64%68%40%60%60%60%76%80%92%80%100%88%88%
PurpleWave56.21%40%12%40%48%56%88%36%36%36%88%12%92%56%80%52%100%56%48%92%
LetaBot63.16%72%12%60%72%56%72%88%76%68%56%28%52%56%72%64%68%44%88%96%
UAlbertaBot59.79%16%36%52%28%32%52%68%84%8%56%76%40%64%96%88%72%80%88%100%
MegaBot63.58%20%36%44%44%68%64%40%60%20%48%48%60%72%96%100%88%100%100%100%
Overkill59.37%4%48%12%28%48%36%36%92%12%48%84%44%80%100%88%92%96%84%96%
CasiaBot58.95%88%36%64%12%32%60%64%36%4%64%52%84%24%76%76%92%64%92%100%
Ziabot54.95%28%32%64%24%16%40%8%64%8%52%24%52%84%80%96%80%100%92%100%
Iron77.47%72%60%64%32%92%80%88%96%92%84%28%48%84%92%88%96%96%92%88%
AIUR57.26%16%40%12%44%44%52%52%36%48%16%76%72%80%80%92%60%88%88%92%
McRave62.32%8%40%88%72%24%52%16%48%76%72%24%48%92%96%88%56%88%100%96%
Tyr44.42%8%40%8%48%60%40%56%16%48%52%28%52%20%56%56%48%60%52%96%
SRbotOne42.95%8%24%44%44%36%28%20%76%16%16%20%8%80%32%64%100%48%56%96%
TerranUAB33.05%8%20%20%28%4%4%0%24%20%8%20%4%44%68%72%52%84%56%92%
Bonjwa31.79%8%8%48%36%12%0%12%24%4%12%8%12%44%36%28%64%76%72%100%
Bigeyes29.26%12%20%0%32%28%12%8%8%20%4%40%44%52%0%48%36%44%60%88%
OpprimoBot29.68%8%0%44%56%20%0%4%36%0%4%12%12%40%52%16%24%56%84%96%
Sling24.21%12%12%52%12%12%0%16%8%8%8%12%0%48%44%44%28%40%16%88%
Salsa5.89%16%12%8%4%0%0%4%0%0%12%8%4%4%4%8%0%12%4%12%

(4)Andromeda

overallZZZKtscmPurpLetaUAlbMegaOverCasiZiabIronAIURMcRaTyrSRboTerrBonjBigeOpprSlinSals
ZZZKBot72.84%44%64%20%84%72%92%36%56%60%92%68%76%76%92%84%84%96%92%96%
tscmoo74.32%56%84%84%60%72%64%44%76%44%88%68%68%68%92%92%76%92%88%96%
PurpleWave56.63%36%16%32%40%72%88%40%28%12%60%20%72%80%88%68%92%60%84%88%
LetaBot68.42%80%16%68%84%72%92%84%68%76%64%56%48%44%68%84%80%32%84%100%
UAlbertaBot61.47%16%40%60%16%20%60%72%76%28%48%88%56%80%84%88%76%92%76%92%
MegaBot60.00%28%28%28%28%80%48%36%48%20%36%52%60%72%80%100%100%96%100%100%
Overkill57.47%8%36%12%8%40%52%48%88%24%72%48%40%84%92%88%96%88%84%84%
CasiaBot54.53%64%56%60%16%28%64%52%40%20%36%36%64%44%36%80%96%84%80%80%
Ziabot57.26%44%24%72%32%24%52%12%60%36%68%32%52%64%88%92%80%88%76%92%
Iron74.74%40%56%88%24%72%80%76%80%64%76%56%60%88%88%88%96%92%96%100%
AIUR54.95%8%12%40%36%52%64%28%64%32%24%48%64%76%72%96%80%84%68%96%
McRave62.11%32%32%80%44%12%48%52%64%68%44%52%40%88%96%80%68%92%96%92%
Tyr45.68%24%32%28%52%44%40%60%36%48%40%36%60%28%28%48%44%72%52%96%
SRbotOne43.79%24%32%20%56%20%28%16%56%36%12%24%12%72%40%64%100%44%84%92%
TerranUAB38.74%8%8%12%32%16%20%8%64%12%12%28%4%72%60%76%68%72%72%92%
Bonjwa26.95%16%8%32%16%12%0%12%20%8%12%4%20%52%36%24%52%44%56%88%
Bigeyes27.79%16%24%8%20%24%0%4%4%20%4%20%32%56%0%32%48%76%56%84%
OpprimoBot30.32%4%8%40%68%8%4%12%16%12%8%16%8%28%56%28%56%24%88%92%
Sling24.63%8%12%16%16%24%0%16%20%24%4%32%4%48%16%28%44%44%12%100%
Salsa7.37%4%4%12%0%8%0%16%20%8%0%4%8%4%8%8%12%16%8%0%

(4)Python

overallZZZKtscmPurpLetaUAlbMegaOverCasiZiabIronAIURMcRaTyrSRboTerrBonjBigeOpprSlinSals
ZZZKBot74.68%64%16%76%68%80%92%40%62%48%96%92%76%76%88%88%80%96%88%92%
tscmoo72.84%36%92%76%40%72%76%52%68%44%84%80%76%64%96%84%76%96%72%100%
PurpleWave67.37%84%8%48%68%68%92%56%40%64%80%44%88%76%92%60%84%72%60%96%
LetaBot61.89%24%24%52%56%52%80%84%76%72%68%56%72%28%68%68%84%44%80%88%
UAlbertaBot58.95%32%60%32%44%24%60%48%48%24%60%76%36%68%88%92%76%76%88%88%
MegaBot60.63%20%28%32%48%76%40%44%68%20%24%56%52%80%92%100%84%96%92%100%
Overkill56.84%8%24%8%20%40%60%52%76%8%64%52%44%88%92%92%92%96%72%92%
CasiaBot55.37%60%48%44%16%52%56%48%28%24%80%64%72%28%40%68%96%56%84%88%
Ziabot55.91%38%32%60%24%52%32%24%72%24%48%28%68%68%76%84%76%84%80%92%
Iron73.89%52%56%36%28%76%80%92%76%76%92%68%56%84%92%84%92%96%84%84%
AIUR54.95%4%16%20%32%40%76%36%20%52%8%72%76%80%80%92%68%96%84%92%
McRave53.89%8%20%56%44%24%44%48%36%72%32%28%36%84%88%84%40%88%96%96%
Tyr42.74%24%24%12%28%64%48%56%28%32%44%24%64%36%52%68%16%64%40%88%
SRbotOne44.21%24%36%24%72%32%20%12%72%32%16%20%16%64%44%48%100%36%76%96%
TerranUAB33.47%12%4%8%32%12%8%8%60%24%8%20%12%48%56%84%52%64%36%88%
Bonjwa29.68%12%16%40%32%8%0%8%32%16%16%8%16%32%52%16%64%60%44%92%
Bigeyes32.84%20%24%16%16%24%16%8%4%24%8%32%60%84%0%48%36%84%28%92%
OpprimoBot29.05%4%4%28%56%24%4%4%44%16%4%4%12%36%64%36%40%16%68%88%
Sling32.42%12%28%40%20%12%8%28%16%20%16%16%4%60%24%64%56%72%32%88%
Salsa8.42%8%0%4%12%12%0%8%12%8%16%8%4%12%4%12%8%8%12%12%

observations

The difficult maps are Hitchhiker, which has complex map features that bots are not ready for and which BWEM apparently does not support, and Alchemist, which has 2 entrances into each base, a nonstandard strategic feature. Sure enough, on Hitchhiker we see that BWEM users #10 Iron and #12 McRave were virtually helpless. #6 MegaBot suffered to a lesser extent. Other extreme results on Hitchhiker: #3 PurpleWave blanked #1 ZZZKBot with 100% wins, a surprise since Hitchhiker is a 2 player map with a short rush distance which should favor the 4 pool. I guess the defensible narrow path between bases made up for that. Then PurpleWave got smashed by #2 tscmoo with 4% wins. The map also brought #20 Salsa most of its wins, with plus scores versus #10 Iron and #12 McRave and, in a surprise, #19 Sling; on every other map, Salsa lost heavily in all matchups. It’s not the same syndrome as the BWEM users, though.

Alchemist is a 3 player map with a circular path all the way around the map which passes through each base. Out one of your base entrances is the short path to your enemy’s base, and out the other entrance is the long path. You get a natural expansion outside each entrance. It’s bad for bots which want to set up one defensive line, whether by walling the ramp or cannoning the natural, because the enemy can bypass the defenses. It favors bots which reason about map topology and distances—if there are any. Comparing across maps, we see that the map cost Iron (which makes a wall at the ramp or at the natural) over 20% win rate and cost #12 McRave perhaps half as much. #3 PurpleWave, #5 UAlbertaBot, and #6 MegaBot did well on the map. I’m not sure why PurpleWave liked the map. I don’t think any of the bots does much map analysis; most bots were not much affected by the nonstandard map, probably because they are blind to its possibilities.

Curiously, #3 PurpleWave had some trouble on Tau Cross and Andromeda. Those maps are also played on SSCAIT, so I expected PurpleWave would be well tuned for them.

There’s a lot more in these crosstables, but that’s enough for now.

Next: What would have happened without Hitchhiker and Alchemist?

CIG 2017 crosstable

My version of the CIG 2017 crosstable. I have small differences from the official results—see the explanation below. My results reverse the finishing places of #8 CasiaBot and #9 Ziabot as well as #17 Bigeyes and #18 OpprimoBot, because even small differences affect ranking.

The format of the results file has changed since last year. There is no documentation, so I don’t know what all the columns mean, but I only needed a few of them and I was able to pick them out. It turns out that column 6 is true if the first player won, otherwise false. It looks like each game is recorded twice, with the same winner, loser, and map but some differences in other data. I imagine that each player is running in its own instance, and each instance records its own data. The games are numbered so the duplicates can be recognized, and sometimes the games are recorded out of order. I had to rewrite parsing code, but only a handful of lines.

The results file turned out to have a section of corrupted data in the middle. Information about a small number of games is missing or corrupted, and I had to delete it from my input. The tournament format of 125 rounds with 20 participants called for 125 * 20 * 19 / 2 = 23750 games. Each game was recorded twice, so there should be 47500 lines in the results file. One game was expected to be missed because the tournament manager software has an off-by-one bug and doesn’t play one game. The last and highest-numbered game recorded in the results file is 23721, and numbering starts from 0 so it looks as though 3 games in fact went unplayed, or at least uncounted. There are 47487 lines remaining in the input file, accounting for 23722 games or 99.88% of the ideal 23750, or 99.89% of the expected and claimed 23749 games.

Anyway, my winning percentages are different from the official numbers mostly starting in the 3rd decimal place, which is what you expect with a discrepancy in game count in the third decimal place. Apparently the official numbers don’t suffer from corrupted data. I have written to the organizers to see if they can provide a clean result file.

overallZZZKtscmPurpLetaUAlbMegaOverCasiZiabIronAIURMcRaTyrSRboTerrBonjBigeOpprSlinSals
ZZZKBot75.43%52%37%53%82%77%90%31%67%66%93%83%84%82%88%89%86%94%90%90%
tscmoo73.50%48%90%82%54%67%65%46%66%58%69%74%69%75%92%89%79%94%85%94%
PurpleWave66.51%63%10%42%62%74%89%54%41%55%82%51%89%74%86%61%90%74%73%94%
LetaBot62.75%47%18%58%62%49%76%82%70%75%51%66%54%42%70%72%78%41%87%95%
UAlbertaBot61.67%18%46%38%38%30%62%60%70%42%60%79%46%67%88%90%73%87%86%92%
MegaBot61.06%23%33%26%51%70%55%37%61%44%44%60%50%81%90%82%77%97%82%96%
Overkill59.65%10%35%11%24%38%45%55%76%43%65%62%42%84%94%91%93%91%83%92%
CasiaBot58.32%69%54%46%18%40%63%45%34%46%62%59%70%29%47%82%93%75%84%94%
Ziabot58.49%33%34%59%30%30%39%24%66%47%58%39%60%73%79%89%82%90%81%96%
Iron58.11%34%42%45%25%58%56%57%54%53%69%56%45%69%74%70%76%76%73%74%
AIUR56.73%7%31%18%49%40%56%35%38%42%31%64%74%81%82%92%72%90%83%93%
McRave47.20%17%26%49%34%21%40%38%41%61%44%36%44%65%61%59%52%56%77%76%
Tyr45.32%16%31%11%46%54%50%58%30%40%55%26%56%24%45%58%49%63%58%91%
SRbotOne45.24%18%25%26%58%33%19%16%71%27%31%19%35%76%35%61%97%43%75%94%
TerranUAB38.58%12%8%14%30%12%10%6%53%21%26%18%39%55%65%77%63%67%65%93%
Bonjwa33.04%11%11%39%28%10%18%9%18%11%30%8%41%42%39%23%61%67%65%95%
Bigeyes30.90%14%21%10%22%27%23%7%7%18%24%28%48%51%3%37%39%60%58%91%
OpprimoBot31.90%6%6%26%59%13%3%9%25%10%24%10%44%37%57%33%33%40%79%93%
Sling26.07%10%15%27%13%14%18%17%16%19%27%17%23%42%25%35%35%42%21%77%
Salsa9.52%10%6%6%5%8%4%8%6%4%26%7%24%9%6%7%5%9%7%23%

observations

Newcomers #3 PurpleWave and #8 CasiaBot were the only players with positive scores versus #1 ZZZKBot. When I have time I’ll look into the replays and see if ZZZKBot was ready for the old-timers with special builds, or if it was the old-timers who weren’t ready for ZZZKBot. Perhaps ZZZKBot now has a learning feature and switches to a backup build if its 4 pool doesn’t work? I’m interested to find out.

#2 Tscmoo was edged out by #1 ZZZKBot and #8 CasiaBot, but otherwise had plus scores across the board. It showed stable performance across opponents—except for its crushing 90% win over #3 PurpleWave.

#3 PurpleWave did reasonably well against all except #2 tscmoo (10% wins) and #9 Zia. (I count 42% against #4 LetaBot as reasonably good, though it’s technically an upset.) But there were several weaker opponents that it edged out more narrowly than you might expect. My conclusion: Strong, but a little lacking in the solidity needed to defeat weaker opponents consistently. With more maturity it will likely become even stronger.

#8 CasiaBot seems to have the most uneven results, with severe upsets in both directions—69% versus #1 ZZZKBot and 29% against #14 SRbotOne.

The biggest upset is #18 OpprimoBot at 59% against #4 LetaBot.

The winning rates of #10 Iron and #12 McRave versus tail-ender #20 Salsa, which lost nearly all games against other opponents, are 74% and 76%. It backs up the claim that the two played only about 75% of games due to map problems with BWEM.

What do you notice in the crosstable?

Next: Per-map crosstables. We can expect dramatic numbers in some table cells thanks to Iron and McRave.

Update: I heard back from the organizers. They say that they had to alter the results file to compensate for errors in the tournament manager. And they think that all the problems are slight given the large number of games played. I think that’s true as far as it goes, but it leaves me feeling a little uneasy about the official results.