CIG 2016 - crosstables per map

Today I crush you under a mass of charts, crosstables for each of the 5 maps in CIG 2016. This is 4 dimensional data (bot 1, bot 2, map, winning rate) and I imagine there’s a clearer way to present it, but I don’t know what it is so you get it in the first form I thought of. At the end is a link to the software.

As a reminder, here are the maps.

(2)RideofValkyries1.0
(3)Alchemist1.0
(3)TauCross1.1
(4)LunaTheFinal2.3
(4)Python1.3

With 100 rounds and 5 maps, for each pairing 20 games were played on each map (minus a few games missing due to errors). So the percentages vary in steps of 5% (or more if games are missing), and the error bars are wide.

The qualifier tables are big. The first is the full tournament for comparison, the rest are the subtournaments played on each map.

	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	79.20%		56%	39%	56%	38%	63%	53%	91%	99%	95%	100%	100%	98%	100%	100%	100%
tscmoo	76.97%	44%		48%	53%	75%	87%	82%	43%	45%	81%	100%	100%	100%	97%	100%	100%
LetaBot	74.07%	61%	52%		81%	28%	60%	51%	31%	72%	78%	100%	100%	99%	99%	99%	100%
Overkill	70.98%	44%	47%	19%		84%	32%	56%	79%	31%	88%	98%	95%	93%	99%	100%	100%
MegaBot	70.11%	62%	25%	72%	16%		52%	7%	66%	85%	99%	96%	95%	87%	91%	99%	100%
UAlbertaBot	69.25%	37%	13%	40%	68%	48%		55%	77%	42%	88%	100%	89%	85%	97%	100%	100%
ZZZKBot	69.18%	47%	18%	49%	44%	93%	45%		69%	29%	49%	100%	100%	100%	94%	100%	100%
Aiur	63.15%	9%	57%	69%	21%	34%	23%	31%		44%	89%	86%	96%	99%	94%	96%	100%
Tyr	61.64%	1%	55%	28%	69%	15%	58%	71%	56%		24%	74%	98%	94%	88%	95%	99%
Ziabot	46.43%	5%	19%	22%	12%	1%	12%	51%	11%	76%		59%	98%	100%	32%	100%	100%
TerranUAB	33.51%	0%	0%	0%	2%	4%	0%	0%	14%	26%	41%		81%	76%	90%	70%	98%
SRbotOne	22.15%	0%	0%	0%	5%	5%	11%	0%	4%	2%	2%	19%		19%	92%	74%	99%
OpprimoBot	22.10%	2%	0%	1%	7%	13%	15%	0%	1%	6%	0%	24%	81%		56%	27%	99%
XelnagaII	20.71%	0%	3%	1%	1%	9%	3%	6%	6%	12%	68%	10%	8%	44%		56%	83%
Bonjwa	18.95%	0%	0%	1%	0%	1%	0%	0%	4%	5%	0%	30%	26%	73%	44%		100%
Salsa	1.47%	0%	0%	0%	0%	0%	0%	0%	0%	1%	0%	2%	1%	1%	17%	0%

Valkyries	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	72.67%		35%	10%	40%	25%	75%	40%	75%	95%	100%	100%	100%	95%	100%	100%	100%
tscmoo	80.67%	65%		55%	60%	80%	90%	80%	30%	55%	95%	100%	100%	100%	100%	100%	100%
LetaBot	72.00%	90%	45%		95%	30%	60%	10%	10%	85%	55%	100%	100%	100%	100%	100%	100%
Overkill	71.33%	60%	40%	5%		85%	25%	50%	95%	45%	70%	95%	100%	100%	100%	100%	100%
MegaBot	70.33%	75%	20%	70%	15%		30%	0%	65%	90%	100%	95%	95%	100%	100%	100%	100%
UAlbertaBot	69.67%	25%	10%	40%	75%	70%		50%	85%	35%	90%	100%	75%	90%	100%	100%	100%
ZZZKBot	77.67%	60%	20%	90%	50%	100%	50%		100%	55%	40%	100%	100%	100%	100%	100%	100%
Aiur	63.00%	25%	70%	90%	5%	35%	15%	0%		70%	70%	80%	95%	100%	90%	100%	100%
Tyr	59.67%	5%	45%	15%	55%	10%	65%	45%	30%		65%	70%	100%	100%	95%	95%	100%
Ziabot	51.33%	0%	5%	45%	30%	0%	10%	60%	30%	35%		65%	100%	100%	90%	100%	100%
TerranUAB	34.67%	0%	0%	0%	5%	5%	0%	0%	20%	30%	35%		80%	80%	90%	80%	95%
SRbotOne	24.33%	0%	0%	0%	0%	5%	25%	0%	5%	0%	0%	20%		25%	85%	100%	100%
OpprimoBot	20.00%	5%	0%	0%	0%	0%	10%	0%	0%	0%	0%	20%	75%		75%	20%	95%
XelnagaII	12.00%	0%	0%	0%	0%	0%	0%	0%	10%	5%	10%	10%	15%	25%		40%	65%
Bonjwa	17.67%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	20%	0%	80%	60%		100%
Salsa	3.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	5%	35%	0%

Alchemist	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	77.00%		80%	45%	80%	40%	40%	0%	95%	100%	75%	100%	100%	100%	100%	100%	100%
tscmoo	69.90%	20%		35%	75%	50%	80%	75%	15%	35%	75%	100%	100%	100%	89%	100%	100%
LetaBot	72.33%	55%	65%		55%	15%	40%	80%	15%	75%	90%	100%	100%	100%	100%	95%	100%
Overkill	66.67%	20%	25%	45%		90%	10%	55%	60%	45%	85%	100%	95%	75%	95%	100%	100%
MegaBot	70.67%	60%	50%	85%	10%		45%	15%	60%	90%	100%	85%	90%	95%	80%	95%	100%
UAlbertaBot	77.00%	60%	20%	60%	90%	55%		40%	85%	80%	95%	100%	85%	85%	100%	100%	100%
ZZZKBot	72.67%	100%	25%	20%	45%	85%	60%		65%	50%	45%	100%	100%	100%	95%	100%	100%
Aiur	68.00%	5%	85%	85%	40%	40%	15%	35%		60%	90%	80%	90%	100%	100%	95%	100%
Tyr	51.33%	0%	65%	25%	55%	10%	20%	50%	40%		10%	60%	90%	80%	80%	85%	100%
Ziabot	47.33%	25%	25%	10%	15%	0%	5%	55%	10%	90%		50%	100%	100%	25%	100%	100%
TerranUAB	36.67%	0%	0%	0%	0%	15%	0%	0%	20%	40%	50%		80%	80%	90%	80%	95%
SRbotOne	23.67%	0%	0%	0%	5%	10%	15%	0%	10%	10%	0%	20%		5%	100%	80%	100%
OpprimoBot	22.33%	0%	0%	0%	25%	5%	15%	0%	0%	20%	0%	20%	95%		35%	20%	100%
XelnagaII	25.08%	0%	11%	0%	5%	20%	0%	5%	0%	20%	75%	10%	0%	65%		75%	90%
Bonjwa	18.33%	0%	0%	5%	0%	5%	0%	0%	5%	15%	0%	20%	20%	80%	25%		100%
Salsa	1.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	0%	10%	0%

Tau Cross	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	82.33%		65%	40%	75%	15%	55%	95%	90%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	77.33%	35%		35%	40%	100%	90%	95%	45%	35%	85%	100%	100%	100%	100%	100%	100%
LetaBot	80.67%	60%	65%		90%	25%	80%	80%	60%	50%	100%	100%	100%	100%	100%	100%	100%
Overkill	73.33%	25%	60%	10%		100%	40%	55%	95%	20%	95%	100%	100%	100%	100%	100%	100%
MegaBot	72.33%	85%	0%	75%	0%		55%	15%	85%	95%	100%	100%	100%	80%	95%	100%	100%
UAlbertaBot	67.33%	45%	10%	20%	60%	45%		65%	75%	35%	85%	100%	95%	75%	100%	100%	100%
ZZZKBot	57.67%	5%	5%	20%	45%	85%	35%		25%	5%	55%	100%	100%	100%	85%	100%	100%
Aiur	61.20%	10%	55%	40%	5%	15%	25%	75%		15%	95%	95%	100%	100%	95%	95%	100%
Tyr	68.33%	0%	65%	50%	80%	5%	65%	95%	85%		10%	80%	100%	100%	90%	100%	100%
Ziabot	43.00%	0%	15%	0%	5%	0%	15%	45%	5%	90%		65%	95%	100%	10%	100%	100%
TerranUAB	33.44%	0%	0%	0%	0%	0%	0%	0%	5%	20%	35%		80%	90%	100%	70%	100%
SRbotOne	21.00%	0%	0%	0%	0%	0%	5%	0%	0%	0%	5%	20%		20%	95%	70%	100%
OpprimoBot	20.67%	0%	0%	0%	0%	20%	25%	0%	0%	0%	0%	10%	80%		55%	20%	100%
XelnagaII	22.00%	0%	0%	0%	0%	5%	0%	15%	5%	10%	90%	0%	5%	45%		70%	85%
Bonjwa	18.33%	0%	0%	0%	0%	0%	0%	0%	5%	0%	0%	30%	30%	80%	30%		100%
Salsa	1.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	15%	0%

Luna	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	81.33%		35%	55%	50%	50%	65%	70%	95%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	81.00%	65%		40%	35%	70%	95%	90%	65%	75%	85%	100%	100%	100%	95%	100%	100%
LetaBot	74.00%	45%	60%		70%	25%	55%	55%	30%	85%	85%	100%	100%	100%	100%	100%	100%
Overkill	68.90%	50%	65%	30%		70%	35%	55%	65%	0%	95%	100%	80%	89%	100%	100%	100%
MegaBot	71.24%	50%	30%	75%	30%		70%	0%	65%	75%	95%	100%	95%	90%	95%	100%	100%
UAlbertaBot	66.22%	35%	5%	45%	65%	30%		60%	75%	35%	75%	100%	95%	80%	95%	100%	100%
ZZZKBot	68.33%	30%	10%	45%	45%	100%	40%		80%	30%	55%	100%	100%	100%	90%	100%	100%
Aiur	60.67%	5%	35%	70%	35%	35%	25%	20%		35%	90%	85%	95%	95%	90%	95%	100%
Tyr	63.21%	0%	25%	15%	100%	25%	65%	70%	65%		30%	75%	100%	100%	85%	100%	95%
Ziabot	45.33%	0%	15%	15%	5%	5%	25%	45%	10%	70%		60%	95%	100%	35%	100%	100%
TerranUAB	31.67%	0%	0%	0%	0%	0%	0%	0%	15%	25%	40%		85%	65%	80%	65%	100%
SRbotOne	20.67%	0%	0%	0%	20%	5%	5%	0%	5%	0%	5%	15%		10%	80%	65%	100%
OpprimoBot	24.16%	0%	0%	0%	11%	10%	20%	0%	5%	0%	0%	35%	90%		60%	35%	100%
XelnagaII	22.07%	0%	5%	0%	0%	5%	5%	10%	10%	15%	65%	20%	20%	40%		45%	90%
Bonjwa	19.67%	0%	0%	0%	0%	0%	0%	0%	5%	0%	0%	35%	35%	65%	55%		100%
Salsa	1.01%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	0%	0%	0%	10%	0%

Python	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	82.67%		65%	45%	35%	60%	80%	60%	100%	100%	100%	100%	100%	95%	100%	100%	100%
tscmoo	75.92%	35%		75%	55%	75%	80%	70%	60%	25%	65%	100%	100%	100%	100%	100%	100%
LetaBot	71.33%	55%	25%		95%	45%	65%	30%	40%	65%	60%	100%	100%	95%	95%	100%	100%
Overkill	74.67%	65%	45%	5%		75%	50%	65%	80%	45%	95%	95%	100%	100%	100%	100%	100%
MegaBot	66.00%	40%	25%	55%	25%		60%	5%	55%	75%	100%	100%	95%	70%	85%	100%	100%
UAlbertaBot	66.00%	20%	20%	35%	50%	40%		60%	65%	25%	95%	100%	95%	95%	90%	100%	100%
ZZZKBot	69.57%	40%	30%	70%	35%	95%	40%		75%	5%	53%	100%	100%	100%	100%	100%	100%
Aiur	62.88%	0%	40%	60%	20%	45%	35%	25%		40%	100%	90%	100%	100%	95%	95%	100%
Tyr	65.67%	0%	75%	35%	55%	25%	75%	95%	60%		5%	85%	100%	90%	90%	95%	100%
Ziabot	45.12%	0%	35%	40%	5%	0%	5%	47%	0%	95%		53%	100%	100%	0%	100%	100%
TerranUAB	31.10%	0%	0%	0%	5%	0%	0%	0%	10%	15%	47%		80%	65%	90%	55%	100%
SRbotOne	21.07%	0%	0%	0%	0%	5%	5%	0%	0%	0%	0%	20%		35%	100%	55%	95%
OpprimoBot	23.33%	5%	0%	5%	0%	30%	5%	0%	0%	10%	0%	35%	65%		55%	40%	100%
XelnagaII	22.41%	0%	0%	5%	0%	15%	10%	0%	5%	10%	100%	10%	0%	45%		50%	85%
Bonjwa	20.74%	0%	0%	0%	0%	0%	0%	0%	5%	5%	0%	45%	45%	60%	50%		100%
Salsa	1.33%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	15%	0%

The final tables are small. Again, the first is the full tournament, the rest are the maps.

	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	65.14%		52%	44%	79%	71%	77%	83%	50%
Iron	54.43%	48%		38%	49%	49%	74%	30%	93%
LetaBot	53.71%	56%	62%		49%	81%	69%	30%	29%
ZZZKBot	53.08%	21%	51%	51%		42%	35%	93%	78%
Overkill	51.43%	29%	51%	19%	58%		43%	81%	79%
UAlbertaBot	49.07%	23%	26%	31%	65%	57%		76%	66%
MegaBot	38.00%	17%	70%	70%	7%	19%	24%		59%
Aiur	35.14%	50%	7%	71%	22%	21%	34%	41%

Valkyries	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	68.57%		60%	30%	75%	75%	85%	90%	65%
Iron	47.86%	40%		25%	40%	50%	60%	25%	95%
LetaBot	50.71%	70%	75%		0%	85%	70%	30%	25%
ZZZKBot	67.86%	25%	60%	100%		40%	55%	95%	100%
Overkill	52.14%	25%	50%	15%	60%		40%	100%	75%
UAlbertaBot	50.00%	15%	40%	30%	45%	60%		90%	70%
MegaBot	35.71%	10%	75%	70%	5%	0%	10%		80%
Aiur	27.14%	35%	5%	75%	0%	25%	30%	20%

Alchemist	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	49.29%		45%	35%	65%	80%	55%	50%	15%
Iron	44.29%	55%		50%	0%	35%	60%	30%	80%
LetaBot	51.43%	65%	50%		75%	60%	65%	20%	25%
ZZZKBot	61.43%	35%	100%	25%		45%	45%	95%	85%
Overkill	51.43%	20%	65%	40%	55%		45%	75%	60%
UAlbertaBot	52.14%	45%	40%	35%	55%	55%		75%	60%
MegaBot	42.14%	50%	70%	80%	5%	25%	25%		40%
Aiur	47.86%	85%	20%	75%	15%	40%	40%	60%

Tau Cross	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	61.43%		25%	45%	90%	45%	100%	95%	30%
Iron	66.43%	75%		55%	95%	50%	85%	15%	90%
LetaBot	61.43%	55%	45%		85%	90%	80%	35%	40%
ZZZKBot	29.29%	10%	5%	15%		50%	25%	75%	25%
Overkill	52.14%	55%	50%	10%	50%		40%	70%	90%
UAlbertaBot	45.00%	0%	15%	20%	75%	60%		75%	70%
MegaBot	37.86%	5%	85%	65%	25%	30%	25%		30%
Aiur	46.43%	70%	10%	60%	75%	10%	30%	70%

Luna	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	71.43%		70%	60%	80%	75%	75%	90%	50%
Iron	55.00%	30%		30%	55%	50%	80%	40%	100%
LetaBot	51.43%	40%	70%		65%	75%	65%	20%	25%
ZZZKBot	52.14%	20%	45%	35%		60%	25%	100%	80%
Overkill	48.57%	25%	50%	25%	40%		45%	85%	70%
UAlbertaBot	50.00%	25%	20%	35%	75%	55%		70%	70%
MegaBot	40.00%	10%	60%	80%	0%	15%	30%		85%
Aiur	31.43%	50%	0%	75%	20%	30%	30%	15%

Python	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	75.00%		60%	50%	85%	80%	70%	90%	90%
Iron	58.57%	40%		30%	55%	60%	85%	40%	100%
LetaBot	53.57%	50%	70%		20%	95%	65%	45%	30%
ZZZKBot	54.68%	15%	45%	80%		15%	26%	100%	100%
Overkill	52.86%	20%	40%	5%	85%		45%	75%	100%
UAlbertaBot	48.20%	30%	15%	35%	74%	55%		70%	60%
MegaBot	34.29%	10%	60%	55%	0%	25%	30%		60%
Aiur	22.86%	10%	0%	70%	0%	0%	40%	40%

The charts are full of small insights—more than I have time to examine. See for example how XelnagaII’s upset of Ziabot occurred on all maps except Ride of Valkyries; I’m sure that says something about at least one of those bots. We can tease out which pairings the map imbalances spring from. ZZZKBot did poorly on Tau Cross, as explained by Martin Rooijackers due to the long rush distance. And so on.

My strongest impression is how much results vary from map to map. I still think 5 maps are not enough to judge strength fairly. To my eye, the datapoint that stands out most is that ZZZKBot defeated the powerful Iron 100% of the time on Alchemist, in both the qualifier and the final, although otherwise Alchemist was a mediocre map for ZZZKBot. It looks as though Iron has a strategy bug on that map which ZZZKBot exploits. All bot authors who competed may want to eye the charts for hints about weaknesses to fix.

Download a zip file of the perl scripts with documentation.

Trackbacks

No Trackbacks

Comments

Igor Dimitrijevic on Monday, September 26. 2016:

Not a bug, but lack of time, not a priority. Iron's strategy against early zergling rushes consists in producing marines and blocking the entrance of the starting base. If there are more than one entrance, or if the entrance is too large, Iron still produces marines, but it doesn't try to block anything. Against strong 4-pool bots like ZZZKBot, this is obviouly not enough.
Interesting highlights BTW.

Jay Scott on Monday, September 26. 2016:

Oh right, I’d forgotten, Alchemist has two entrances into each main. It’s not a very standard map. I approve, in a way—that makes it a good strategy prod. :-) If Iron messes up, it’s not a code bug, but I would call it a strategy bug: The strategy doesn’t account for that case. Strategy bugs are common on non-standard maps, because bots are optimized for the usual case... which is what makes it a low priority. I favor sneaking a few non-standard maps into the map pool because it pushes bots to become more flexible.

Add Comment

Name*

Homepage

Comment*

In reply to

E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA