Entries by Jay Scott | Starcraft AI blog

AIIDE 2016 - crosstables per map

AIIDE 2016 crosstables for each map in the tournament. First, the whole tournament crosstable, for reference. It should match the one on the official results page, though for some reason some of the overall win percentages differ in the second decimal (by a trivial amount). The official results are correct, so I may have some floating point error accumulation (an off-by-one error would cause a bigger discrepancy).

AIIDE 2016	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	87.44%		23%	84%	87%	86%	99%	88%	96%	68%	98%	81%	91%	57%	100%	96%	100%	99%	98%	100%	100%	100%
ZZZKBot	85.05%	77%		50%	51%	77%	98%	39%	87%	92%	93%	46%	100%	94%	100%	100%	100%	98%	100%	100%	100%	100%
tscmoo	82.71%	16%	50%		43%	42%	91%	97%	80%	90%	99%	91%	87%	83%	100%	98%	100%	100%	99%	89%	100%	100%
LetaBot	74.00%	13%	49%	57%		66%	68%	94%	34%	52%	84%	88%	39%	62%	93%	93%	92%	99%	99%	97%	100%	100%
UAlbertaBot	70.37%	14%	23%	58%	34%		76%	42%	80%	47%	77%	71%	100%	79%	81%	83%	99%	86%	59%	99%	100%	100%
Ximp	64.54%	1%	2%	9%	32%	24%		52%	81%	67%	7%	96%	62%	98%	94%	91%	98%	92%	96%	92%	97%	100%
Overkill	61.93%	12%	61%	3%	6%	58%	48%		30%	34%	79%	52%	24%	62%	94%	97%	96%	98%	94%	92%	100%	100%
Aiur	61.26%	4%	13%	20%	66%	20%	19%	70%		53%	79%	53%	56%	86%	87%	44%	88%	96%	92%	88%	91%	100%
MegaBot	58.42%	32%	8%	10%	48%	53%	33%	66%	47%		49%	42%	32%	78%	47%	91%	94%	83%	89%	84%	91%	90%
IceBot	57.43%	2%	7%	1%	16%	23%	93%	21%	21%	51%		56%	36%	61%	61%	100%	100%	100%	99%	100%	100%	100%
JiaBot	57.25%	19%	54%	9%	12%	29%	4%	48%	47%	58%	44%		49%	37%	98%	89%	90%	82%	99%	86%	91%	100%
Xelnaga	56.98%	9%	0%	13%	61%	0%	38%	76%	44%	68%	64%	51%		42%	2%	99%	100%	92%	94%	89%	96%	100%
Skynet	55.03%	43%	6%	17%	38%	21%	2%	38%	14%	22%	39%	63%	58%		100%	39%	100%	100%	100%	100%	100%	100%
GarmBot	42.52%	0%	0%	0%	7%	19%	6%	6%	13%	53%	39%	2%	98%	0%		84%	84%	100%	93%	47%	99%	100%
NUSBot	27.46%	4%	0%	2%	7%	17%	9%	3%	56%	9%	0%	11%	1%	61%	16%		57%	34%	61%	81%	30%	90%
TerranUAB	27.39%	0%	0%	0%	8%	1%	2%	4%	12%	6%	0%	10%	0%	0%	16%	43%		90%	73%	88%	96%	99%
SRbotOne	22.46%	1%	2%	0%	1%	14%	8%	2%	4%	17%	0%	18%	8%	0%	0%	66%	10%		53%	88%	57%	100%
Cimex	20.50%	2%	0%	1%	1%	41%	4%	6%	8%	11%	1%	1%	6%	0%	7%	39%	27%	47%		59%	51%	99%
Oritaka	19.51%	0%	0%	11%	3%	1%	8%	8%	12%	16%	0%	14%	11%	0%	53%	19%	12%	12%	41%		68%	100%
CruzBot	16.75%	0%	0%	0%	0%	0%	3%	0%	9%	9%	0%	9%	4%	0%	1%	70%	4%	43%	49%	32%		100%
Tyr	1.11%	0%	0%	0%	0%	0%	0%	0%	0%	10%	0%	0%	0%	0%	0%	10%	1%	0%	1%	0%	0%

Now each of the 10 maps. Each cell represents only 9 games, occasionally fewer when games are missing.

(2)Benzene	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	95.00%		89%	100%	78%	100%	100%	78%	100%	89%	100%	89%	100%	78%	100%	100%	100%	100%	100%	100%	100%	100%
ZZZKBot	73.89%	11%		22%	11%	78%	100%	11%	11%	89%	100%	67%	100%	78%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	83.33%	0%	78%		22%	44%	100%	100%	78%	89%	100%	100%	78%	78%	100%	100%	100%	100%	100%	100%	100%	100%
LetaBot	81.67%	22%	89%	78%		67%	67%	100%	44%	33%	100%	100%	44%	89%	100%	100%	100%	100%	100%	100%	100%	100%
UAlbertaBot	64.80%	0%	22%	56%	33%		44%	22%	88%	44%	78%	67%	100%	78%	78%	89%	89%	67%	56%	89%	100%	100%
Ximp	59.22%	0%	0%	0%	33%	56%		25%	11%	78%	11%	100%	44%	100%	100%	33%	100%	100%	100%	89%	100%	100%
Overkill	68.93%	22%	89%	0%	0%	78%	75%		22%	44%	89%	78%	22%	100%	89%	100%	89%	100%	100%	89%	100%	100%
Aiur	72.07%	0%	89%	22%	56%	12%	89%	78%		78%	67%	78%	78%	89%	100%	33%	100%	89%	89%	100%	89%	100%
MegaBot	56.11%	11%	11%	11%	67%	56%	22%	56%	22%		22%	44%	22%	89%	56%	100%	89%	78%	100%	100%	67%	100%
IceBot	60.00%	0%	0%	0%	0%	22%	89%	11%	33%	78%		67%	33%	100%	67%	100%	100%	100%	100%	100%	100%	100%
JiaBot	52.78%	11%	33%	0%	0%	33%	0%	22%	22%	56%	33%		67%	56%	100%	67%	100%	89%	100%	89%	78%	100%
Xelnaga	56.67%	0%	0%	22%	56%	0%	56%	78%	22%	78%	67%	33%		67%	0%	100%	100%	100%	89%	89%	78%	100%
Skynet	45.81%	22%	22%	22%	11%	22%	0%	0%	11%	11%	0%	44%	33%		100%	11%	100%	100%	100%	100%	100%	100%
GarmBot	42.78%	0%	0%	0%	0%	22%	0%	11%	0%	44%	33%	0%	100%	0%		78%	100%	100%	100%	67%	100%	100%
NUSBot	31.11%	0%	0%	0%	0%	11%	67%	0%	67%	0%	0%	33%	0%	89%	22%		22%	33%	67%	100%	11%	100%
TerranUAB	27.22%	0%	0%	0%	0%	11%	0%	11%	0%	11%	0%	0%	0%	0%	0%	78%		100%	56%	89%	89%	100%
SRbotOne	20.56%	0%	0%	0%	0%	33%	0%	0%	11%	22%	0%	11%	0%	0%	0%	67%	0%		44%	100%	22%	100%
Cimex	22.78%	0%	0%	0%	0%	44%	0%	0%	11%	0%	0%	0%	11%	0%	0%	33%	44%	56%		67%	89%	100%
Oritaka	14.44%	0%	0%	0%	0%	11%	11%	11%	0%	0%	0%	11%	11%	0%	33%	0%	11%	0%	33%		56%	100%
CruzBot	21.23%	0%	0%	0%	0%	0%	0%	0%	11%	33%	0%	22%	22%	0%	0%	89%	11%	78%	11%	44%		100%
Tyr	0.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%

(2)Destination	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	91.67%		22%	89%	67%	89%	100%	100%	100%	78%	100%	89%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
ZZZKBot	89.44%	78%		56%	56%	100%	100%	44%	100%	100%	100%	67%	100%	89%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	85.47%	11%	44%		44%	67%	100%	89%	67%	100%	100%	89%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
LetaBot	84.44%	33%	44%	56%		67%	89%	100%	56%	89%	100%	78%	78%	100%	100%	100%	100%	100%	100%	100%	100%	100%
UAlbertaBot	68.33%	11%	0%	33%	33%		56%	11%	78%	78%	78%	67%	100%	100%	89%	89%	100%	78%	67%	100%	100%	100%
Ximp	70.00%	0%	0%	0%	11%	44%		78%	100%	78%	11%	89%	89%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Overkill	63.89%	0%	56%	11%	0%	89%	22%		44%	67%	78%	67%	33%	56%	78%	89%	100%	100%	89%	100%	100%	100%
Aiur	61.67%	0%	0%	33%	44%	22%	0%	56%		67%	67%	33%	100%	100%	89%	56%	100%	100%	78%	89%	100%	100%
MegaBot	50.56%	22%	0%	0%	11%	22%	22%	33%	33%		11%	67%	22%	89%	11%	89%	89%	100%	89%	100%	100%	100%
IceBot	60.56%	0%	0%	0%	0%	22%	89%	22%	33%	89%		56%	33%	89%	89%	100%	100%	100%	89%	100%	100%	100%
JiaBot	56.67%	11%	33%	11%	22%	33%	11%	33%	67%	33%	44%		67%	22%	100%	78%	89%	78%	100%	100%	100%	100%
Xelnaga	51.11%	0%	0%	0%	22%	0%	11%	67%	0%	78%	67%	33%		67%	11%	100%	100%	78%	100%	89%	100%	100%
Skynet	46.67%	0%	11%	0%	0%	0%	0%	44%	0%	11%	11%	78%	33%		100%	44%	100%	100%	100%	100%	100%	100%
GarmBot	43.89%	0%	0%	0%	0%	11%	0%	22%	11%	89%	11%	0%	89%	0%		67%	78%	100%	100%	100%	100%	100%
NUSBot	35.00%	0%	0%	0%	0%	11%	0%	11%	44%	11%	0%	22%	0%	56%	33%		89%	100%	78%	78%	67%	100%
TerranUAB	24.44%	0%	0%	0%	0%	0%	0%	0%	0%	11%	0%	11%	0%	0%	22%	11%		89%	78%	78%	89%	100%
SRbotOne	21.79%	0%	0%	0%	0%	22%	0%	0%	0%	0%	0%	22%	22%	0%	0%	0%	11%		78%	100%	78%	100%
Cimex	17.78%	0%	0%	0%	0%	33%	0%	11%	22%	11%	11%	0%	0%	0%	0%	22%	22%	22%		67%	33%	100%
Oritaka	13.89%	0%	0%	0%	0%	0%	0%	0%	11%	0%	0%	0%	11%	0%	0%	22%	22%	0%	33%		78%	100%
CruzBot	12.78%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	33%	11%	22%	67%	22%		100%
Tyr	0.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%

(2)Heartbreak Ridge	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	88.89%		22%	89%	89%	89%	100%	100%	100%	33%	100%	89%	78%	89%	100%	100%	100%	100%	100%	100%	100%	100%
ZZZKBot	87.78%	78%		78%	67%	78%	100%	44%	100%	100%	100%	11%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	80.56%	11%	22%		44%	56%	78%	100%	67%	89%	100%	89%	89%	89%	100%	100%	100%	100%	100%	78%	100%	100%
LetaBot	72.22%	11%	33%	56%		78%	44%	100%	22%	11%	89%	100%	0%	100%	100%	100%	100%	100%	100%	100%	100%	100%
UAlbertaBot	70.00%	11%	22%	44%	22%		56%	44%	89%	56%	78%	56%	100%	89%	100%	100%	100%	67%	67%	100%	100%	100%
Ximp	73.33%	0%	0%	22%	56%	44%		67%	100%	78%	11%	100%	89%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Overkill	51.18%	0%	56%	0%	0%	56%	33%		0%	12%	83%	22%	12%	22%	100%	100%	89%	89%	89%	88%	100%	100%
Aiur	68.89%	0%	0%	33%	78%	11%	0%	100%		67%	100%	78%	78%	100%	78%	100%	78%	100%	100%	78%	100%	100%
MegaBot	54.19%	67%	0%	11%	89%	44%	22%	88%	33%		100%	44%	22%	11%	44%	100%	100%	100%	100%	22%	89%	0%
IceBot	54.80%	0%	0%	0%	11%	22%	89%	17%	0%	0%		56%	11%	89%	89%	100%	100%	100%	100%	100%	100%	100%
JiaBot	58.33%	11%	89%	11%	0%	44%	0%	78%	22%	56%	44%		44%	22%	100%	100%	100%	67%	100%	89%	89%	100%
Xelnaga	60.89%	22%	0%	11%	100%	0%	11%	88%	22%	78%	89%	56%		67%	0%	100%	100%	100%	100%	89%	89%	100%
Skynet	56.11%	11%	0%	11%	0%	11%	0%	78%	0%	89%	11%	78%	33%		100%	100%	100%	100%	100%	100%	100%	100%
GarmBot	38.33%	0%	0%	0%	0%	0%	0%	0%	22%	56%	11%	0%	100%	0%		100%	78%	100%	100%	11%	89%	100%
NUSBot	0.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%		0%	0%	0%	0%	0%	0%
TerranUAB	30.56%	0%	0%	0%	0%	0%	0%	11%	22%	0%	0%	0%	0%	0%	22%	100%		89%	78%	100%	89%	100%
SRbotOne	22.22%	0%	0%	0%	0%	33%	0%	11%	0%	0%	0%	33%	0%	0%	0%	100%	11%		44%	67%	44%	100%
Cimex	21.67%	0%	0%	0%	0%	33%	0%	11%	0%	0%	0%	0%	0%	0%	0%	100%	22%	56%		56%	56%	100%
Oritaka	28.49%	0%	0%	22%	0%	0%	0%	12%	22%	78%	0%	11%	11%	0%	89%	100%	0%	33%	44%		44%	100%
CruzBot	20.79%	0%	0%	0%	0%	0%	0%	0%	0%	11%	0%	11%	11%	0%	11%	100%	11%	56%	44%	56%		100%
Tyr	10.06%	0%	0%	0%	0%	0%	0%	0%	0%	100%	0%	0%	0%	0%	0%	100%	0%	0%	0%	0%	0%

(3)Aztec	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	91.11%		11%	78%	89%	100%	100%	89%	100%	89%	100%	89%	89%	89%	100%	100%	100%	100%	100%	100%	100%	100%
ZZZKBot	85.56%	89%		56%	44%	89%	100%	44%	78%	89%	100%	22%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	81.67%	22%	44%		22%	56%	78%	100%	67%	89%	100%	100%	100%	67%	100%	89%	100%	100%	100%	100%	100%	100%
LetaBot	78.33%	11%	56%	78%		44%	89%	100%	33%	56%	100%	100%	11%	100%	100%	89%	100%	100%	100%	100%	100%	100%
UAlbertaBot	65.56%	0%	11%	44%	56%		100%	11%	44%	56%	56%	89%	100%	78%	56%	89%	100%	89%	33%	100%	100%	100%
Ximp	62.78%	0%	0%	22%	11%	0%		67%	100%	67%	0%	78%	89%	100%	89%	100%	89%	78%	78%	100%	89%	100%
Overkill	60.00%	11%	56%	0%	0%	89%	33%		22%	11%	100%	56%	11%	67%	89%	100%	100%	100%	100%	56%	100%	100%
Aiur	63.89%	0%	22%	33%	67%	56%	0%	78%		67%	44%	44%	67%	89%	89%	67%	89%	89%	100%	89%	89%	100%
MegaBot	60.00%	11%	11%	11%	44%	44%	33%	89%	33%		67%	22%	11%	89%	56%	100%	100%	100%	89%	100%	89%	100%
IceBot	59.44%	0%	0%	0%	0%	44%	100%	0%	56%	33%		44%	33%	100%	78%	100%	100%	100%	100%	100%	100%	100%
JiaBot	60.00%	11%	78%	0%	0%	11%	22%	44%	56%	78%	56%		56%	56%	89%	78%	89%	100%	100%	100%	78%	100%
Xelnaga	57.78%	11%	0%	0%	89%	0%	11%	89%	33%	89%	67%	44%		33%	0%	100%	100%	100%	89%	100%	100%	100%
Skynet	48.89%	11%	0%	33%	0%	22%	0%	33%	11%	11%	0%	44%	67%		100%	44%	100%	100%	100%	100%	100%	100%
GarmBot	45.00%	0%	0%	0%	0%	44%	11%	11%	11%	44%	22%	11%	100%	0%		100%	100%	100%	100%	44%	100%	100%
NUSBot	22.22%	0%	0%	11%	11%	11%	0%	0%	33%	0%	0%	22%	0%	56%	0%		11%	22%	44%	89%	33%	100%
TerranUAB	27.78%	0%	0%	0%	0%	0%	11%	0%	11%	0%	0%	11%	0%	0%	0%	89%		89%	67%	89%	89%	100%
SRbotOne	23.89%	0%	0%	0%	0%	11%	22%	0%	11%	0%	0%	0%	0%	0%	0%	78%	11%		67%	100%	78%	100%
Cimex	24.44%	0%	0%	0%	0%	67%	22%	0%	0%	11%	0%	0%	11%	0%	0%	56%	33%	33%		78%	78%	100%
Oritaka	16.67%	0%	0%	0%	0%	0%	0%	44%	11%	0%	0%	0%	0%	0%	56%	11%	11%	0%	22%		78%	100%
CruzBot	15.00%	0%	0%	0%	0%	0%	11%	0%	11%	11%	0%	22%	0%	0%	0%	67%	11%	22%	22%	22%		100%
Tyr	0.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%

(3)Tau Cross	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	86.67%		22%	100%	89%	67%	100%	89%	89%	44%	100%	78%	67%	89%	100%	100%	100%	100%	100%	100%	100%	100%
ZZZKBot	83.80%	78%		44%	12%	89%	100%	22%	89%	100%	100%	44%	100%	100%	100%	100%	100%	89%	100%	100%	100%	100%
tscmoo	84.44%	0%	56%		44%	78%	89%	89%	100%	100%	100%	100%	78%	89%	100%	100%	100%	100%	100%	67%	100%	100%
LetaBot	76.70%	11%	88%	56%		78%	44%	100%	44%	33%	89%	100%	22%	89%	88%	100%	100%	100%	100%	100%	100%	100%
UAlbertaBot	67.22%	33%	11%	22%	22%		44%	56%	78%	22%	67%	67%	100%	78%	100%	56%	100%	100%	89%	100%	100%	100%
Ximp	70.56%	0%	0%	11%	56%	56%		56%	78%	67%	11%	100%	78%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Overkill	59.78%	11%	78%	11%	0%	44%	44%		22%	33%	56%	33%	44%	44%	100%	100%	89%	100%	89%	89%	100%	100%
Aiur	61.67%	11%	11%	0%	56%	22%	22%	78%		67%	89%	56%	67%	100%	78%	44%	67%	100%	100%	78%	89%	100%
MegaBot	63.89%	56%	0%	0%	67%	78%	33%	67%	33%		56%	44%	33%	89%	56%	89%	100%	100%	89%	89%	100%	100%
IceBot	58.89%	0%	0%	0%	11%	33%	89%	44%	11%	44%		67%	44%	89%	44%	100%	100%	100%	100%	100%	100%	100%
JiaBot	53.63%	22%	56%	0%	0%	33%	0%	67%	44%	56%	33%		33%	11%	100%	100%	67%	78%	100%	78%	89%	100%
Xelnaga	56.11%	33%	0%	22%	78%	0%	22%	56%	33%	67%	56%	67%		22%	0%	100%	100%	100%	100%	67%	100%	100%
Skynet	50.56%	11%	0%	11%	11%	22%	0%	56%	0%	11%	11%	89%	78%		100%	11%	100%	100%	100%	100%	100%	100%
GarmBot	40.78%	0%	0%	0%	12%	0%	0%	0%	22%	44%	56%	0%	100%	0%		67%	78%	100%	100%	33%	100%	100%
NUSBot	34.44%	0%	0%	0%	0%	44%	0%	0%	56%	11%	0%	0%	0%	89%	33%		100%	33%	100%	89%	33%	100%
TerranUAB	26.67%	0%	0%	0%	0%	0%	0%	11%	33%	0%	0%	33%	0%	0%	22%	0%		100%	67%	67%	100%	100%
SRbotOne	17.78%	0%	11%	0%	0%	0%	0%	0%	0%	0%	0%	22%	0%	0%	0%	67%	0%		67%	78%	11%	100%
Cimex	15.56%	0%	0%	0%	0%	11%	0%	11%	0%	11%	0%	0%	0%	0%	0%	0%	33%	33%		56%	56%	100%
Oritaka	25.56%	0%	0%	33%	0%	0%	0%	11%	22%	11%	0%	22%	33%	0%	67%	11%	33%	22%	44%		100%	100%
CruzBot	16.11%	0%	0%	0%	0%	0%	0%	0%	11%	0%	0%	11%	0%	0%	0%	67%	0%	89%	44%	0%		100%
Tyr	0.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%

(4)Andromeda	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	82.78%		22%	56%	78%	78%	89%	100%	100%	56%	100%	67%	100%	11%	100%	100%	100%	100%	100%	100%	100%	100%
ZZZKBot	82.78%	78%		33%	22%	67%	100%	67%	100%	78%	67%	56%	100%	89%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	85.56%	44%	67%		44%	33%	89%	100%	100%	89%	89%	100%	78%	78%	100%	100%	100%	100%	100%	100%	100%	100%
LetaBot	81.11%	22%	78%	56%		89%	78%	100%	33%	78%	89%	100%	89%	33%	89%	100%	100%	100%	100%	89%	100%	100%
UAlbertaBot	73.33%	22%	33%	67%	11%		100%	56%	89%	33%	89%	78%	100%	78%	78%	89%	100%	89%	56%	100%	100%	100%
Ximp	60.56%	11%	0%	11%	22%	0%		56%	89%	89%	0%	100%	67%	89%	78%	100%	100%	78%	100%	44%	78%	100%
Overkill	62.78%	0%	33%	0%	0%	44%	44%		44%	44%	78%	78%	67%	44%	100%	100%	100%	89%	89%	100%	100%	100%
Aiur	55.56%	0%	0%	0%	67%	11%	11%	56%		33%	89%	56%	44%	78%	78%	11%	89%	100%	100%	89%	100%	100%
MegaBot	57.22%	44%	22%	11%	22%	67%	11%	56%	67%		22%	33%	56%	78%	33%	78%	100%	78%	89%	89%	89%	100%
IceBot	57.22%	0%	33%	11%	11%	11%	100%	22%	11%	78%		44%	67%	22%	33%	100%	100%	100%	100%	100%	100%	100%
JiaBot	55.00%	33%	44%	0%	0%	22%	0%	22%	44%	67%	56%		56%	33%	100%	89%	78%	89%	100%	67%	100%	100%
Xelnaga	45.00%	0%	0%	22%	11%	0%	33%	33%	56%	44%	33%	44%		44%	0%	100%	100%	67%	67%	56%	89%	100%
Skynet	62.78%	89%	11%	22%	67%	22%	11%	56%	22%	22%	78%	67%	56%		100%	33%	100%	100%	100%	100%	100%	100%
GarmBot	46.11%	0%	0%	0%	11%	22%	22%	0%	22%	67%	67%	0%	100%	0%		89%	78%	100%	100%	44%	100%	100%
NUSBot	29.44%	0%	0%	0%	0%	11%	0%	0%	89%	22%	0%	11%	0%	67%	11%		78%	11%	100%	78%	11%	100%
TerranUAB	28.89%	0%	0%	0%	0%	0%	0%	0%	11%	0%	0%	22%	0%	0%	22%	22%		100%	100%	100%	100%	100%
SRbotOne	25.56%	0%	0%	0%	0%	11%	22%	11%	0%	22%	0%	11%	33%	0%	0%	89%	0%		33%	89%	89%	100%
Cimex	17.78%	0%	0%	0%	0%	44%	0%	11%	0%	11%	0%	0%	33%	0%	0%	0%	0%	67%		33%	56%	100%
Oritaka	25.56%	0%	0%	0%	11%	0%	56%	0%	11%	11%	0%	33%	44%	0%	56%	22%	0%	11%	67%		89%	100%
CruzBot	15.00%	0%	0%	0%	0%	0%	22%	0%	0%	11%	0%	0%	11%	0%	0%	89%	0%	11%	44%	11%		100%
Tyr	0.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%

(4)Circuit Breaker	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	85.00%		44%	100%	89%	89%	100%	67%	100%	67%	89%	78%	100%	11%	100%	78%	100%	89%	100%	100%	100%	100%
ZZZKBot	83.33%	56%		67%	44%	56%	89%	44%	100%	89%	78%	44%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	79.44%	0%	33%		22%	33%	100%	100%	89%	78%	100%	67%	89%	78%	100%	100%	100%	100%	100%	100%	100%	100%
LetaBot	81.67%	11%	56%	78%		67%	89%	100%	33%	89%	100%	100%	78%	33%	100%	100%	100%	100%	100%	100%	100%	100%
UAlbertaBot	70.56%	11%	44%	67%	33%		89%	22%	89%	22%	78%	78%	100%	78%	67%	89%	100%	100%	44%	100%	100%	100%
Ximp	56.67%	0%	11%	0%	11%	11%		33%	56%	44%	0%	100%	44%	100%	89%	100%	89%	67%	78%	100%	100%	100%
Overkill	62.78%	33%	56%	0%	0%	78%	67%		0%	11%	78%	67%	11%	56%	100%	100%	100%	100%	100%	100%	100%	100%
Aiur	60.56%	0%	0%	11%	67%	11%	44%	100%		44%	89%	22%	33%	89%	100%	44%	89%	89%	100%	100%	78%	100%
MegaBot	64.44%	33%	11%	22%	11%	78%	56%	89%	56%		67%	56%	33%	67%	67%	89%	100%	67%	89%	100%	100%	100%
IceBot	53.33%	11%	22%	0%	0%	22%	100%	22%	11%	33%		44%	33%	33%	33%	100%	100%	100%	100%	100%	100%	100%
JiaBot	56.67%	22%	56%	33%	0%	22%	0%	33%	78%	44%	56%		22%	22%	89%	100%	89%	100%	100%	78%	89%	100%
Xelnaga	58.33%	0%	0%	11%	22%	0%	56%	89%	67%	67%	67%	78%		22%	0%	100%	100%	89%	100%	100%	100%	100%
Skynet	62.22%	89%	0%	22%	67%	22%	0%	44%	11%	33%	67%	78%	78%		100%	33%	100%	100%	100%	100%	100%	100%
GarmBot	42.78%	0%	0%	0%	0%	33%	11%	0%	0%	33%	67%	11%	100%	0%		100%	89%	100%	100%	11%	100%	100%
NUSBot	31.67%	22%	0%	0%	0%	11%	0%	0%	56%	11%	0%	0%	0%	67%	0%		67%	67%	56%	100%	78%	100%
TerranUAB	27.22%	0%	0%	0%	0%	0%	11%	0%	11%	0%	0%	11%	0%	0%	11%	33%		89%	78%	100%	100%	100%
SRbotOne	21.67%	11%	0%	0%	0%	0%	33%	0%	11%	33%	0%	0%	11%	0%	0%	33%	11%		33%	89%	67%	100%
Cimex	19.44%	0%	0%	0%	0%	56%	22%	0%	0%	11%	0%	0%	0%	0%	0%	44%	22%	67%		22%	44%	100%
Oritaka	17.78%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	22%	0%	0%	89%	0%	0%	11%	78%		56%	100%
CruzBot	14.44%	0%	0%	0%	0%	0%	0%	0%	22%	0%	0%	11%	0%	0%	0%	22%	0%	33%	56%	44%		100%
Tyr	0.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%

(4)Empire of the Sun	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	86.11%		0%	100%	89%	67%	100%	89%	100%	100%	100%	89%	89%	11%	100%	89%	100%	100%	100%	100%	100%	100%
ZZZKBot	85.00%	100%		22%	67%	67%	89%	33%	89%	89%	89%	56%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	81.67%	0%	78%		44%	0%	100%	100%	78%	100%	100%	89%	67%	89%	100%	100%	100%	100%	100%	89%	100%	100%
LetaBot	72.78%	11%	33%	56%		44%	78%	100%	56%	67%	89%	100%	22%	22%	100%	78%	100%	100%	100%	100%	100%	100%
UAlbertaBot	76.11%	33%	33%	100%	56%		100%	78%	100%	44%	78%	67%	100%	78%	67%	56%	100%	100%	33%	100%	100%	100%
Ximp	63.89%	0%	11%	0%	22%	0%		56%	89%	56%	0%	89%	56%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Overkill	58.33%	11%	67%	0%	0%	22%	44%		11%	33%	56%	56%	11%	78%	89%	100%	89%	100%	100%	100%	100%	100%
Aiur	56.67%	0%	11%	22%	44%	0%	11%	89%		56%	78%	56%	22%	89%	89%	22%	89%	100%	67%	89%	100%	100%
MegaBot	51.67%	0%	11%	0%	33%	56%	44%	67%	44%		56%	22%	22%	78%	11%	100%	89%	78%	67%	67%	89%	100%
IceBot	53.33%	0%	11%	0%	11%	22%	100%	44%	22%	44%		33%	11%	22%	44%	100%	100%	100%	100%	100%	100%	100%
JiaBot	60.56%	11%	44%	11%	0%	33%	11%	44%	44%	78%	67%		67%	67%	100%	100%	100%	78%	100%	67%	89%	100%
Xelnaga	63.33%	11%	0%	33%	78%	0%	44%	89%	78%	78%	89%	33%		22%	11%	100%	100%	100%	100%	100%	100%	100%
Skynet	59.44%	89%	0%	11%	78%	22%	0%	22%	11%	22%	78%	33%	78%		100%	44%	100%	100%	100%	100%	100%	100%
GarmBot	43.89%	0%	0%	0%	0%	33%	0%	11%	11%	89%	56%	0%	89%	0%		89%	89%	100%	89%	22%	100%	100%
NUSBot	27.78%	11%	0%	0%	22%	44%	0%	0%	78%	0%	0%	0%	0%	56%	11%		44%	56%	44%	78%	11%	100%
TerranUAB	28.33%	0%	0%	0%	0%	0%	0%	11%	11%	11%	0%	0%	0%	0%	11%	56%		67%	100%	100%	100%	100%
SRbotOne	21.11%	0%	0%	0%	0%	0%	0%	0%	0%	22%	0%	22%	0%	0%	0%	44%	33%		67%	89%	44%	100%
Cimex	21.67%	0%	0%	0%	0%	67%	0%	0%	33%	33%	0%	0%	0%	0%	11%	56%	0%	33%		56%	44%	100%
Oritaka	21.11%	0%	0%	11%	0%	0%	0%	0%	11%	33%	0%	33%	0%	0%	78%	22%	0%	11%	44%		78%	100%
CruzBot	17.22%	0%	0%	0%	0%	0%	0%	0%	0%	11%	0%	11%	0%	0%	0%	89%	0%	56%	56%	22%		100%
Tyr	0.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%

(4)Fortress	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	81.11%		0%	33%	100%	78%	100%	78%	89%	56%	89%	67%	100%	67%	100%	89%	100%	100%	78%	100%	100%	100%
ZZZKBot	88.33%	100%		56%	100%	67%	100%	22%	100%	89%	100%	44%	100%	89%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	86.11%	67%	44%		89%	33%	89%	100%	78%	78%	100%	89%	89%	78%	100%	89%	100%	100%	100%	100%	100%	100%
LetaBot	39.44%	0%	0%	11%		78%	0%	44%	0%	22%	0%	0%	22%	0%	56%	78%	22%	89%	89%	78%	100%	100%
UAlbertaBot	75.00%	22%	33%	67%	22%		100%	78%	89%	33%	78%	89%	100%	78%	100%	78%	100%	78%	56%	100%	100%	100%
Ximp	63.89%	0%	0%	11%	100%	0%		22%	89%	44%	11%	100%	56%	89%	89%	78%	100%	100%	100%	89%	100%	100%
Overkill	68.89%	22%	78%	0%	56%	22%	78%		78%	22%	89%	22%	22%	89%	100%	100%	100%	100%	100%	100%	100%	100%
Aiur	55.56%	11%	0%	22%	100%	11%	11%	22%		22%	78%	56%	33%	67%	78%	44%	89%	89%	100%	100%	78%	100%
MegaBot	70.00%	44%	11%	22%	78%	67%	56%	78%	78%		56%	33%	67%	89%	78%	89%	89%	89%	89%	89%	100%	100%
IceBot	61.11%	11%	0%	0%	100%	22%	89%	11%	22%	44%		56%	33%	44%	89%	100%	100%	100%	100%	100%	100%	100%
JiaBot	62.78%	33%	56%	11%	100%	11%	0%	78%	44%	67%	44%		44%	33%	100%	78%	89%	67%	100%	100%	100%	100%
Xelnaga	56.67%	0%	0%	11%	78%	0%	44%	78%	67%	33%	67%	56%		22%	0%	89%	100%	89%	100%	100%	100%	100%
Skynet	59.44%	33%	11%	22%	100%	22%	11%	11%	33%	11%	56%	67%	78%		100%	33%	100%	100%	100%	100%	100%	100%
GarmBot	39.44%	0%	0%	0%	44%	0%	11%	0%	22%	22%	11%	0%	100%	0%		56%	67%	100%	56%	100%	100%	100%
NUSBot	33.89%	11%	0%	11%	22%	22%	22%	0%	56%	11%	0%	22%	11%	67%	44%		67%	22%	56%	100%	33%	100%
TerranUAB	29.44%	0%	0%	0%	78%	0%	0%	0%	11%	11%	0%	11%	0%	0%	33%	33%		89%	56%	67%	100%	100%
SRbotOne	23.89%	0%	0%	0%	11%	22%	0%	0%	11%	11%	0%	33%	11%	0%	0%	78%	11%		56%	67%	67%	100%
Cimex	23.89%	22%	0%	0%	11%	44%	0%	0%	0%	11%	0%	0%	0%	0%	44%	44%	44%	44%		100%	22%	89%
Oritaka	11.67%	0%	0%	0%	22%	0%	11%	0%	0%	11%	0%	0%	0%	0%	0%	0%	33%	33%	0%		22%	100%
CruzBot	18.89%	0%	0%	0%	0%	0%	0%	0%	22%	0%	0%	0%	0%	0%	0%	67%	0%	33%	78%	78%		100%
Tyr	0.56%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	11%	0%	0%

(4)Python	overall	Iron	ZZZK	tscm	Leta	UAlb	Ximp	Over	Aiur	Mega	IceB	JiaB	Xeln	Skyn	Garm	NUSB	Terr	SRbo	Cime	Orit	Cruz	Tyr
Iron	86.11%		0%	100%	100%	100%	100%	89%	78%	67%	100%	78%	89%	22%	100%	100%	100%	100%	100%	100%	100%	100%
ZZZKBot	90.56%	100%		67%	78%	78%	100%	56%	100%	100%	100%	44%	100%	100%	100%	100%	100%	89%	100%	100%	100%	100%
tscmoo	78.89%	0%	33%		56%	22%	89%	89%	78%	89%	100%	89%	100%	89%	100%	100%	100%	100%	89%	56%	100%	100%
LetaBot	71.67%	0%	22%	44%		44%	100%	100%	22%	44%	89%	100%	22%	56%	100%	89%	100%	100%	100%	100%	100%	100%
UAlbertaBot	72.78%	0%	22%	78%	56%		67%	44%	56%	78%	89%	56%	100%	56%	78%	100%	100%	89%	89%	100%	100%	100%
Ximp	64.44%	0%	0%	11%	0%	33%		56%	100%	67%	11%	100%	11%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Overkill	62.22%	11%	44%	11%	0%	56%	44%		56%	56%	89%	44%	0%	67%	100%	78%	100%	100%	89%	100%	100%	100%
Aiur	56.11%	22%	0%	22%	78%	44%	0%	44%		33%	89%	56%	33%	56%	89%	22%	89%	100%	89%	67%	89%	100%
MegaBot	56.11%	33%	0%	11%	56%	22%	33%	44%	67%		33%	56%	33%	100%	56%	78%	89%	44%	89%	89%	89%	100%
IceBot	55.56%	0%	0%	0%	11%	11%	89%	11%	11%	67%		89%	56%	22%	44%	100%	100%	100%	100%	100%	100%	100%
JiaBot	56.11%	22%	56%	11%	0%	44%	0%	56%	44%	44%	11%		33%	44%	100%	100%	100%	78%	89%	89%	100%	100%
Xelnaga	63.89%	11%	0%	0%	78%	0%	89%	100%	67%	67%	44%	67%		56%	0%	100%	100%	100%	100%	100%	100%	100%
Skynet	58.33%	78%	0%	11%	44%	44%	0%	33%	44%	0%	78%	56%	44%		100%	33%	100%	100%	100%	100%	100%	100%
GarmBot	42.22%	0%	0%	0%	0%	22%	0%	0%	11%	44%	56%	0%	100%	0%		100%	89%	100%	89%	33%	100%	100%
NUSBot	28.89%	0%	0%	0%	11%	0%	0%	22%	78%	22%	0%	0%	0%	67%	0%		89%	0%	67%	100%	22%	100%
TerranUAB	23.33%	0%	0%	0%	0%	0%	0%	0%	11%	11%	0%	0%	0%	0%	11%	11%		89%	56%	89%	100%	89%
SRbotOne	26.11%	0%	11%	0%	0%	11%	0%	0%	0%	56%	0%	22%	0%	0%	0%	100%	11%		44%	100%	67%	100%
Cimex	20.00%	0%	0%	11%	0%	11%	0%	11%	11%	11%	0%	11%	0%	0%	11%	33%	44%	56%		56%	33%	100%
Oritaka	20.00%	0%	0%	44%	0%	0%	0%	0%	33%	11%	0%	11%	0%	0%	67%	0%	11%	0%	44%		78%	100%
CruzBot	16.11%	0%	0%	0%	0%	0%	0%	0%	11%	11%	0%	0%	0%	0%	0%	78%	0%	33%	67%	22%		100%
Tyr	0.56%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	11%	0%	0%	0%	0%

I could look at these charts all day and keep finding stuff to think about. Winner Iron was upset by ZZZKBot and scored overwhelmingly against every other bot except the two dragoon-heavy protosses MegaBot and Skynet, which earned upsets on some maps. ZZZKBot’s results fall with rush distance, closely in line with Martin Rooijacker’s explanation. LetaBot did well on most maps, but struggled on Fortress. NUSBot apparently crashed on Heartbreak Ridge, giving Tyr nearly half of its wins. Weak bot Cimex did surprisingly well against strong UAlbertaBot on every map except Tau Cross and Python. Why is that?

AIIDE 2016 Bayesian Elo ratings

Again I have Elo as calculated by Remi Coulom’s bayeselo program. The # column gives the official ranking, so you can see how it differs from the rank by Elo (the bayeselo ranking is slightly more accurate because it takes into account all the information in the tournament results, not only the raw winning rate). I left out the 95% confidence interval column as relatively uninteresting, since the “better?” column tells us how likely each bot is to be superior to the one below it.

#	bot	score	Elo	better?
1	Iron	87%	2016	99.4%
2	ZZZKBot	85%	1974	99.6%
3	tscmoo zerg	83%	1932	99.9%
4	LetaBot	74%	1815	99.8%
5	UAlbertaBot	70%	1774	99.9%
6	Ximp	65%	1699	99.6%
8	Aiur	61%	1663	51.6%
7	Overkill	62%	1663	99.6%
9	MegaBot	58%	1627	88.5%
10	IceBot	57%	1611	57.5%
12	Xelnaga	57%	1608	50.0%
11	JiaBot	57%	1608	98.1%
13	Skynet	55%	1581	100%
14	GarmBot	43%	1441	100%
16	TerranUAB	27%	1250	74.7%
15	NUSBot	27%	1240	99.9%
17	SRbotOne	22%	1167	99.0%
18	Cimex	21%	1130	92.6%
19	Oritaka	20%	1106	99.3%
20	CruzBot	17%	1064	100%
21	Tyr	1%	533	-

There are some switches from the official ranking, due to bots being statistically indistinguishable. Overkill and Aiur are in a dead heat. IceBot (terran), Xelnaga (protoss) and JiaBot (zerg) are also virtually even. bayeselo gives IceBot a 57.6% chance of being better than JiaBot two ranks down, essentially the same as its 57.5% chance of being better than Xelnaga one rank down.

Tomorrow: The per-map crosstables.

AIIDE 2016 results discussion

The AIIDE 2016 results are out. The top finishers, in order, are Iron, ZZZKBot, Tscmoo zerg, Letabot—half terran and half zerg. As usual, Martin Rooijackers sent me a few observations. Here are my thoughts so far.

• The tournament was played on the same maps as last year. It doesn’t call for that in the rules. We know that they’re OK maps, though.

• Martin Rooijackers says that all four top finishers were updated in between the CIG 2016 entry deadline and the AIIDE deadline a month later. [This turned out to be wrong. See the comments. I misread Martin Rooijacker’s message—he wrote that #1 Iron and #2 ZZZKBot had been updated.] You need continual work to stay at the top. It’s fierce up there.

• In particular, ZZZKBot moved up to #2, with minus scores only against fellow zergs Overkill and JiaBot. I’m curious to know what the improvements are.

• Tscmoo zerg scored even against ZZZKBot (a poor performance; zerg should easily beat the 4 pool), and worse than even against the rest of the top 5. Tscmoo owed its #3 finish to terrorizing its inferiors. Martin Rooijackers took this to mean that terran dominated. But with 2 of the top 4, I have to say that zerg is hanging in there.

• In last year’s AIIDE, UAlbertaBot and Overkill were neck-and-neck. The Bayesian Elo calculation said that UAlbertaBot was about 60% likely to be stronger, a narrow margin. Neither has been updated since. [This was also wrong! See comments.] In CIG 2016, Overkill was ahead. In this tournament, UAlbertaBot finished well ahead. These two can’t make up their minds! Apparently this time UAlbertaBot with its rush builds was better able to cope with the changing field.

• XIMP made it to #6. On SSCAIT its last update was February 2015, and I believe it was a minor adjustment to timings. The carrier strategy seems resilient. By contrast IceBot, AIIDE and CIG champion in 2014 ahead of XIMP, was in the middle of the pack. All its smarts did not make it resilient against improvements in its enemies.

• “JiaBot” in AIIDE is probably the same as “Ziabot” in CIG and “Zia bot” in SSCAIT. In some languages, the same sound can transliterate into English as either J or Z. Looking at the Ziabot source from CIG 2016, I think that the author is Korean based on variable names.

• Xelnaga finished ahead of Skynet, which surprised me. In CIG, XelnagaII was near the bottom. It seems a surprisingly sudden change.

• Tyr finished dead last by a mile, with a 1% win rate which seems so low that it must be due to bugs. Tyr had major updates this year which left it fairly strong. It finished in the middle of the pack in CIG 2016. I imagine it was updated at the last minute with a grievous error.

Tomorrow: The Elo table. I’ve done the calculations, but I have to draw up the table.

Rob Bogie’s MaasCraft has map problems

What is it with Rob Bogie’s SSCAIT version of MaasCraft (originally written by Dennis Soemers)? Why does it do so much better on some maps and so much worse on others, with far more variation than the rest of the bots?

I looked up MaasCraft’s results in the AIIDE 2014 and CIG 2014 competitions (it didn’t play in other years). Both map pools overlap with the SSCAIT map pool. CIG 2014 had an extra wide and diverse map pool, with some nonstandard maps, and it only overlaps a little. The overall row is across all maps in that competition, not only the ones in the table.

map	CIG 2014 win %	AIIDE 2014 win %	SSCAIT Elo diff
Andromeda		63%	-361
Benzene		55%	+42
Circuit Breaker		52%	+246
Destination		59%	-313
Empire of the Sun		61%	-333
Fighting Spirit	50%		-418
Heartbreak Ridge		61%	-193
Icarus	50%		+291
Python		60%	-306
Tau Cross		57%	+365
overall	55%	59%	0

I’m comparing raw win percentages with Elo differences, but the pattern is clear—I mean, the lack of pattern. The 2014 tournament results look normal, the SSCAIT results look extreme, and the two are nothing alike.

CIG 2014 and AIIDE 2014 both released source code. The MaasCraft source is exactly the same in both. It looks as though some difference or bug is affecting only the SSCAIT version under Rob Bogie.

Later today: First thoughts on the AIIDE 2016 results.

how bots like maps

I decided to analyze the maps to see how bots feel about them overall. This data is derived from yesterday’s big table of how much bots like each SSCAIT map. The “spread” column is the mean of the absolute value of the Elo deviation numbers for a given map, across all the bots. I thought of calling it “controversy”; it measures how much bots like or dislike the map. Maps that all bots do OK on get low numbers; maps that some bots love and others hate get high numbers.

The “RMS” column is the root mean square of the same data. Statistically, it’s a fairer measure of the differences. It’s bigger because it puts more weight on outliers. The two measures don’t agree closely.

Destination is the most “controversial” map, with 60 Elo spread. If you pick one bot that likes Destination and one bot that dislikes it, on average the bot that likes it will have a 60 Elo advantage, which means a 59% win rate if the bots are otherwise even—nothing devastating. Neo Moon Glaive has Elo spread 41 or about 56% advantage, not much different. Even if you go with the RMS number, the peak 81 Elo RMS difference means a 61% win rate, still not much different.

map	spread	RMS
Benzene	45	57
Destination	60	78
HeartbreakRidge	53	81
NeoMoonGlaive	41	53
TauCross	51	74
Andromeda	49	70
CircuitBreaker	54	69
EmpireoftheSun	50	69
FightingSpirit	51	72
Icarus	46	60
Jade	50	64
LaMancha1.1	49	65
Python	47	60
Roadrunner	46	63

Bottom line: On this analysis, the maps don’t seem to be distorting the competition. No highly “controversial” maps are introducing widespread unfairness.

Elo rating variations by map

From the SSCAIT data, I calculated the Elo advantage or disadvantage that each bot sees on each map. If it played all its games on that map, its Elo would change by that much. More or less; there isn’t as much data, so the advantage numbers are less accurate than the original Elo. I increased the Elo K factor to account for the smaller amount of data.

The 14 active maps:

(2)Benzene.scx
(2)Destination.scx
(2)HeartbreakRidge.scx
(3)NeoMoonGlaive.scx
(3)TauCross.scx
(4)Andromeda.scx
(4)CircuitBreaker.scx
(4)EmpireoftheSun.scm
(4)FightingSpirit.scx
(4)Icarus.scm
(4)Jade.scx
(4)LaMancha1.1.scx
(4)Python.scx
(4)Roadrunner.scx

The Elo ratings are repeated from the original post. I dropped 5 bots of the 103 for lack of data. The top number in each colored cell is the advantage or disadvantage that bot sees when playing on that map, in Elo points. You can look up winning rates for a given advantage in the Elo table. The bottom number is the count of games. Some bots have few games, like the new ZerGreenBot. A few bots have not played on every map, and get “-” instead of numbers.

bot	Elo	Benz	Dest	Hear	NeoM	TauC	Andr	Circ	Empi	Figh	Icar	Jade	LaMa	Pyth	Road	earliest	latest
krasi0	2163 2128	53 164	57 169	26 146	7 159	18 143	26 126	-76 144	-35 158	-73 140	-31 158	23 156	-63 155	-6 156	72 154	2015 Nov 30	2016 Sep 27
Iron bot	2081 1990	139 141	-15 144	47 134	-35 157	53 140	-34 133	-79 129	-36 142	17 145	-12 142	17 148	1 150	-78 137	15 148	2015 Nov 27	2016 Sep 26
Marian Devecka	2065 4117	-57 320	-18 255	70 303	-12 287	-37 322	104 273	45 285	-22 297	23 301	-41 289	-72 309	28 310	26 284	-38 282	2014 Oct 29	2016 Sep 27
Martin Rooijackers	2011 6449	-25 462	56 473	-67 478	0 450	40 477	44 449	88 478	-37 458	-15 463	42 480	-156 446	0 429	31 475	-2 431	2014 Oct 29	2016 Sep 27
tscmooz	1991 4972	-23 380	120 354	-20 323	-52 316	50 354	-50 393	21 354	-9 341	-78 370	43 364	-6 354	29 377	-54 328	29 364	2015 Feb 27	2016 Sep 27
tscmoo	1978 5682	-8 359	44 355	-15 438	39 389	27 445	30 402	12 447	-14 420	-97 410	53 408	-70 385	29 397	41 396	-71 431	2015 Jan 22	2016 Sep 27
LetaBot CIG 2016	1932 444	120 26	33 38	-126 35	5 29	-81 30	40 30	-12 33	-64 26	100 23	2 29	-92 35	14 37	35 45	27 28	2016 Aug 01	2016 Sep 27
WuliBot	1871 984	-9 73	-58 83	66 56	77 77	-32 68	14 59	-22 71	36 84	12 64	-21 64	54 80	-37 67	-55 68	-26 70	2016 Apr 19	2016 Sep 26
Simon Prins	1867 5400	-20 388	-25 410	51 381	67 432	19 356	-18 412	-110 374	97 376	-21 396	-25 385	95 375	-54 324	17 373	-73 418	2015 Jan 25	2016 Sep 27
ICELab	1865 6078	-21 441	-69 442	73 458	-127 377	-52 447	-17 398	-2 455	3 440	-13 425	-25 471	49 421	34 437	72 435	96 431	2014 Oct 29	2016 Sep 27
FlashTest	1863 204	-71 13	-106 12	-117 13	149 22	94 18	31 14	-121 15	2 20	78 15	-57 10	79 13	59 13	4 12	-23 14	2016 Mar 22	2016 Jul 27
Sijia Xu	1849 2313	22 171	30 155	19 182	-8 166	-2 148	-49 164	19 166	-25 165	-45 138	24 171	36 161	-72 153	51 201	0 172	2015 Oct 10	2016 Sep 27
LetaBot SSCAI 2015 Final	1813 416	-10 28	83 31	-41 22	31 34	15 27	-223 37	199 28	-44 29	-96 33	-44 36	-32 27	1 29	79 25	81 30	2016 Aug 04	2016 Sep 27
Dave Churchill	1804 6023	-29 473	200 428	-82 436	18 413	19 433	-74 412	-50 417	72 429	47 445	-68 396	-53 408	100 438	-46 455	-53 440	2014 Oct 29	2016 Sep 27
Chris Coxe	1800 2195	31 153	188 169	66 153	-106 165	-25 149	-79 149	-34 166	-7 167	-18 153	115 146	-124 153	-114 154	114 156	-8 162	2015 Sep 03	2016 Sep 27
Tomas Vajda	1790 6088	-2 441	9 439	21 441	-77 449	-14 443	-18 421	34 398	-7 458	55 422	17 425	61 440	-80 424	-6 439	7 448	2014 Oct 29	2016 Sep 27
Flash	1777 991	-19 70	-163 65	16 59	7 68	-70 59	-12 85	3 76	72 69	23 69	64 87	71 84	-25 61	30 75	3 64	2016 Apr 18	2016 Sep 27
LetaBot IM noMCTS	1766 1226	61 79	127 88	-116 83	39 106	85 87	-114 93	-106 89	-24 93	-28 89	58 93	-66 70	18 86	-2 92	68 78	2016 May 18	2016 Aug 01
Zia bot	1757 536	54 36	-63 40	15 39	-22 41	50 39	-63 43	93 42	-7 34	20 36	-78 33	-58 30	-55 40	66 39	48 44	2016 Jul 07	2016 Sep 27
A Jarocki	1741 932	20 63	121 66	125 62	-42 72	-48 74	18 53	-104 78	74 81	25 76	-93 59	-110 64	-39 55	52 67	3 62	2015 Oct 04	2016 Jan 26
PeregrineBot	1728 1262	-21 104	94 89	-28 95	123 76	-133 90	77 95	-4 77	10 85	-67 94	-32 83	-51 94	11 90	14 103	9 87	2016 Feb 09	2016 Sep 10
tscmoop	1721 1982	155 139	56 145	47 140	50 127	-7 130	53 154	-68 127	13 141	-17 140	-54 171	-59 128	-44 158	-51 148	-74 134	2015 Nov 11	2016 Sep 26
Andrew Smith	1718 6160	63 460	-37 387	46 445	-10 457	-11 452	33 449	-21 463	89 480	22 441	-58 455	-25 445	-29 418	-58 400	-3 408	2014 Oct 29	2016 Sep 27
Florian Richoux	1716 5970	-35 408	-107 425	120 441	42 411	-43 446	-46 443	37 397	152 468	66 416	-25 451	46 433	-90 410	-29 391	-88 430	2014 Oct 29	2016 Sep 27
Carsten Nielsen	1695 4683	-19 361	27 341	-65 334	33 315	-21 330	-1 319	-27 353	53 336	88 369	-35 318	6 318	-78 282	26 340	12 367	2015 Mar 17	2016 Sep 27
Soeren Klett	1687 6002	39 459	35 426	-12 420	63 396	-108 440	39 481	-20 402	-25 451	-4 454	36 433	-63 407	-19 406	71 410	-32 417	2014 Oct 29	2016 Sep 27
Vaclav Horazny	1686 4140	33 291	78 291	-17 302	18 295	-22 274	-46 332	33 290	-31 308	4 284	-38 270	58 313	-96 315	25 296	2 279	2014 Oct 29	2015 Nov 18
La Nuee	1662 558	-66 36	41 24	81 40	100 35	57 44	-11 41	-82 44	-21 36	50 52	-7 34	-78 43	-11 36	68 55	-121 38	2015 Dec 13	2016 Mar 18
Jakub Trancik	1657 6136	102 427	79 454	1 434	-32 443	-64 445	-10 459	-16 438	-106 424	18 439	-35 413	-23 404	-40 463	63 445	63 448	2014 Oct 29	2016 Sep 27
Marek Suppa	1655 4397	61 322	78 334	1 330	14 321	-48 304	-59 328	-88 315	-6 291	25 336	56 326	6 292	-25 300	-41 302	25 296	2015 Jan 05	2016 Mar 18
Krasimir Krystev	1653 4292	-24 322	-59 283	92 304	49 298	8 327	-56 295	-51 296	-43 318	-76 326	-9 318	-24 280	132 318	-58 299	117 308	2014 Oct 29	2016 Mar 10
ASPbot2011	1652 222	-22 21	-178 17	-27 12	-13 17	98 20	154 11	-29 13	87 17	17 10	75 14	-59 17	-135 15	21 18	11 20	2015 Jan 29	2016 Feb 25
Marcin Bartnicki	1633 1377	59 97	43 106	63 88	-54 110	-33 109	-20 110	-21 92	-46 96	-20 97	-12 92	53 92	-38 99	75 102	-48 87	2014 Nov 28	2016 Mar 18
Tomas Cere	1631 6131	-27 446	-41 419	-2 444	-46 443	109 459	14 448	2 429	2 424	48 433	-24 442	-69 417	49 480	-12 426	-2 421	2014 Oct 29	2016 Sep 27
MegaBot	1630 419	-22 29	27 27	6 34	36 27	-27 26	-21 25	-28 21	-1 28	44 41	64 28	-18 37	-2 37	-66 34	9 25	2016 Aug 01	2016 Sep 27
Aurelien Lermant	1622 3673	-30 258	-8 249	15 268	-32 247	30 260	-37 287	-46 250	-73 266	87 262	38 253	51 255	80 285	-20 261	-53 272	2015 Jun 22	2016 Sep 27
Matej Kravjar	1619 983	-29 69	22 67	-96 75	16 71	-266 75	12 65	78 73	74 79	-25 59	20 75	82 75	68 58	105 78	-60 64	2014 Oct 29	2015 Feb 18
Daniel Blackburn	1605 4591	-61 327	-83 332	74 337	8 326	-8 306	66 330	-56 354	-26 350	74 327	-23 312	17 311	-3 319	-10 345	31 315	2014 Oct 29	2016 Jan 26
Gabriel Synnaeve	1584 18	-3 1	-38 3	31 1	24 1	-26 2	-	-	19 1	-	-5 2	13 1	13 2	-9 2	-19 2	2015 Jan 30	2015 Nov 24
David Milec	1566 49	-	-11 5	48 2	-47 4	-73 4	9 4	-2 3	-26 6	-13 2	16 4	25 3	31 5	70 5	-26 2	2015 Jan 13	2015 Jan 20
Odin2014	1565 5602	-60 416	-40 377	16 396	-40 399	-38 400	42 402	-45 380	-68 401	-29 424	58 416	52 393	66 411	14 365	73 422	2014 Dec 21	2016 Sep 11
Gaoyuan Chen	1559 5106	-18 387	-40 348	-20 357	-64 387	59 379	48 352	-18 330	75 359	2 391	-24 362	-53 383	33 371	18 345	1 355	2015 Feb 10	2016 Sep 27
Henri Kumpulainen	1553 881	62 75	-128 64	37 68	51 64	-23 69	33 69	-48 58	15 67	-36 62	22 74	27 48	1 58	-11 54	-1 51	2016 Jan 13	2016 May 31
Martin Dekar	1533 2627	-58 195	14 186	25 189	5 196	-105 168	-27 189	73 178	-3 206	30 174	18 207	72 202	-9 166	-53 201	17 170	2014 Oct 29	2016 Jan 25
Serega	1505 3802	-35 280	104 257	131 262	51 260	-3 278	26 261	-23 275	-114 285	-76 276	6 270	-4 268	-1 279	-100 291	38 260	2015 Jan 31	2016 Jan 26
Chris Ayers	1481 1520	-34 115	-40 124	11 112	-28 106	0 106	-72 111	65 88	69 105	38 113	-52 117	49 109	-44 89	-44 102	82 123	2015 Aug 10	2016 Jan 26
Nathan a David	1481 991	13 57	0 61	-91 77	31 88	-7 65	-34 75	-23 72	103 61	68 65	43 72	-26 78	-110 70	-34 64	68 86	2016 Feb 23	2016 Aug 08
DAIDOES	1471 485	131 36	39 32	47 31	-27 30	1 28	-25 42	123 39	-159 28	29 44	32 34	-64 34	-80 36	-51 30	5 41	2016 Jun 13	2016 Sep 08
Igor Lacik	1454 5852	19 420	48 399	13 447	-28 408	-38 375	31 418	-95 461	111 418	-28 386	-9 442	12 415	-54 405	26 429	-7 429	2014 Oct 29	2016 Sep 08
Matej Istenik	1449 6054	-63 412	18 457	44 458	12 421	-20 429	-9 435	-109 417	-4 472	8 458	88 382	-3 414	8 426	-2 456	33 417	2014 Oct 29	2016 Sep 27
EradicatumXVR	1443 4539	-41 340	18 324	62 309	66 319	-14 322	-5 315	60 322	-90 303	-24 331	-39 330	11 340	24 311	19 347	-47 326	2014 Nov 04	2016 Jan 23
Tomasz Michalski	1432 433	6 29	34 30	-109 34	-35 23	142 27	30 40	-16 27	-92 31	174 32	-54 44	-5 23	12 31	-24 28	-63 34	2015 Dec 22	2016 Mar 18
Oleg Ostroumov	1431 1345	10 92	38 83	63 69	-33 116	74 96	22 98	60 96	-1 116	101 109	-82 80	-104 97	-89 90	80 102	-138 101	2014 Oct 29	2016 Jan 26
NUS Bot	1426 3333	78 216	-20 210	-525 233	64 236	29 249	25 221	-19 257	100 241	23 257	-39 254	133 240	104 232	36 257	11 230	2015 May 19	2016 Sep 06
Martin Pinter	1425 1580	53 114	-54 110	69 108	7 110	-44 122	0 123	8 101	77 108	-60 97	26 131	-31 139	61 112	-53 95	-59 110	2014 Oct 29	2015 Dec 11
Roman Danielis	1417 2945	-82 222	35 209	23 198	18 206	-9 202	13 221	89 202	-5 206	38 207	10 200	-18 225	-32 211	-29 226	-50 210	2014 Oct 29	2016 Sep 26
ZerGreenBot	1416 36	-	-21 3	-13 3	-6 2	13 3	42 2	-29 3	8 1	6 1	-73 5	45 4	-7 4	36 4	0 1	2016 Sep 22	2016 Sep 27
Marek Kadek	1413 5246	41 406	75 382	8 392	5 385	-28 383	-24 357	-12 369	-62 394	-49 359	-24 367	18 379	37 350	18 348	-3 375	2014 Oct 29	2016 May 22
Ian Nicholas DaCosta	1404 2928	-38 217	-31 210	55 214	29 213	-134 222	-62 192	5 193	10 218	-17 205	-18 213	73 233	-47 199	90 186	84 213	2015 Apr 27	2016 Sep 08
AwesomeBot	1403 473	-19 32	-24 31	-11 32	-23 49	20 29	149 29	-18 26	-3 30	9 53	-67 36	4 36	33 30	-12 32	-39 28	2016 Jun 16	2016 Sep 08
Radim Bobek	1390 1151	-85 70	11 96	-36 68	54 77	-39 94	-95 81	-11 78	0 89	184 87	43 91	61 70	-58 75	-72 100	44 75	2015 Oct 01	2016 Mar 06
Adrian Sternmuller	1375 4379	8 316	78 313	-85 330	-74 321	73 325	79 335	73 287	-42 315	-59 330	69 293	-115 288	33 331	-109 305	72 290	2014 Oct 30	2016 Jul 22
Martin Strapko	1366 1144	-55 98	-2 103	-21 82	12 65	-6 83	88 89	20 73	-51 76	23 72	69 80	-57 81	-108 80	50 81	38 81	2014 Oct 29	2016 Jan 26
Maja Nemsilajova	1363 4117	73 301	-73 292	73 322	7 309	31 298	-51 302	-77 269	-10 290	76 264	-36 302	-82 299	14 309	44 270	10 290	2014 Nov 04	2015 Nov 29
Johan Kayser	1361 413	-58 24	153 19	22 33	-50 29	-2 29	27 36	-156 19	-150 28	-59 40	66 28	21 27	128 28	26 42	34 31	2016 Jul 29	2016 Sep 27
UPStarcraftAI	1360 600	-16 43	56 48	6 41	45 48	11 36	12 43	-76 56	22 40	-28 44	52 37	37 43	23 32	-47 40	-98 49	2015 Dec 24	2016 Apr 13
Martin Vlcak	1353 1210	-49 68	-83 97	-51 81	-26 86	-25 98	-31 87	88 60	119 94	35 79	-24 102	16 90	45 90	33 83	-46 95	2016 Feb 16	2016 Sep 07
Johannes Holzfuss	1351 674	-46 52	98 39	24 36	59 51	-42 47	-27 54	-73 57	92 64	-82 49	-6 33	-4 57	-82 52	80 44	8 39	2016 Mar 05	2016 Jun 15
Vojtech Jirsa	1350 2759	-87 212	-79 182	32 200	-8 207	-127 205	6 174	49 184	83 211	27 199	-30 186	79 207	117 193	31 214	-93 185	2015 Jan 12	2015 Sep 05
JompaBot	1349 1043	-151 67	107 78	-180 70	-44 67	-52 83	81 61	95 88	1 83	71 75	82 80	-9 70	-72 77	68 73	2 71	2016 Feb 04	2016 Aug 13
Rob Bogie	1346 651	42 48	-313 54	-193 38	135 45	365 49	-361 47	246 38	-333 43	-418 52	291 55	273 45	298 43	-306 40	274 54	2016 May 14	2016 Sep 06
Christoffer Artmann	1344 395	30 25	-123 23	14 22	57 29	-45 31	-155 22	-36 27	-52 34	143 30	109 32	28 41	-17 31	-51 26	99 22	2016 Aug 07	2016 Sep 27
Marek Gajdos	1331 1370	2 91	-102 100	90 95	-13 102	-139 99	93 87	42 81	39 107	-78 107	-30 106	-7 97	9 94	79 101	15 103	2016 Jan 30	2016 Sep 11
Travis Shelton	1314 1212	38 72	-4 78	77 84	-31 80	-1 70	47 90	-25 105	2 92	-27 100	-30 104	-9 93	44 86	-87 82	5 76	2016 Feb 28	2016 Sep 06
Peter Dobsa	1307 3015	27 213	26 205	-45 218	-71 199	-19 215	82 228	-4 232	81 197	58 212	-28 224	-32 207	42 204	-65 237	-54 224	2015 Jan 11	2015 Oct 02
VeRLab	1304 888	-75 65	-5 52	-16 64	25 75	-20 56	-27 63	84 51	-6 79	42 71	57 77	99 51	6 52	-32 54	-131 78	2016 Feb 28	2016 Aug 01
Bjorn P Mattsson	1295 4432	18 303	75 328	48 340	-4 303	-20 307	39 302	6 333	-26 317	12 304	-58 315	-96 326	-50 345	113 280	-57 329	2015 Apr 05	2016 Sep 27
Lukas Sedlacek	1293 63	48 3	-80 10	74 3	-78 5	2 4	-14 6	-53 9	39 3	77 3	29 4	-15 2	7 3	-18 5	-20 3	2015 Jan 12	2015 Jan 20
Sergei Lebedinskij	1293 1083	-16 59	11 71	-19 77	69 77	47 72	-97 83	70 70	46 69	3 86	-85 85	55 71	36 106	-51 74	-68 83	2015 May 28	2015 Sep 03
Vladimir Jurenka	1278 6041	-87 429	-66 435	-33 454	-28 432	-15 402	11 438	44 453	77 435	-96 443	33 406	15 467	77 435	57 413	10 399	2014 Nov 04	2016 Sep 27
neverdieTRX	1272 334	65 29	-3 27	56 21	-9 26	-38 19	13 20	-82 27	-50 28	44 26	-159 27	67 21	35 25	-3 24	62 14	2016 Jul 19	2016 Sep 10
OpprimoBot	1256 1994	8 131	-23 138	14 144	122 146	70 143	-8 160	33 131	30 153	-70 135	-88 134	38 140	1 149	-93 139	-35 151	2015 Nov 18	2016 Sep 27
Marek Kruzliak	1255 399	-1 31	-92 35	66 23	-46 27	146 28	111 27	-123 23	112 25	-43 36	-99 32	80 26	15 30	-25 25	-101 31	2014 Nov 28	2015 Jan 20
Sungguk Cha	1250 697	-9 47	-38 46	34 54	-148 40	50 52	-62 51	130 48	-23 41	-3 65	-63 51	117 43	76 48	-44 61	-17 50	2016 Jun 05	2016 Sep 27
Jacob Knudsen	1247 1244	-13 79	-32 89	92 99	-20 89	36 81	34 75	70 87	-87 88	-77 81	-5 99	55 87	-7 103	3 94	-47 93	2016 Feb 23	2016 Sep 10
Ludmila Nemsilajova	1228 409	52 27	15 21	13 22	74 25	-90 27	-9 27	65 46	-71 37	-51 32	-75 28	1 30	-24 29	58 30	43 28	2014 Nov 28	2015 Jan 21
Karin Valisova	1226 1067	159 84	-42 86	-11 71	-2 72	-28 74	0 79	-62 68	-40 76	6 72	-16 66	4 68	-29 85	31 82	31 84	2014 Nov 04	2016 Jan 26
HoangPhuc	1209 300	-54 28	-46 32	25 22	3 16	83 21	-88 24	-21 16	51 20	-52 24	56 17	-70 25	-27 15	-37 25	178 15	2016 Jul 18	2016 Sep 07
Sebastian Mahr	1182 1191	-2 67	-61 89	-18 83	-34 96	14 71	118 74	95 87	25 81	25 79	-13 97	-63 94	-40 89	17 97	-61 87	2016 Jan 13	2016 Aug 08
Jan Pajan	1179 997	-15 85	58 71	14 64	-49 77	14 67	37 72	-61 78	2 61	-29 67	-19 70	-77 91	52 72	21 62	53 60	2014 Nov 04	2016 Jan 05
Pablo Garcia Sanchez	1174 579	26 33	11 53	-28 42	-74 33	3 45	35 39	84 34	-49 43	52 39	26 37	24 50	-21 46	-37 51	-51 34	2015 Dec 24	2016 Apr 13
Ivana Kellyerova	1131 1499	-59 115	-89 113	2 99	-20 113	71 111	38 108	-39 95	-6 99	19 125	30 92	53 106	-54 110	-5 97	60 116	2014 Nov 04	2015 Apr 01
Lucia Pivackova	1090 717	-69 50	-1 53	-32 55	20 50	94 41	-13 58	-25 47	23 42	-57 47	39 49	-42 55	-32 50	48 59	46 61	2014 Oct 30	2015 Jan 20
Tae Jun Oh	1036 138	43 11	21 8	-7 10	-40 7	96 8	-18 9	95 6	-35 9	-90 17	129 7	-40 14	-95 11	-21 9	-38 12	2016 Mar 22	2016 Apr 11
Denis Ivancik	1022 418	-65 37	46 30	-23 25	-34 26	109 29	78 21	-26 36	96 26	-90 43	4 28	-35 27	-28 18	-43 34	11 38	2014 Nov 28	2015 Jan 20
ButcherBoy	970 422	38 21	38 23	-68 32	-30 35	-43 34	100 31	46 29	-40 29	-6 35	-40 31	-49 34	128 23	7 30	-81 35	2016 Jun 21	2016 Sep 06
Jon W	964 790	-30 58	47 66	-100 62	6 59	83 45	27 67	4 52	-44 50	7 60	35 47	-9 57	79 49	-66 62	-39 56	2015 Apr 30	2015 Jul 09
Matyas Novy	885 1693	77 103	-76 132	-60 133	-69 122	-2 110	5 120	1 107	67 119	110 104	-59 145	-20 122	66 126	-83 124	44 126	2015 Feb 04	2015 Jul 09

There are some interesting things to see in the chart, but first look at Rob Bogie! That’s the bot MaasCraft. All the bots have preferences, some have strong preferences, but MaasCraft loves some maps and hates others. Why is that? If it could be made to love all the maps....

updates on ZerGreenBot and Bereaver

ZerGreenBot has been updated repeatedly, and now plays a richer game. I liked its PvZ best; it opens with a forge fast expand and follows up with corsairs, reavers, and zealots, a strategy that is somewhat similar to mainline human play.

Here are shots from a game on Benzene against Tscmoo zerg. ZerGreenBot opened pylon, forge, cannon, nexus, gateway, cannon. It looks to me like a sound wall-in in the correct place (a couple spots need blocking units to be tight). On maps that don’t allow a wall-in, like Destination, it necessarily builds something looser.

ZerGreenBot’s wall

ZerGreenBot’s first tech choice at the next level was stargate, starting air attack +1 before the stargate finished warping. Normal would be ground attack +1, but its choice seemed justified because the corsairs killed overlords throughout the game despite determined resistance. ZerGreenBot still tries its reaver drops versus zerg, but overlord hunting has become its top skill.

ZerGreenBot hunts overlords

Unfortunately ZerGreenBot’s ground units fought so poorly that their code must be buggy. Even the air units showed some stuck-in-a-loop behaviors. Tscmoo won a crushing ground victory in the center and easily took it home. If its author keeps up the good work, though, ZerGreenBot is a threat to reach the top ranks. It has reaver drop and overlord hunting skills beyond any other bot. I think it also has the best protoss wall.

Meanwhile, Bereaver is already a top bot. I watched it battle Tscmoo protoss on the 2-player map Destination. Bereaver made an ill-advised attack across the bridges into Tscmoo’s natural and lost too much army. Tscmoo should have consolidated its advantage by expanding and containing, but instead tried to cash it in immediately with its own attack across Bereaver’s bridges. It could have worked against a weaker bot, but Bereaver defended efficiently with cannons, a reaver, and a handful of troops. Tscmoo overpressed, fell behind, and lost. It was a decent win against a tough opponent.

Next: SSCAIT map comparisons.

SSCAIT top 10 crosstables

Krasi0 passed Marian Devecka’s Killerbot to become #1 on SSCAIT on 18 August. A 100 by 100 crosstable is too big, but here’s a crosstable of the games played among the top 10 SSCAIT bots from 17 August to 27 September. The top 10 are chosen based on their Elo ratings at the end of the period. The top number in each box is the winning rate of the bot in that row against the bot in that column; the bottom number is the count of games. The overall column is the winning rate against the other 9 top bots; it doesn’t have to be closely related to the Elo rating, which is computed with all 100 bots.

since 17 Aug	overall	kras	Iron	Mari	Mart	tscm	tscm	Leta	Wuli	Simo	ICEL
krasi0	66.67%		20% 10	57% 14	100% 17	0% 14	55% 11	100% 7	100% 9	100% 8	100% 9
Iron bot	71.28%	80% 10		78% 9	36% 11	89% 9	64% 14	40% 15	100% 9	90% 10	100% 7
Marian Devecka	65.38%	43% 14	22% 9		100% 6	86% 7	58% 12	82% 11	80% 5	50% 6	100% 8
Martin Rooijackers	56.82%	0% 17	64% 11	0% 6		64% 14	78% 9	70% 10	100% 8	75% 4	100% 9
tscmooz	54.43%	100% 14	11% 9	14% 7	36% 14		50% 8	38% 8	43% 7	100% 6	100% 6
tscmoo	48.75%	45% 11	36% 14	42% 12	22% 9	50% 8		50% 8	0% 4	100% 6	100% 8
LetaBot CIG 2016	55.06%	0% 7	60% 15	18% 11	30% 10	62% 8	50% 8		80% 10	100% 14	67% 6
WuliBot	19.12%	0% 9	0% 9	20% 5	0% 8	57% 7	100% 4	20% 10		0% 8	25% 8
Simon Prins	31.94%	0% 8	10% 10	50% 6	25% 4	0% 6	0% 6	0% 14	100% 8		100% 10
ICELab	11.27%	0% 9	0% 7	0% 8	0% 9	0% 6	0% 8	33% 6	75% 8	0% 10

It’s amazing that Tscmoo zerg scored 100% versus Krasi0 during the period. It’s equally amazing that former champion IceBot has a 0% score against 7 of the other 9 top bots; apparently once you’re better, you’re a lot better. Bots have improved that much in the last 2 years.

Here’s the same crosstable, except starting 1 January 2016.

since 1 Jan	overall	kras	Iron	Mari	Mart	tscm	tscm	Leta	Wuli	Simo	ICEL
krasi0	45.47%		49% 57	26% 57	78% 64	13% 61	16% 73	100% 10	83% 35	55% 56	55% 51
Iron bot	53.37%	51% 57		41% 37	39% 66	58% 43	54% 59	40% 15	85% 27	55% 47	64% 50
Marian Devecka	63.12%	74% 57	59% 37		57% 100	72% 67	53% 95	79% 14	83% 12	80% 95	49% 106
Martin Rooijackers	53.71%	22% 64	61% 66	43% 100		57% 168	47% 230	64% 11	74% 31	44% 162	81% 164
tscmooz	62.23%	87% 61	42% 43	28% 67	43% 168		66% 142	27% 11	57% 23	77% 138	84% 128
tscmoo	62.43%	84% 73	46% 59	47% 95	53% 230	34% 142		43% 14	53% 17	88% 154	84% 161
LetaBot CIG 2016	56.64%	0% 10	60% 15	21% 14	36% 11	73% 11	57% 14		69% 13	100% 17	75% 8
WuliBot	24.63%	17% 35	15% 27	17% 12	26% 31	43% 23	47% 17	31% 13		16% 19	19% 26
Simon Prins	33.01%	45% 56	45% 47	20% 95	56% 162	23% 138	12% 154	0% 17	84% 19		38% 136
ICELab	33.73%	45% 51	36% 50	51% 106	19% 164	16% 128	16% 161	25% 8	81% 26	62% 136

The number of games in each cell depends on the lifetimes of both bots; not all were active over the whole period. The cells are mostly in paler colors, because many of the bots were updated during the year—they were neither always weak nor always strong.

I want to create a breakdown by map, but the data is not supporting it. There are 14 active maps. If I render data from the shorter period, there aren’t enough games for all the maps to be played in each pairing. If I use the longer period, there are usually enough games but the bots vary in strength with time so we’ll see a smear.

To break out crosstables by map, I’ll have to combine data one way or another. I could combine over time, using the longer period. I could combine bots by race, or try to compare groups of macro bots versus rush bots. I could lump together maps by number of starting spots. Or I could just do bot-map instead of bot-opponent-map. What do you think is useful? Maybe I should start with bot-map and then break it down further to bot-some group of opponents-map?

comparing AIIDE 2015 and CIG 2016 Elo ratings

The cool technique I had in mind to compare ratings across tournaments turned out not to work. Not cool after all. But 6 bots played unchanged in both AIIDE 2015 and CIG 2016, and we can compare their relative ratings. In this table the subtract column gives the AIIDE 2015 rating minus the CIG 2016 rating.

bot	AIIDE Elo	CIG elo	subtract	normalize
UAlbertaBot	1895	1778	117	35
Overkill	1890	1796	94	12
Aiur	1784	1687	97	15
TerranUAB	1372	1338	34	-48
OpprimoBot	1231	1154	77	-5
Bonjwa	1171	1099	72	-10
average			82	0

As you might expect, two tournaments with different maps and different opponents give different ratings. UAlbertaBot and Overkill swapped ranks among the 6. But after correcting for the 82 point offset (since only rating differences matter), the ratings turn out to be quite close between the tournaments. The biggest difference is for TerranUAB. Look up 48 points in the Elo table—it says that TerranUAB has a 57% probability of beating itself, not a drastic error.

You can try to convert a CIG 2016 rating into a rough estimate of an AIIDE 2015 rating by adding 82. For example, tscmoo terran earned a CIG rating of 1888, which corresponds to an AIIDE rating of 1888+82 = 1970, whereas the tscmoo zerg that played in AIIDE earned a rating there of 2026. So the estimate appears to be way off. But estimates made this way are likely to be closer for bots near the middle of the pack.

Next: Another mass of colorful crosstables.

CIG 2016 Bayesian Elo ratings

Same as yesterday, Bayesian Elo ratings calculated by bayeselo, this time for CIG 2016. I included both the qualifier and the final, of course. That gives the best possible ratings, so that confidence is higher for the 8 finalists. But the “score” column becomes difficult to interpret, because part of the score of the top 8 bots comes from the final when they faced tougher opposition. You can’t directly compare the scores of bots 1-8 with the scores of 9-16, only the ratings.

Also, with this analysis it doesn’t make sense to compare the rating values between tournaments. Each tournament is independently scaled to have an average rating of 1500. Only the relative ratings of bots in the same tournament can be compared. Ratings are relative.

	bot	score	Elo	95% conf.	better?
1	tscmoo	73%	1888	1872-1904	98.5%
2	Iron	71%	1864	1848-1880	99.9%
3	LetaBot	68%	1827	1811-1843	99.7%
4	Overkill	65%	1796	1781-1812	70.9%
5	ZZZKBot	64%	1790	1775-1805	86.8%
6	UAlbertaBot	63%	1778	1763-1793	99.8%
7	MegaBot	60%	1746	1731-1761	99.9%
8	Aiur	54%	1687	1671-1702	72.7%
9	Tyr	62%	1679	1659-1699	100%
10	Ziabot	46%	1500	1479-1521	100%
11	TerranUAB	34%	1338	1316-1360	100%
12	SRbotOne	22%	1158	1133-1183	59.1%
13	OpprimoBot	22%	1154	1128-1179	97.1%
14	XelnagaII	21%	1119	1092-1145	86.3%
15	Bonjwa	19%	1099	1072-1125	100%
16	Salsa	1%	579	510-636	-

The official results have LetaBot a hair ahead of ZZZKBot, then Overkill following. bayeselo has ZZZKBot and Overkill reversed, saying that LetaBot is clearly superior to Overkill, which is fairly likely to be superior to ZZZKBot. The difference comes about because, of course, the official results include only the final. Martin Rooijackers was justified after all in saying that ZZZKBot had fallen from the top 3. All other results agree with the official ranking. The tailing finalist Aiur is 72.7% likely to be superior to Tyr, so there is some doubt that the best finalists won through (in general the doubt can’t be avoided, though).

The tail-ender Salsa has a wide and asymmetrical confidence interval. It takes more evidence to pin down an extreme rating than a middle-of-the-road rating.

Tomorrow: I’ll try an analysis in which the ratings of unchanged bots are carried over from AIIDE 2015 to CIG 2016, so that we can compare between tournaments. I’m not sure how well it will work, or even if I can get it to work at all, but it will be interesting to try.

AIIDE 2015 Bayesian Elo ratings

Krasi0 asked me to calculate ratings for tournaments using Rémi Coulom’s excellent bayeselo program. Here are ratings for AIIDE 2015.

bayeselo does not calculate basic Elo ratings like my little code snippets. It can’t calculate an Elo curve over time. It assumes that the players are fixed and have one true rating, and it crunches a full-on Bayesian statistical analysis to not only find the rating as accurately as possible, but also a 95% confidence interval so you can see how accurate the rating is. The ratings for the bots that learn, which aren’t fixed in strength as bayeselo assumes, can be seen as measuring the average strength over the tournament—the tournament score is no different in that respect.

The last column of the table is the probability of superiority, bayeselo’s calculated probability that the bot truly is better than the bot ranked immediately below it. The last bot doesn’t get one, of course. (bayeselo calculates this for all pairs, but in a tournament this long it rounds off to 100% for most.)

	bot	score	Elo	95% conf.	better?
1	tscmoo	89%	2026	2002-2050	81.0%
2	ZZZKBot	88%	2011	1988-2035	99.9%
3	UAlbertaBot	80%	1895	1874-1916	61.2%
4	Overkill	81%	1890	1870-1911	99.9%
5	Aiur	73%	1784	1765-1803	99.9%
6	Ximp	68%	1712	1694-1731	99.9%
7	Skynet	64%	1666	1648-1684	50.7%
8	IceBot	64%	1666	1648-1684	88.4%
9	Xelnaga	63%	1650	1632-1668	81.4%
10	LetaBot	61%	1638	1620-1656	99.9%
11	Tyr	54%	1553	1534-1572	96.0%
12	GarmBot	52%	1531	1513-1549	100%
13	NUSBot	39%	1380	1362-1398	73.1%
14	TerranUAB	38%	1372	1354-1390	99.8%
15	Cimex	36%	1335	1316-1353	99.6%
16	CruzBot	32%	1299	1280-1317	99.9%
17	OpprimoBot	28%	1231	1211-1250	96.7%
18	Oritaka	26%	1205	1185-1225	84.0%
19	Stone	25%	1190	1170-1210	91.3%
20	Bonjwa	23%	1171	1151-1191	100%
21	Yarmouk	9%	913	885-939	95.0%
22	SusanooTricks	8%	882	853-910	-

In the official results, Overkill came in ahead of UAlbertaBot with a higher tournament score. bayeselo ratings are more accurate than score because they take into account more information, and bayeselo says UAlbertaBot > Overkill with probability 61%. As explained in the original results, it’s a statistical tie, but bayeselo says it’s not an even tie but a little tilted in a counterintuitive way.

Skynet looks dead even with IceBot in the rounded-off numbers above. bayeselo says that Skynet > IceBot with probability 50.7%, a hair off dead even. Even the large number of games in this tournament could not rank all the bots accurately.

Tomorrow: The same for CIG 2016.

7 eras of SSCAIT champions

The player with the highest rating on each day I’ll call the “champion” of the day. It turns out that the SSCAIT daily champions change frequently. I calculated all the daily champions and manually divided the time period into eras that looked to me to have different sets of champions.

dates		champions
2013 12 25	2015 02 16	usually ICELab, sometimes Tomas Vajda (XIMP)
2015 02 17	2015 02 25	Tomas Vajda and Dave Churchill (UAlbertaBot)
2015 02 26	2015 06 17	mostly tscmoo, sometimes Tomas Vajda or ICELab
2015 06 21	2015 11 19	largely tscmooz, sometimes Tomas Vajda or ICELab, occasionally Dave Churchill or Florian Richoux (AIUR), 3 days of Sijia Xu (Overkill)
2015 11 20	2016 02 08	Marian Devecka (Killerbot), 2 days of tscmooz
2016 02 09	2016 08 17	mostly Marian Devecka, sometimes tscmoop, 4 days of Iron
2016 08 18	2016 09 26	mostly Krasi0, sometimes Marian Devecka

You can download the SSCAIT champions file in .csv format.

SSCAIT Elo ratings over time

Here it is, the great chart of SSCAIT Elo ratings over time. Well, not here actually, I put it on a separate page so that not every blog visitor has to load the mass of Javascript and data.

SSCAIT interactive ratings chart for 100 bots

The chart is generated from this csv file. Spreadsheet software or stats software should open it right up, if you want to poke the data yourself. It’s 950 lines of 101 columns each, a date and ratings for 100 bots.

Data in the csv file is filled in for each day from the bot’s first to its last game in the original raw data file (which is just a list of games), and left blank on other days. There may be an off-by-one error causing some bots to miss their last day of data; I didn’t bother to verify it since it’s hardly visible. Some bots have short lifetimes and only appear on the graph as a brief squiggle. Some bots have inactive periods in between their first and last games; the inactive periods with no games graph as flat lines. In excluding the 3 bots with insufficient games, I also removed them from the rating calculation, which improves the ratings to a tiny degree. The rankings stay exactly the same for all 100 bots, though.

Elo ratings are easy to calculate

Elo ratings may seem mysterious and complicated, but basic Elo ratings are super easy to calculate. The entire code is in these 2 yellow boxes. It’s perl, a particularly ugly language, but any coder should be able to read it.

First, given two ratings, here’s how you find out the probability that the player with the first rating beats the player with the second. I broke it out because it’s useful on its own. It’s literally 1 line of calculation. This is the logistic function, which has been shown empirically to be a good fit for the job. 400 is a scaling constant which is standard for Elo.

sub expected_win_rate ($, $) {
  my $rating1 = shift;
  my $rating2 = shift;
  
  return 1.0 / (1.0 + 10.0 ** (($rating2 - $rating1) / 400.0));
}

Second, given two ratings and a game result, here’s how you figure out the two new ratings after the game. This time it’s 2 lines of calculation, and it calls the expected win rate function above. $actual is the game result, 0.0 if the first player lost and 1.0 if the first player won. You can use 0.5 for a draw, but I skipped over draws because I’m not sure what ‘draw’ means in the file I have (there are 1211 draws in the 141,164 games, a negligible number, so it shouldn’t make much difference). $elo_k is the K constant for the Elo formula, which is the maximum rating change per game. Setting $elo_k high means that ratings react quickly to changes, and setting $elo_k low means ratings are more accurate if changes are slow. I have a lot of games, so I set $elo_k to a low value, 16. Other common values are 24 and 32.

sub update_elo ($, $, $) {
  my $rating1 = shift;
  my $rating2 = shift;
  my $actual = shift;

  my $delta = $elo_k * ($actual - &expected_win_rate ($rating1, $rating2));

  return ($rating1 + $delta, $rating2 - $delta);
}

Third and last, you need to already have an Elo rating before you can use the Elo formula. How do you set a player’s initial rating? If you don’t have any better idea, it’s standard to set it to 1500. It will take a while to become accurate. I used the calculate-backward trick to get accurate initial ratings, but that only works if you have the whole dataset ahead of time. Sometimes players are rated by a different “provisional” system for some small number of early games, before Elo kicks in.

And that’s the story! There are a bunch of fancy variations of Elo which try to do a little better. And though I think they mostly do do a little better, they’re more complicated and not very much better.

The bottom line: Calculating Elo ratings is easy and works well, so you should do it. If you care about playing strength and have the data, ratings are your answer.

a few preliminary Elo charts

The SSCAIT data includes 103 bots, and 3 of them have 10 or fewer games, leaving exactly 100 with useful rating curves. I’ve crunched and formatted the data, and now all I have to do is draw it. I hope to create a humongalicious zoomable graph of daily rating data for all 100 bots—if I can find a way to draw that many lines on a graph in a way that’s usable. Well, I’ll think of something. I chose powerful graphing software that’s fully capable of doing the job, but it’s complicated and my skill and patience may be less than fully capable....

Anyway, another appetizer. Here are static rating graphs for 2016 for the top 3 CIG finishers, all of which had many updates this year. The graphs run from 1 January 2016 to 27 September 2016. The authors may be interested in comparing their updates with movements in their graph. Krasi0 shows steady improvement since April, while the other two look more irregular.

graph of Krasi0’s rating in 2016

graph of Iron’s rating in 2016

graph of Tacmoo terran’s rating in 2016

ZerGreenBot does zealot bombs

A quick note: The current version of ZerGreenBot has given up on reaver drops (at least for now), but it does know how to bomb tanks with zealots. I think it’s the first bot with that skill.