Starcraft AI blog

The SSCAIT data includes 103 bots, and 3 of them have 10 or fewer games, leaving exactly 100 with useful rating curves. I’ve crunched and formatted the data, and now all I have to do is draw it. I hope to create a humongalicious zoomable graph of daily rating data for all 100 bots—if I can find a way to draw that many lines on a graph in a way that’s usable. Well, I’ll think of something. I chose powerful graphing software that’s fully capable of doing the job, but it’s complicated and my skill and patience may be less than fully capable....

Anyway, another appetizer. Here are static rating graphs for 2016 for the top 3 CIG finishers, all of which had many updates this year. The graphs run from 1 January 2016 to 27 September 2016. The authors may be interested in comparing their updates with movements in their graph. Krasi0 shows steady improvement since April, while the other two look more irregular.

A quick note: The current version of ZerGreenBot has given up on reaver drops (at least for now), but it does know how to bomb tanks with zealots. I think it’s the first bot with that skill.

Wow, did you see the new protoss bot Bereaver? It was uploaded today at SSCAIT. In its first game it failed to start (oops). But in its second game it played strikingly well against Krasi0, breaking Krasi0’s bunker in the early game even while expanding and teching; Krasi0 defended well and held, of course, because that’s what Krasi0 does. Then Bereaver put up a fierce fight in the middle game with reavers and high templar, repeatedly wearing down and breaking Krasi0’s pushes until Bereaver ran out of resources, unable to expand beyond its third in the face of terran map control.

Somebody with game-scheduling power must have seen the game too, because games against other top bots came up right away. Bereaver lost more than it won, but it did defeat IceBot despite misplacing its natural nexus and being unable to take a third, ignorant of how to clear the mine blocking the expansion spot.

I was also impressed with the game against Andrew Smith’s Skynet. Bereaver apparently diagnosed Skynet’s dark templar rush and prepared against it, cannoning and easily holding its ramp. The cannons were misplaced and blocked dragoons and reavers inside, a fatal blunder, but the basic skill is there.

It’s a great start for a new bot. Many rough edges are in plain sight, which means that improvements should come easily!

Here’s a table that explains what Elo ratings mean. To find out the chance that one bot will beat another, subtract their Elo ratings and look up the difference in the table. Iron is rated 2081 and Wulibot is rated 1871. The difference is 210—look it up in the table!

The probability estimate is not perfect, but it is good on average.

rating diff	win %	rating diff	win %	rating diff	win %	rating diff	win %
0	50%	200	76%	400	91%	600	97%
10	51%	210	77%	410	91%	610	97%
20	53%	220	78%	420	92%	620	97%
30	54%	230	79%	430	92%	630	97%
40	56%	240	80%	440	93%	640	98%
50	57%	250	81%	450	93%	650	98%
60	59%	260	82%	460	93%	660	98%
70	60%	270	83%	470	94%	670	98%
80	61%	280	83%	480	94%	680	98%
90	63%	290	84%	490	94%	690	98%
100	64%	300	85%	500	95%	700	98%
110	65%	310	86%	510	95%	710	98%
120	67%	320	86%	520	95%	720	98%
130	68%	330	87%	530	95%	730	99%
140	69%	340	88%	540	96%	740	99%
150	70%	350	88%	550	96%	750	99%
160	72%	360	89%	560	96%	760	99%
170	73%	370	89%	570	96%	770	99%
180	74%	380	90%	580	97%	780	99%
190	75%	390	90%	590	97%	790	99%
200	76%	400	91%	600	97%	800	99%

I’m still working on Elo curves over time, but today I have Elo ratings for each bot in the SSCAIT data at the beginning and end of its career. Here is yesterday’s table plus the new info, now sorted by decreasing current rating—the bot’s real strength yesterday as best we can measure. The topmost ratings are, to my surprise, exactly in the order I expected!

To make the ratings easier to interpret, I added two columns labeled “expect”. These are the expected winning rate of the bot against the average opponent. The rating system is designed so that the average Elo rating is constant at 1500, and it’s easy to compute the expected winning rate against an opponent rated 1500. The constant average rating, by the way, means that a bot which remains the same can see its rating decline over time if its opponents improve.

Ratings are not accurate for bots with a very small number of games. I plan to exclude those bots from the curves over time.

		initial		current
bot	win %	Elo	expect	Elo	expect	games	earliest	latest
krasi0	68.77%	1593	63.07%	2163	97.85%	2142	2015 Nov 30	2016 Sep 27
Iron bot	77.74%	1580	61.31%	2081	96.59%	1999	2015 Nov 27	2016 Sep 26
Marian Devecka	58.66%	1790	84.15%	2065	96.28%	6289	2013 Dec 25	2016 Sep 27
Martin Rooijackers	68.50%	1840	87.62%	2011	94.99%	7290	2014 Jul 28	2016 Sep 27
tscmooz	79.80%	1823	86.52%	1991	94.41%	5006	2015 Feb 27	2016 Sep 27
tscmoo	72.06%	1838	87.50%	1978	94.00%	5719	2015 Jan 22	2016 Sep 27
LetaBot CIG 2016	75.68%	1748	80.65%	1932	92.32%	444	2016 Aug 01	2016 Sep 27
WuliBot	72.76%	1773	82.80%	1871	89.43%	984	2016 Apr 19	2016 Sep 26
Simon Prins	55.48%	1513	51.87%	1867	89.21%	5431	2015 Jan 25	2016 Sep 27
ICELab	81.12%	2189	98.14%	1865	89.10%	8344	2013 Dec 25	2016 Sep 27
FlashTest	69.44%	1744	80.29%	1863	88.99%	216	2016 Mar 22	2016 Jul 27
Sijia Xu	71.65%	1850	88.23%	1849	88.17%	2328	2015 Oct 10	2016 Sep 27
LetaBot SSCAI 2015 Final	65.87%	1710	77.01%	1813	85.84%	416	2016 Aug 04	2016 Sep 27
Dave Churchill	75.48%	1985	94.22%	1804	85.19%	8275	2013 Dec 25	2016 Sep 27
Chris Coxe	73.10%	1754	81.19%	1800	84.90%	2201	2015 Sep 03	2016 Sep 27
Tomas Vajda	79.37%	2169	97.92%	1790	84.15%	8372	2013 Dec 25	2016 Sep 27
Flash	65.69%	1458	43.98%	1777	83.13%	991	2016 Apr 18	2016 Sep 27
LetaBot IM noMCTS	60.93%	1645	69.73%	1766	82.22%	1226	2016 May 18	2016 Aug 01
Zia bot	52.24%	1568	59.66%	1757	81.45%	536	2016 Jul 07	2016 Sep 27
A Jarocki	62.77%	1711	77.11%	1741	80.02%	932	2015 Oct 04	2016 Jan 26
PeregrineBot	57.29%	1692	75.12%	1728	78.79%	1276	2016 Feb 09	2016 Sep 10
tscmoop	78.16%	1895	90.67%	1721	78.11%	1992	2015 Nov 11	2016 Sep 26
Andrew Smith	65.00%	1705	76.50%	1718	77.81%	8391	2013 Dec 25	2016 Sep 27
Florian Richoux	62.11%	1770	82.55%	1716	77.62%	8203	2013 Dec 25	2016 Sep 27
Carsten Nielsen	66.08%	1708	76.81%	1695	75.45%	4711	2015 Mar 17	2016 Sep 27
Soeren Klett	63.62%	2068	96.34%	1687	74.58%	8277	2013 Dec 25	2016 Sep 27
Vaclav Horazny	37.35%	1066	7.60%	1686	74.47%	6455	2013 Dec 25	2015 Nov 18
La Nuee	51.61%	1499	49.86%	1662	71.76%	558	2015 Dec 13	2016 Mar 18
Jakub Trancik	45.08%	1755	81.27%	1657	71.17%	8416	2013 Dec 25	2016 Sep 27
Marek Suppa	51.85%	1746	80.47%	1655	70.94%	4413	2015 Jan 05	2016 Mar 18
Krasimir Krystev	70.52%	2033	95.56%	1653	70.70%	6510	2013 Dec 25	2016 Mar 10
ASPbot2011	49.78%	1671	72.80%	1652	70.58%	227	2015 Jan 29	2016 Feb 25
Marcin Bartnicki	60.42%	1855	88.53%	1633	68.26%	1435	2014 Nov 28	2016 Mar 18
Tomas Cere	61.11%	1888	90.32%	1631	68.01%	8373	2013 Dec 25	2016 Sep 27
MegaBot	49.40%	1576	60.77%	1630	67.88%	419	2016 Aug 01	2016 Sep 27
Aurelien Lermant	58.26%	1688	74.69%	1622	66.87%	3687	2015 Jun 22	2016 Sep 27
Matej Kravjar	49.57%	1723	78.31%	1619	66.49%	3234	2013 Dec 25	2015 Feb 18
Daniel Blackburn	43.79%	1651	70.46%	1605	64.67%	6883	2013 Dec 25	2016 Jan 26
Gabriel Synnaeve	45.96%	1737	79.65%	1584	61.86%	1658	2013 Dec 25	2015 Nov 24
David Milec	49.09%	1552	57.43%	1566	59.39%	55	2015 Jan 13	2015 Jan 20
Odin2014	55.65%	1659	71.41%	1565	59.25%	5648	2014 Dec 21	2016 Sep 11
Gaoyuan Chen	48.05%	1582	61.59%	1559	58.41%	5118	2015 Feb 10	2016 Sep 27
Henri Kumpulainen	38.81%	1447	42.43%	1553	57.57%	894	2016 Jan 13	2016 May 31
Martin Dekar	33.14%	1429	39.92%	1533	54.73%	4910	2013 Dec 25	2016 Jan 25
Serega	48.20%	1771	82.64%	1505	50.72%	3803	2015 Jan 31	2016 Jan 26
Chris Ayers	35.53%	1610	65.32%	1481	47.27%	1520	2015 Aug 10	2016 Jan 26
Nathan a David	39.34%	1446	42.29%	1481	47.27%	1004	2016 Feb 23	2016 Aug 08
DAIDOES	34.02%	1370	32.12%	1471	45.84%	485	2016 Jun 13	2016 Sep 08
FlashZerg	0.00%	1474	46.27%	1459	44.13%	7	2016 Apr 24	2016 May 12
Igor Lacik	39.32%	1608	65.06%	1454	43.42%	8073	2013 Dec 25	2016 Sep 08
Matej Istenik	44.74%	1709	76.91%	1449	42.71%	8297	2013 Dec 25	2016 Sep 27
EradicatumXVR	40.88%	1537	55.30%	1443	41.87%	4687	2013 Dec 25	2016 Jan 23
Ibrahim Awwal	30.57%	1510	51.44%	1437	41.03%	530	2013 Dec 25	2014 Mar 24
Tomasz Michalski	27.02%	1314	25.53%	1432	40.34%	433	2015 Dec 22	2016 Mar 18
Oleg Ostroumov	48.75%	1714	77.41%	1431	40.20%	3641	2013 Dec 25	2016 Jan 26
NUS Bot	35.72%	1482	47.41%	1426	39.51%	3337	2015 May 19	2016 Sep 06
Martin Pinter	28.98%	1409	37.20%	1425	39.37%	3740	2013 Dec 25	2015 Dec 11
Roman Danielis	45.63%	1688	74.69%	1417	38.28%	5155	2013 Dec 25	2016 Sep 26
ZerGreenBot	22.22%	1404	36.53%	1416	38.14%	36	2016 Sep 22	2016 Sep 27
Rafael Bocquet	0.00%	1450	42.85%	1415	38.01%	10	2015 Jun 23	2015 Jun 26
Flashrelease	0.00%	1449	42.71%	1413	37.73%	8	2016 Apr 24	2016 Apr 24
Marek Kadek	37.29%	1557	58.13%	1413	37.73%	7641	2013 Dec 25	2016 May 22
Ian Nicholas DaCosta	37.12%	1394	35.20%	1404	36.53%	2928	2015 Apr 27	2016 Sep 08
AwesomeBot	29.81%	1326	26.86%	1403	36.39%	473	2016 Jun 16	2016 Sep 08
Radim Bobek	23.37%	1315	25.64%	1390	34.68%	1151	2015 Oct 01	2016 Mar 06
Adrian Sternmuller	26.89%	1436	40.89%	1375	32.75%	4529	2013 Dec 25	2016 Jul 22
Martin Strapko	19.76%	1388	34.42%	1366	31.62%	3386	2013 Dec 25	2016 Jan 26
Maja Nemsilajova	23.81%	1365	31.49%	1363	31.25%	4246	2013 Dec 25	2015 Nov 29
Johan Kayser	24.46%	1294	23.40%	1361	31.00%	413	2016 Jul 29	2016 Sep 27
UPStarcraftAI	24.75%	1346	29.18%	1360	30.88%	610	2015 Dec 24	2016 Apr 13
Martin Vlcak	28.92%	1370	32.12%	1353	30.02%	1224	2016 Feb 16	2016 Sep 07
Johannes Holzfuss	35.04%	1531	54.45%	1351	29.78%	685	2016 Mar 05	2016 Jun 15
Vojtech Jirsa	14.14%	1186	14.09%	1350	29.66%	2786	2015 Jan 12	2015 Sep 05
JompaBot	21.99%	1316	25.75%	1349	29.54%	1055	2016 Feb 04	2016 Aug 13
Rob Bogie	31.34%	1335	27.89%	1346	29.18%	651	2016 May 14	2016 Sep 06
Christoffer Artmann	20.51%	1289	22.89%	1344	28.95%	395	2016 Aug 07	2016 Sep 27
Marek Gajdos	22.69%	1251	19.26%	1331	27.43%	1384	2016 Jan 30	2016 Sep 11
Travis Shelton	23.59%	1390	34.68%	1314	25.53%	1221	2016 Feb 28	2016 Sep 06
Peter Dobsa	13.25%	1227	17.20%	1307	24.77%	3027	2015 Jan 11	2015 Oct 02
VeRLab	17.06%	1241	18.38%	1304	24.45%	897	2016 Feb 28	2016 Aug 01
Andrej Sekac	11.76%	1359	30.75%	1296	23.61%	68	2013 Dec 25	2014 Jan 04
Bjorn P Mattsson	22.22%	1351	29.78%	1295	23.50%	4442	2015 Apr 05	2016 Sep 27
Lukas Sedlacek	22.86%	1344	28.95%	1293	23.30%	70	2015 Jan 12	2015 Jan 20
Sergei Lebedinskij	13.30%	1178	13.55%	1293	23.30%	1083	2015 May 28	2015 Sep 03
Vladimir Jurenka	38.45%	1635	68.51%	1278	21.79%	6167	2013 Dec 25	2016 Sep 27
neverdieTRX	20.66%	1265	20.54%	1272	21.21%	334	2016 Jul 19	2016 Sep 10
OpprimoBot	21.85%	1321	26.30%	1256	19.71%	2009	2015 Nov 18	2016 Sep 27
Marek Kruzliak	14.45%	1151	11.83%	1255	19.62%	934	2013 Dec 25	2015 Jan 20
Sungguk Cha	18.65%	1207	15.62%	1250	19.17%	697	2016 Jun 05	2016 Sep 27
Jacob Knudsen	20.53%	1083	8.31%	1247	18.90%	1257	2016 Feb 23	2016 Sep 10
Ludmila Nemsilajova	16.04%	1133	10.79%	1228	17.28%	505	2013 Dec 25	2015 Jan 21
Karin Valisova	17.68%	1238	18.12%	1226	17.12%	1171	2013 Dec 25	2016 Jan 26
HoangPhuc	15.67%	1132	10.73%	1209	15.77%	300	2016 Jul 18	2016 Sep 07
Sebastian Mahr	15.06%	1205	15.47%	1182	13.82%	1202	2016 Jan 13	2016 Aug 08
Jan Pajan	14.48%	1210	15.85%	1179	13.61%	1119	2013 Dec 25	2016 Jan 05
Pablo Garcia Sanchez	12.20%	1123	10.25%	1174	13.28%	590	2015 Dec 24	2016 Apr 13
Ivana Kellyerova	11.47%	1129	10.57%	1131	10.68%	1630	2013 Dec 25	2015 Apr 01
Lucia Pivackova	13.29%	1111	9.63%	1090	8.63%	835	2013 Dec 25	2015 Jan 20
Tae Jun Oh	4.55%	1069	7.72%	1036	6.47%	154	2016 Mar 22	2016 Apr 11
Denis Ivancik	10.76%	1102	9.19%	1022	6.00%	502	2013 Dec 25	2015 Jan 20
ButcherBoy	4.74%	921	3.45%	970	4.52%	422	2016 Jun 21	2016 Sep 06
Jon W	5.06%	920	3.43%	964	4.37%	790	2015 Apr 30	2015 Jul 09
Matyas Novy	6.32%	1130	10.62%	885	2.82%	1693	2015 Feb 04	2015 Jul 09

How did I get the initial ratings? I had a cute idea. One of the issues with computing Elo ratings over time is: How do you initialize the ratings? Most systems either start everybody with the same rating, which makes an ugly graph, or use a different and less accurate method to estimate the rating in early games. But in this case I have the whole data set in hand. I set the final rating of every bot to the same rating and computed ratings backwards in time to find an initial rating. Then I threw away everything except the initial rating, and calculated the real ratings forward in time to find the ratings over time and the final ratings. That way every data point is equally good, from beginning to end. I doubt I’m the first to think of it, but it’s a cute idea and I’m pleased.

Next: I’ll find some sensible way to plot the curves. Stand by!

Krasimir Krastev aka Krasi0 sent me a file of game results from SSCAIT, including 141,163 games recorded between 25 December 2013 and today. (Obviously it doesn’t include all games played today.) He’s particularly interested in the evolution of Elo ratings over time and my colorful crosstables per map.

It may take me a while to get to that stuff. Here’s a down payment. First, the career record of the 103 bots in the data, with win rates and dates. The top career win rate is IceBot from ICELab, followed by Tscmoo zerg and Tomas Vajda’s XIMP. Of course career win rate is not a fair comparison for bots which improved greatly over their careers, or which have shorter careers.

bot	win %	games	earliest	latest
A Jarocki	62.77%	932	2015 Oct 04	2016 Jan 26
Adrian Sternmuller	26.89%	4529	2013 Dec 25	2016 Jul 22
Andrej Sekac	11.76%	68	2013 Dec 25	2014 Jan 04
Andrew Smith	65.00%	8391	2013 Dec 25	2016 Sep 27
ASPbot2011	49.78%	227	2015 Jan 29	2016 Feb 25
Aurelien Lermant	58.26%	3687	2015 Jun 22	2016 Sep 27
AwesomeBot	29.81%	473	2016 Jun 16	2016 Sep 08
Bjorn P Mattsson	22.22%	4442	2015 Apr 05	2016 Sep 27
ButcherBoy	4.74%	422	2016 Jun 21	2016 Sep 06
Carsten Nielsen	66.08%	4711	2015 Mar 17	2016 Sep 27
Chris Ayers	35.53%	1520	2015 Aug 10	2016 Jan 26
Chris Coxe	73.10%	2201	2015 Sep 03	2016 Sep 27
Christoffer Artmann	20.51%	395	2016 Aug 07	2016 Sep 27
DAIDOES	34.02%	485	2016 Jun 13	2016 Sep 08
Daniel Blackburn	43.79%	6883	2013 Dec 25	2016 Jan 26
Dave Churchill	75.48%	8275	2013 Dec 25	2016 Sep 27
David Milec	49.09%	55	2015 Jan 13	2015 Jan 20
Denis Ivancik	10.76%	502	2013 Dec 25	2015 Jan 20
EradicatumXVR	40.88%	4687	2013 Dec 25	2016 Jan 23
Flash	65.69%	991	2016 Apr 18	2016 Sep 27
Flashrelease	0.00%	8	2016 Apr 24	2016 Apr 24
FlashTest	69.44%	216	2016 Mar 22	2016 Jul 27
FlashZerg	0.00%	7	2016 Apr 24	2016 May 12
Florian Richoux	62.11%	8203	2013 Dec 25	2016 Sep 27
Gabriel Synnaeve	45.96%	1658	2013 Dec 25	2015 Nov 24
Gaoyuan Chen	48.05%	5118	2015 Feb 10	2016 Sep 27
Henri Kumpulainen	38.81%	894	2016 Jan 13	2016 May 31
HoangPhuc	15.67%	300	2016 Jul 18	2016 Sep 07
Ian Nicholas DaCosta	37.12%	2928	2015 Apr 27	2016 Sep 08
Ibrahim Awwal	30.57%	530	2013 Dec 25	2014 Mar 24
ICELab	81.12%	8344	2013 Dec 25	2016 Sep 27
Igor Lacik	39.32%	8073	2013 Dec 25	2016 Sep 08
Iron bot	77.74%	1999	2015 Nov 27	2016 Sep 26
Ivana Kellyerova	11.47%	1630	2013 Dec 25	2015 Apr 01
Jacob Knudsen	20.53%	1257	2016 Feb 23	2016 Sep 10
Jakub Trancik	45.08%	8416	2013 Dec 25	2016 Sep 27
Jan Pajan	14.48%	1119	2013 Dec 25	2016 Jan 05
Johan Kayser	24.46%	413	2016 Jul 29	2016 Sep 27
Johannes Holzfuss	35.04%	685	2016 Mar 05	2016 Jun 15
JompaBot	21.99%	1055	2016 Feb 04	2016 Aug 13
Jon W	5.06%	790	2015 Apr 30	2015 Jul 09
Karin Valisova	17.68%	1171	2013 Dec 25	2016 Jan 26
krasi0	68.77%	2142	2015 Nov 30	2016 Sep 27
Krasimir Krystev	70.52%	6510	2013 Dec 25	2016 Mar 10
La Nuee	51.61%	558	2015 Dec 13	2016 Mar 18
LetaBot CIG 2016	75.68%	444	2016 Aug 01	2016 Sep 27
LetaBot IM noMCTS	60.93%	1226	2016 May 18	2016 Aug 01
LetaBot SSCAI 2015 Final	65.87%	416	2016 Aug 04	2016 Sep 27
Lucia Pivackova	13.29%	835	2013 Dec 25	2015 Jan 20
Ludmila Nemsilajova	16.04%	505	2013 Dec 25	2015 Jan 21
Lukas Sedlacek	22.86%	70	2015 Jan 12	2015 Jan 20
Maja Nemsilajova	23.81%	4246	2013 Dec 25	2015 Nov 29
Marcin Bartnicki	60.42%	1435	2014 Nov 28	2016 Mar 18
Marek Gajdos	22.69%	1384	2016 Jan 30	2016 Sep 11
Marek Kadek	37.29%	7641	2013 Dec 25	2016 May 22
Marek Kruzliak	14.45%	934	2013 Dec 25	2015 Jan 20
Marek Suppa	51.85%	4413	2015 Jan 05	2016 Mar 18
Marian Devecka	58.66%	6289	2013 Dec 25	2016 Sep 27
Martin Dekar	33.14%	4910	2013 Dec 25	2016 Jan 25
Martin Pinter	28.98%	3740	2013 Dec 25	2015 Dec 11
Martin Rooijackers	68.50%	7290	2014 Jul 28	2016 Sep 27
Martin Strapko	19.76%	3386	2013 Dec 25	2016 Jan 26
Martin Vlcak	28.92%	1224	2016 Feb 16	2016 Sep 07
Matej Istenik	44.74%	8297	2013 Dec 25	2016 Sep 27
Matej Kravjar	49.57%	3234	2013 Dec 25	2015 Feb 18
Matyas Novy	6.32%	1693	2015 Feb 04	2015 Jul 09
MegaBot	49.40%	419	2016 Aug 01	2016 Sep 27
Nathan a David	39.34%	1004	2016 Feb 23	2016 Aug 08
neverdieTRX	20.66%	334	2016 Jul 19	2016 Sep 10
NUS Bot	35.72%	3337	2015 May 19	2016 Sep 06
Odin2014	55.65%	5648	2014 Dec 21	2016 Sep 11
Oleg Ostroumov	48.75%	3641	2013 Dec 25	2016 Jan 26
OpprimoBot	21.85%	2009	2015 Nov 18	2016 Sep 27
Pablo Garcia Sanchez	12.20%	590	2015 Dec 24	2016 Apr 13
PeregrineBot	57.29%	1276	2016 Feb 09	2016 Sep 10
Peter Dobsa	13.25%	3027	2015 Jan 11	2015 Oct 02
Radim Bobek	23.37%	1151	2015 Oct 01	2016 Mar 06
Rafael Bocquet	0.00%	10	2015 Jun 23	2015 Jun 26
Rob Bogie	31.34%	651	2016 May 14	2016 Sep 06
Roman Danielis	45.63%	5155	2013 Dec 25	2016 Sep 26
Sebastian Mahr	15.06%	1202	2016 Jan 13	2016 Aug 08
Serega	48.20%	3803	2015 Jan 31	2016 Jan 26
Sergei Lebedinskij	13.30%	1083	2015 May 28	2015 Sep 03
Sijia Xu	71.65%	2328	2015 Oct 10	2016 Sep 27
Simon Prins	55.48%	5431	2015 Jan 25	2016 Sep 27
Soeren Klett	63.62%	8277	2013 Dec 25	2016 Sep 27
Sungguk Cha	18.65%	697	2016 Jun 05	2016 Sep 27
Tae Jun Oh	4.55%	154	2016 Mar 22	2016 Apr 11
Tomas Cere	61.11%	8373	2013 Dec 25	2016 Sep 27
Tomas Vajda	79.37%	8372	2013 Dec 25	2016 Sep 27
Tomasz Michalski	27.02%	433	2015 Dec 22	2016 Mar 18
Travis Shelton	23.59%	1221	2016 Feb 28	2016 Sep 06
tscmoo	72.06%	5719	2015 Jan 22	2016 Sep 27
tscmoop	78.16%	1992	2015 Nov 11	2016 Sep 26
tscmooz	79.80%	5006	2015 Feb 27	2016 Sep 27
UPStarcraftAI	24.75%	610	2015 Dec 24	2016 Apr 13
Vaclav Horazny	37.35%	6455	2013 Dec 25	2015 Nov 18
VeRLab	17.06%	897	2016 Feb 28	2016 Aug 01
Vladimir Jurenka	38.45%	6167	2013 Dec 25	2016 Sep 27
Vojtech Jirsa	14.14%	2786	2015 Jan 12	2015 Sep 05
WuliBot	72.76%	984	2016 Apr 19	2016 Sep 26
ZerGreenBot	22.22%	36	2016 Sep 22	2016 Sep 27
Zia bot	52.24%	536	2016 Jul 07	2016 Sep 27

Also the maps. Games on 2014 October 24 and earlier did not specify the map; it is blank in the file. The first game with a map specified was 2014 October 29, so there’s a gap in the records (maybe downtime, or tournament stuff). Anyway, we can see that the maps are the usual SSCAIT map pack plus BGH for a small number of games on April Fools.

It is Most Curious that Electric Circuit has fewer games. It was last played on 2015 Feb 3, though I still see it in the map pack that they distribute.

map	games	earliest	latest
(2)Benzene.scx	8257	2014 Oct 29	2016 Sep 27
(2)Destination.scx	8137	2014 Oct 29	2016 Sep 27
(2)HeartbreakRidge.scx	8249	2014 Oct 29	2016 Sep 27
(3)NeoMoonGlaive.scx	8157	2014 Oct 29	2016 Sep 27
(3)TauCross.scx	8182	2014 Oct 29	2016 Sep 27
(4)Andromeda.scx	8233	2014 Oct 29	2016 Sep 27
(4)CircuitBreaker.scx	8083	2014 Oct 29	2016 Sep 27
(4)ElectricCircuit.scx	975	2014 Oct 29	2015 Feb 03
(4)EmpireoftheSun.scm	8318	2014 Oct 29	2016 Sep 26
(4)FightingSpirit.scx	8288	2014 Oct 29	2016 Sep 27
(4)Icarus.scm	8237	2014 Oct 29	2016 Sep 27
(4)Jade.scx	8154	2014 Oct 29	2016 Sep 27
(4)LaMancha1.1.scx	8130	2014 Oct 29	2016 Sep 27
(4)Python.scx	8175	2014 Oct 29	2016 Sep 27
(4)Roadrunner.scx	8172	2014 Oct 29	2016 Sep 27
(8)BGH.scm	463	2015 Apr 01	2016 Apr 02
[none specified]	24953	2013 Dec 25	2014 Oct 24

Today I crush you under a mass of charts, crosstables for each of the 5 maps in CIG 2016. This is 4 dimensional data (bot 1, bot 2, map, winning rate) and I imagine there’s a clearer way to present it, but I don’t know what it is so you get it in the first form I thought of. At the end is a link to the software.

As a reminder, here are the maps.

• (2)RideofValkyries1.0
• (3)Alchemist1.0
• (3)TauCross1.1
• (4)LunaTheFinal2.3
• (4)Python1.3

With 100 rounds and 5 maps, for each pairing 20 games were played on each map (minus a few games missing due to errors). So the percentages vary in steps of 5% (or more if games are missing), and the error bars are wide.

The qualifier tables are big. The first is the full tournament for comparison, the rest are the subtournaments played on each map.

	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	79.20%		56%	39%	56%	38%	63%	53%	91%	99%	95%	100%	100%	98%	100%	100%	100%
tscmoo	76.97%	44%		48%	53%	75%	87%	82%	43%	45%	81%	100%	100%	100%	97%	100%	100%
LetaBot	74.07%	61%	52%		81%	28%	60%	51%	31%	72%	78%	100%	100%	99%	99%	99%	100%
Overkill	70.98%	44%	47%	19%		84%	32%	56%	79%	31%	88%	98%	95%	93%	99%	100%	100%
MegaBot	70.11%	62%	25%	72%	16%		52%	7%	66%	85%	99%	96%	95%	87%	91%	99%	100%
UAlbertaBot	69.25%	37%	13%	40%	68%	48%		55%	77%	42%	88%	100%	89%	85%	97%	100%	100%
ZZZKBot	69.18%	47%	18%	49%	44%	93%	45%		69%	29%	49%	100%	100%	100%	94%	100%	100%
Aiur	63.15%	9%	57%	69%	21%	34%	23%	31%		44%	89%	86%	96%	99%	94%	96%	100%
Tyr	61.64%	1%	55%	28%	69%	15%	58%	71%	56%		24%	74%	98%	94%	88%	95%	99%
Ziabot	46.43%	5%	19%	22%	12%	1%	12%	51%	11%	76%		59%	98%	100%	32%	100%	100%
TerranUAB	33.51%	0%	0%	0%	2%	4%	0%	0%	14%	26%	41%		81%	76%	90%	70%	98%
SRbotOne	22.15%	0%	0%	0%	5%	5%	11%	0%	4%	2%	2%	19%		19%	92%	74%	99%
OpprimoBot	22.10%	2%	0%	1%	7%	13%	15%	0%	1%	6%	0%	24%	81%		56%	27%	99%
XelnagaII	20.71%	0%	3%	1%	1%	9%	3%	6%	6%	12%	68%	10%	8%	44%		56%	83%
Bonjwa	18.95%	0%	0%	1%	0%	1%	0%	0%	4%	5%	0%	30%	26%	73%	44%		100%
Salsa	1.47%	0%	0%	0%	0%	0%	0%	0%	0%	1%	0%	2%	1%	1%	17%	0%

Valkyries	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	72.67%		35%	10%	40%	25%	75%	40%	75%	95%	100%	100%	100%	95%	100%	100%	100%
tscmoo	80.67%	65%		55%	60%	80%	90%	80%	30%	55%	95%	100%	100%	100%	100%	100%	100%
LetaBot	72.00%	90%	45%		95%	30%	60%	10%	10%	85%	55%	100%	100%	100%	100%	100%	100%
Overkill	71.33%	60%	40%	5%		85%	25%	50%	95%	45%	70%	95%	100%	100%	100%	100%	100%
MegaBot	70.33%	75%	20%	70%	15%		30%	0%	65%	90%	100%	95%	95%	100%	100%	100%	100%
UAlbertaBot	69.67%	25%	10%	40%	75%	70%		50%	85%	35%	90%	100%	75%	90%	100%	100%	100%
ZZZKBot	77.67%	60%	20%	90%	50%	100%	50%		100%	55%	40%	100%	100%	100%	100%	100%	100%
Aiur	63.00%	25%	70%	90%	5%	35%	15%	0%		70%	70%	80%	95%	100%	90%	100%	100%
Tyr	59.67%	5%	45%	15%	55%	10%	65%	45%	30%		65%	70%	100%	100%	95%	95%	100%
Ziabot	51.33%	0%	5%	45%	30%	0%	10%	60%	30%	35%		65%	100%	100%	90%	100%	100%
TerranUAB	34.67%	0%	0%	0%	5%	5%	0%	0%	20%	30%	35%		80%	80%	90%	80%	95%
SRbotOne	24.33%	0%	0%	0%	0%	5%	25%	0%	5%	0%	0%	20%		25%	85%	100%	100%
OpprimoBot	20.00%	5%	0%	0%	0%	0%	10%	0%	0%	0%	0%	20%	75%		75%	20%	95%
XelnagaII	12.00%	0%	0%	0%	0%	0%	0%	0%	10%	5%	10%	10%	15%	25%		40%	65%
Bonjwa	17.67%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	20%	0%	80%	60%		100%
Salsa	3.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	5%	35%	0%

Alchemist	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	77.00%		80%	45%	80%	40%	40%	0%	95%	100%	75%	100%	100%	100%	100%	100%	100%
tscmoo	69.90%	20%		35%	75%	50%	80%	75%	15%	35%	75%	100%	100%	100%	89%	100%	100%
LetaBot	72.33%	55%	65%		55%	15%	40%	80%	15%	75%	90%	100%	100%	100%	100%	95%	100%
Overkill	66.67%	20%	25%	45%		90%	10%	55%	60%	45%	85%	100%	95%	75%	95%	100%	100%
MegaBot	70.67%	60%	50%	85%	10%		45%	15%	60%	90%	100%	85%	90%	95%	80%	95%	100%
UAlbertaBot	77.00%	60%	20%	60%	90%	55%		40%	85%	80%	95%	100%	85%	85%	100%	100%	100%
ZZZKBot	72.67%	100%	25%	20%	45%	85%	60%		65%	50%	45%	100%	100%	100%	95%	100%	100%
Aiur	68.00%	5%	85%	85%	40%	40%	15%	35%		60%	90%	80%	90%	100%	100%	95%	100%
Tyr	51.33%	0%	65%	25%	55%	10%	20%	50%	40%		10%	60%	90%	80%	80%	85%	100%
Ziabot	47.33%	25%	25%	10%	15%	0%	5%	55%	10%	90%		50%	100%	100%	25%	100%	100%
TerranUAB	36.67%	0%	0%	0%	0%	15%	0%	0%	20%	40%	50%		80%	80%	90%	80%	95%
SRbotOne	23.67%	0%	0%	0%	5%	10%	15%	0%	10%	10%	0%	20%		5%	100%	80%	100%
OpprimoBot	22.33%	0%	0%	0%	25%	5%	15%	0%	0%	20%	0%	20%	95%		35%	20%	100%
XelnagaII	25.08%	0%	11%	0%	5%	20%	0%	5%	0%	20%	75%	10%	0%	65%		75%	90%
Bonjwa	18.33%	0%	0%	5%	0%	5%	0%	0%	5%	15%	0%	20%	20%	80%	25%		100%
Salsa	1.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	0%	10%	0%

Tau Cross	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	82.33%		65%	40%	75%	15%	55%	95%	90%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	77.33%	35%		35%	40%	100%	90%	95%	45%	35%	85%	100%	100%	100%	100%	100%	100%
LetaBot	80.67%	60%	65%		90%	25%	80%	80%	60%	50%	100%	100%	100%	100%	100%	100%	100%
Overkill	73.33%	25%	60%	10%		100%	40%	55%	95%	20%	95%	100%	100%	100%	100%	100%	100%
MegaBot	72.33%	85%	0%	75%	0%		55%	15%	85%	95%	100%	100%	100%	80%	95%	100%	100%
UAlbertaBot	67.33%	45%	10%	20%	60%	45%		65%	75%	35%	85%	100%	95%	75%	100%	100%	100%
ZZZKBot	57.67%	5%	5%	20%	45%	85%	35%		25%	5%	55%	100%	100%	100%	85%	100%	100%
Aiur	61.20%	10%	55%	40%	5%	15%	25%	75%		15%	95%	95%	100%	100%	95%	95%	100%
Tyr	68.33%	0%	65%	50%	80%	5%	65%	95%	85%		10%	80%	100%	100%	90%	100%	100%
Ziabot	43.00%	0%	15%	0%	5%	0%	15%	45%	5%	90%		65%	95%	100%	10%	100%	100%
TerranUAB	33.44%	0%	0%	0%	0%	0%	0%	0%	5%	20%	35%		80%	90%	100%	70%	100%
SRbotOne	21.00%	0%	0%	0%	0%	0%	5%	0%	0%	0%	5%	20%		20%	95%	70%	100%
OpprimoBot	20.67%	0%	0%	0%	0%	20%	25%	0%	0%	0%	0%	10%	80%		55%	20%	100%
XelnagaII	22.00%	0%	0%	0%	0%	5%	0%	15%	5%	10%	90%	0%	5%	45%		70%	85%
Bonjwa	18.33%	0%	0%	0%	0%	0%	0%	0%	5%	0%	0%	30%	30%	80%	30%		100%
Salsa	1.00%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	15%	0%

Luna	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	81.33%		35%	55%	50%	50%	65%	70%	95%	100%	100%	100%	100%	100%	100%	100%	100%
tscmoo	81.00%	65%		40%	35%	70%	95%	90%	65%	75%	85%	100%	100%	100%	95%	100%	100%
LetaBot	74.00%	45%	60%		70%	25%	55%	55%	30%	85%	85%	100%	100%	100%	100%	100%	100%
Overkill	68.90%	50%	65%	30%		70%	35%	55%	65%	0%	95%	100%	80%	89%	100%	100%	100%
MegaBot	71.24%	50%	30%	75%	30%		70%	0%	65%	75%	95%	100%	95%	90%	95%	100%	100%
UAlbertaBot	66.22%	35%	5%	45%	65%	30%		60%	75%	35%	75%	100%	95%	80%	95%	100%	100%
ZZZKBot	68.33%	30%	10%	45%	45%	100%	40%		80%	30%	55%	100%	100%	100%	90%	100%	100%
Aiur	60.67%	5%	35%	70%	35%	35%	25%	20%		35%	90%	85%	95%	95%	90%	95%	100%
Tyr	63.21%	0%	25%	15%	100%	25%	65%	70%	65%		30%	75%	100%	100%	85%	100%	95%
Ziabot	45.33%	0%	15%	15%	5%	5%	25%	45%	10%	70%		60%	95%	100%	35%	100%	100%
TerranUAB	31.67%	0%	0%	0%	0%	0%	0%	0%	15%	25%	40%		85%	65%	80%	65%	100%
SRbotOne	20.67%	0%	0%	0%	20%	5%	5%	0%	5%	0%	5%	15%		10%	80%	65%	100%
OpprimoBot	24.16%	0%	0%	0%	11%	10%	20%	0%	5%	0%	0%	35%	90%		60%	35%	100%
XelnagaII	22.07%	0%	5%	0%	0%	5%	5%	10%	10%	15%	65%	20%	20%	40%		45%	90%
Bonjwa	19.67%	0%	0%	0%	0%	0%	0%	0%	5%	0%	0%	35%	35%	65%	55%		100%
Salsa	1.01%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	0%	0%	0%	10%	0%

Python	overall	Iron	tscm	Leta	Over	Mega	UAlb	ZZZK	Aiur	Tyr	Ziab	Terr	SRbo	Oppr	Xeln	Bonj	Sals
Iron	82.67%		65%	45%	35%	60%	80%	60%	100%	100%	100%	100%	100%	95%	100%	100%	100%
tscmoo	75.92%	35%		75%	55%	75%	80%	70%	60%	25%	65%	100%	100%	100%	100%	100%	100%
LetaBot	71.33%	55%	25%		95%	45%	65%	30%	40%	65%	60%	100%	100%	95%	95%	100%	100%
Overkill	74.67%	65%	45%	5%		75%	50%	65%	80%	45%	95%	95%	100%	100%	100%	100%	100%
MegaBot	66.00%	40%	25%	55%	25%		60%	5%	55%	75%	100%	100%	95%	70%	85%	100%	100%
UAlbertaBot	66.00%	20%	20%	35%	50%	40%		60%	65%	25%	95%	100%	95%	95%	90%	100%	100%
ZZZKBot	69.57%	40%	30%	70%	35%	95%	40%		75%	5%	53%	100%	100%	100%	100%	100%	100%
Aiur	62.88%	0%	40%	60%	20%	45%	35%	25%		40%	100%	90%	100%	100%	95%	95%	100%
Tyr	65.67%	0%	75%	35%	55%	25%	75%	95%	60%		5%	85%	100%	90%	90%	95%	100%
Ziabot	45.12%	0%	35%	40%	5%	0%	5%	47%	0%	95%		53%	100%	100%	0%	100%	100%
TerranUAB	31.10%	0%	0%	0%	5%	0%	0%	0%	10%	15%	47%		80%	65%	90%	55%	100%
SRbotOne	21.07%	0%	0%	0%	0%	5%	5%	0%	0%	0%	0%	20%		35%	100%	55%	95%
OpprimoBot	23.33%	5%	0%	5%	0%	30%	5%	0%	0%	10%	0%	35%	65%		55%	40%	100%
XelnagaII	22.41%	0%	0%	5%	0%	15%	10%	0%	5%	10%	100%	10%	0%	45%		50%	85%
Bonjwa	20.74%	0%	0%	0%	0%	0%	0%	0%	5%	5%	0%	45%	45%	60%	50%		100%
Salsa	1.33%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	5%	0%	15%	0%

The final tables are small. Again, the first is the full tournament, the rest are the maps.

	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	65.14%		52%	44%	79%	71%	77%	83%	50%
Iron	54.43%	48%		38%	49%	49%	74%	30%	93%
LetaBot	53.71%	56%	62%		49%	81%	69%	30%	29%
ZZZKBot	53.08%	21%	51%	51%		42%	35%	93%	78%
Overkill	51.43%	29%	51%	19%	58%		43%	81%	79%
UAlbertaBot	49.07%	23%	26%	31%	65%	57%		76%	66%
MegaBot	38.00%	17%	70%	70%	7%	19%	24%		59%
Aiur	35.14%	50%	7%	71%	22%	21%	34%	41%

Valkyries	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	68.57%		60%	30%	75%	75%	85%	90%	65%
Iron	47.86%	40%		25%	40%	50%	60%	25%	95%
LetaBot	50.71%	70%	75%		0%	85%	70%	30%	25%
ZZZKBot	67.86%	25%	60%	100%		40%	55%	95%	100%
Overkill	52.14%	25%	50%	15%	60%		40%	100%	75%
UAlbertaBot	50.00%	15%	40%	30%	45%	60%		90%	70%
MegaBot	35.71%	10%	75%	70%	5%	0%	10%		80%
Aiur	27.14%	35%	5%	75%	0%	25%	30%	20%

Alchemist	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	49.29%		45%	35%	65%	80%	55%	50%	15%
Iron	44.29%	55%		50%	0%	35%	60%	30%	80%
LetaBot	51.43%	65%	50%		75%	60%	65%	20%	25%
ZZZKBot	61.43%	35%	100%	25%		45%	45%	95%	85%
Overkill	51.43%	20%	65%	40%	55%		45%	75%	60%
UAlbertaBot	52.14%	45%	40%	35%	55%	55%		75%	60%
MegaBot	42.14%	50%	70%	80%	5%	25%	25%		40%
Aiur	47.86%	85%	20%	75%	15%	40%	40%	60%

Tau Cross	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	61.43%		25%	45%	90%	45%	100%	95%	30%
Iron	66.43%	75%		55%	95%	50%	85%	15%	90%
LetaBot	61.43%	55%	45%		85%	90%	80%	35%	40%
ZZZKBot	29.29%	10%	5%	15%		50%	25%	75%	25%
Overkill	52.14%	55%	50%	10%	50%		40%	70%	90%
UAlbertaBot	45.00%	0%	15%	20%	75%	60%		75%	70%
MegaBot	37.86%	5%	85%	65%	25%	30%	25%		30%
Aiur	46.43%	70%	10%	60%	75%	10%	30%	70%

Luna	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	71.43%		70%	60%	80%	75%	75%	90%	50%
Iron	55.00%	30%		30%	55%	50%	80%	40%	100%
LetaBot	51.43%	40%	70%		65%	75%	65%	20%	25%
ZZZKBot	52.14%	20%	45%	35%		60%	25%	100%	80%
Overkill	48.57%	25%	50%	25%	40%		45%	85%	70%
UAlbertaBot	50.00%	25%	20%	35%	75%	55%		70%	70%
MegaBot	40.00%	10%	60%	80%	0%	15%	30%		85%
Aiur	31.43%	50%	0%	75%	20%	30%	30%	15%

Python	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	75.00%		60%	50%	85%	80%	70%	90%	90%
Iron	58.57%	40%		30%	55%	60%	85%	40%	100%
LetaBot	53.57%	50%	70%		20%	95%	65%	45%	30%
ZZZKBot	54.68%	15%	45%	80%		15%	26%	100%	100%
Overkill	52.86%	20%	40%	5%	85%		45%	75%	100%
UAlbertaBot	48.20%	30%	15%	35%	74%	55%		70%	60%
MegaBot	34.29%	10%	60%	55%	0%	25%	30%		60%
Aiur	22.86%	10%	0%	70%	0%	0%	40%	40%

The charts are full of small insights—more than I have time to examine. See for example how XelnagaII’s upset of Ziabot occurred on all maps except Ride of Valkyries; I’m sure that says something about at least one of those bots. We can tease out which pairings the map imbalances spring from. ZZZKBot did poorly on Tau Cross, as explained by Martin Rooijackers due to the long rush distance. And so on.

My strongest impression is how much results vary from map to map. I still think 5 maps are not enough to judge strength fairly. To my eye, the datapoint that stands out most is that ZZZKBot defeated the powerful Iron 100% of the time on Alchemist, in both the qualifier and the final, although otherwise Alchemist was a mediocre map for ZZZKBot. It looks as though Iron has a strategy bug on that map which ZZZKBot exploits. All bot authors who competed may want to eye the charts for hints about weaknesses to fix.

Download a zip file of the perl scripts with documentation.

Yesterday I claimed that the final stage of CIG 2016 produced little new information, because it was equivalent to drawing a subset from the qualifiers. Is it true? I wrote a script to render crosstables from subsets of game results.

Here’s my rendition of the real finals. I liked the red and green color coding of win rates in the original, but some people are red-green colorblind so my version has red and blue instead. I also went with a more contrasty color curve.

	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	65.14%		52%	44%	79%	71%	77%	83%	50%
Iron	54.43%	48%		38%	49%	49%	74%	30%	93%
LetaBot	53.71%	56%	62%		49%	81%	69%	30%	29%
ZZZKBot	53.08%	21%	51%	51%		42%	35%	93%	78%
Overkill	51.43%	29%	51%	19%	58%		43%	81%	79%
UAlbertaBot	49.07%	23%	26%	31%	65%	57%		76%	66%
MegaBot	38.00%	17%	70%	70%	7%	19%	24%		59%
Aiur	35.14%	50%	7%	71%	22%	21%	34%	41%

Here is the crosstable of the final hidden in the qualifier, which is to say the qualifier games played between finalists.

	overall	tscm	Iron	Leta	ZZZK	Over	UAlb	Mega	Aiur
tscmoo	61.71%		44%	48%	82%	53%	87%	75%	43%
Iron	56.57%	56%		39%	53%	56%	63%	38%	91%
LetaBot	52.00%	52%	61%		51%	81%	60%	28%	31%
ZZZKBot	52.14%	18%	47%	49%		44%	45%	93%	69%
Overkill	51.57%	47%	44%	19%	56%		32%	84%	79%
UAlbertaBot	48.29%	13%	37%	40%	55%	68%		48%	77%
MegaBot	42.86%	25%	62%	72%	7%	16%	52%		66%
Aiur	34.86%	57%	9%	69%	31%	21%	23%	34%

Overall results match closely. LetaBot and ZZZKBot have switched ranks, but that’s not a surprise because their scores were extremely close.

The 2 table cells with the largest differences are Tscmoo vs Overkill and MegaBot vs UAlbertaBot. The Tscmoo-Overkill numbers are within the expected range of statistical variation, according to spot checks with Fisher’s Exact Test, but the MegaBot-UAlbertaBot numbers are highly surprising, far outside the expected range. (The right way to do this would test both whole tables as a sample of samples of samples. :-) So there’s indication that something may be afoot.

I had a new thought. It’s theoretically possible that differences are caused by learning bots which generalize across opponents. Tscmoo and MegaBot are both learning bots (I verified it: they both wrote stuff to their learning files) and both seem as though they might be able to generalize across opponents. (Overkill is a learning bot but does not generalize.) So my original claim is not 100% true: The qualifiers don’t entirely duplicate the final in the presence of learning bots which generalize across opponents. Alternately, there could have been a problem with a big effect on that pairing (such as a bug in MegaBot related to its learning, an example which is equivalent to mis-generalizing across opponents). We have the source and the replays, so a sufficiently deep dig should turn up the issue if it is in the bots. There’s a chance that the issue is with the tournament operations, or with my script.

Here I combine the qualifier results with the final results to get the best numbers available. The organizers for whatever reason explicitly decided not to do this. Luckily, it doesn’t change the ranking of the bots.

	overall
tscmoo	63.43%
Iron	55.50%
LetaBot	52.86%
ZZZKBot	52.61%
Overkill	51.50%
UAlbertaBot	48.68%
MegaBot	40.43%
Aiur	35.00%

Tomorrow: More map analysis. Also I’ll release the script for others to play with.

I got ahead of myself yesterday—I should step back and talk about the CIG 2016 results more generally! Martin Rooijackers aka LetaBot sent me a few observations by e-mail. They mostly match up with my observations, and I’ll add a few of my own.

• Terran Renaissance confirmed, as predicted (probably by everybody who cared to predict).

• The top 3 winners, besides being terran, are all bots with many updates over the last several months.

• 3 bots of the final 8 are carryovers from past years (#5 Overkill, #6 UAlbertaBot, and #8 AIUR). They scored in the lower half. #4 ZZZKBot seems to have been only slightly updated. The long work put into the top 3 paid off in playing strength.

• Martin Rooijackers observes that #7 MegaBot is the highest-scoring brand new bot. It’s true if you count Iron as a continuation of Stone. And given MegaBot’s self-description as a meta-bot that uses the strategies of others, MegaBot is arguably not brand new either. In any case, the point is that it seems to take a long period of work to get to the top. The competition is fierce.

• None of the final 8 bots dominated the others. Even tail-ender AIUR had an equal record against winner Tscmoo and a winning record against LetaBot. The CIG 2016 finals crosstable has upsets throughout. Comparing to the AIIDE 2015 crosstable with 22 participants, the rate of upsets of bots near each other in rank seems visually similar, so with only 8 final bots the upsets run all the way through. Generally, bot #n is not clearly better than bot #n+1; the ranking is not stable at that level. In the qualifying stage, the rate of upsets visually looks steady down to #9 Tyr and then falls. AIIDE 2015 did not have that pattern.

• I predicted that ZZZKBot still had a chance to make it into the top 3. It didn’t, but it scored 53.08% to make #4 in the finals versus #3 LetaBot’s 53.71%. I think the prediction was justified. This was its last chance, though, without big updates.

• The qualifier results and finals results look different. Iron was narrowly on top in the qualifiers, but Tscmoo pulled well ahead in the finals (a surprise to me). Apparently Tscmoo is better tuned to defeat strong opponents.

• The slides on the result page include a chart of win rates over time which shows that learning helps some, but (as in the past) not as much as you’d hope. To learn more we need smarter learning. I’ll drop a few suggestions in a future post.

The bottom line is that we’re making good progress, though we’re still not far along the path. Tscmoo’s long short term memory is a pioneering idea and Tscmoo finished #1, but we don’t know much about it. Did the memory help results? Meanwhile, LetaBot finished #3 here, and is in a strong position as Martin Rooijackers tries to pioneer a next step in another direction, a tactical search derived from MaasCraft. Will the search lead to the hoped-for jump in strength? Tune in next time!

I question the tournament design. They ran a 100-round round robin with 16 bots and used the results to accept half of the entrants into the final—a staged design with qualifier and finals. That’s perfectly reasonable; it says that they’re more interested in who beats the strong than who consistently beats the weak. Having selected the finalists, they discarded the qualifier results and ran an independent final with 100 more rounds on the same maps for the 8 finalists. They even discarded bot learning files from the qualifier, so that nothing carried over. The final duplicated the qualifiers, only with fewer bots, and produced little new information. They could have saved the time and extracted the final results from the qualifier stage. It would have been equivalent.

In a staged tournament, each stage should produce new information. It could add to the qualifier results. It could have more rounds. It could include seeded opponents that skipped the qualifiers (though I wouldn’t recommend that for an academic tournament). It could include different maps. It could follow harsher rules. But something!

I can understand why they didn’t pass the qualifier results through to the final stage. They had the software they had, and an organizer’s time is always short. But this final had no point. I hope future tournaments will remember the lesson.

CIG 2016 reported its results in the same format as AIIDE 2015 (I’m sure they used the same software), so I was able to compute the map balance with a few adjustments to my script. The tournament was run in two halves, qualifiers and finals, each with 100 rounds. With 5 maps, that makes 20 times through the map pool. They could have used twice as many maps without any disadvantage that I see.

The qualifiers, with 16 bots playing 12,000 games total (minus a few lost to errors):

map	TvZ		ZvP		PvT
	wins	n	wins	n	wins	n
(2)RideofValkyries.scx	49%	640	61%	240	57%	480
(3)Alchemist.scm	50%	640	45%	240	60%	479
(3)TauCross.scx	56%	640	43%	240	53%	479
(4)LunaTheFinal.scx	53%	637	47%	240	53%	480
(4)Python.scx	49%	638	45%	240	50%	478
overall	51%	3195	48%	1200	55%	2396

The 3 races came out remarkably even! We already know that’s more due to the strength distribution of bots in the tournament than to the fairness of the game. The low-high spread in TvZ was 56%-49% = 7%; in ZvP 18%, and in PvT 7%. Ride of Valkyries had strikingly different ZvP results than the other maps. I don’t know why. Can anybody guess? The human balance also showed one map standing out in ZvP, but it was Alchemist.

The final, with 8 bots playing 2800 games, looks considerably different:

map	TvZ		ZvP		PvT
	wins	n	wins	n	wins	n
(2)RideofValkyries.scx	54%	120	92%	80	45%	120
(3)Alchemist.scm	52%	120	79%	80	63%	120
(3)TauCross.scx	76%	120	65%	80	49%	120
(4)LunaTheFinal.scx	67%	120	84%	80	46%	120
(4)Python.scx	66%	120	94%	80	34%	120
overall	63%	600	83%	400	48%	600

Here, protoss did poorly because the protoss bots came out on the bottom this time. It’s interesting that the middle-of-the-table zergs did more to hold down the protoss than the winning terrans (but it fits with the game storyline :-). Beyond that, I’m reluctant to draw conclusions from this smaller number of games with fewer players.

I feel vindicated: Map balance can make a difference, even though we don’t understand what the difference is!

The new protoss bot ZerGreenBot was uploaded at SSCAIT today. It describes itself as “terribad” and... I can’t disagree, but it’s fun. To defend its base it builds zealots and dragoons. These units only leave the base if they are lured out. To attack it sends a shuttle with 2 reavers. It seems to keep building shuttles and reavers from one robo, so whether the first lives or dies, more will fly out later to attack independently.

It never expands. It doesn’t scout until the shuttle flies around the map. If it happens to see the enemy natural first, it never seems to realize that the enemy must have a main too. The shuttle disregards danger. Sometimes it drops the reavers near their target, but on the wrong side of a cliff. In one game ZerGreenBot took a couple potshots at an unfinished spire, a good first target, but then moved on—and the attack was later cleared by the first mutalisk. And yet its manic shuttle-reaver micro is fun! The basic procedure seems to be drop, fire at whatever’s near, pick up, move a little around the outside of the base, drop, etc. It’s as if it were trying to duplicate the Berkeley Overmind’s “dismantle the enemy base from the outside in” tactics. When there are two shuttles, they do a wacky dance.

One thing the bot does right is that it keeps the shuttle always moving, so that it never has to accelerate from a stop. I take that as a sign that the author understands shuttle-reaver micro and merely hasn’t implemented much of it yet (because it is crazy hard).

In the games I’ve seen so far, opponents react poorly to the reaver drop. They don’t understand that the shuttle is a high priority target, and they don’t know how to escape or how to attack. Even with no other improvements, better shuttle-reaver control by itself might make ZerGreenBot a dangerous opponent for many bots, though probably not for the top tier. If the goal is to play strongly, I suggest this order of improvements: 1. Better choice of targets and drop locations. 2. Attention to avoiding danger. 3. Smarter scouting, so that the better choice of targets bites harder. And only then work on expanding and being more aggressive with the other units. Well, it’s only my first thought; I’m sure the author knows better than I do.

By the way, I think the name is a joke. “Zerg-reen” sounds like a zerg marine, everything that is not protoss.

I started to think about fancy ways to normalize map balance data so that the numbers could be compared—and then I realized, who the hell cares? The data’s not good enough in the first place, at least the bot data, which is based on only 21 bots with idiosyncratic play styles and big race imbalances regardless of the maps. We can only get a general idea of the comparison anyway.

So I decided on a simple subtraction of the average from each map balance number, so that a map with average balance has normalized balance 0%. Then we can compare maps to see if they have similar relative balance for pros and bots. After normalization, TvZ > 0 means that terran did better than average on that map, and TvZ < 0 means that terran did worse.

map	TvZ		ZvP		PvT
	pro	bot	pro	bot	pro	bot
Benzene	10.8%	-2.9%	-5.3%	1.3%	-5.1%	-0.4%
Destination	-1.0%	-1.9%	2.6%	2.3%	0.7%	-0.4%
Heartbreak Ridge	-4.7%	3.1%	2.2%	-0.7%	5.3%	-3.4%
Aztec	-14.3%	-1.9%	-4.4%	1.3%	11.6%	-0.4%
Tau Cross	-3.3%	-0.9%	-4.4%	-0.7%	-1.8%	-0.4%
Andromeda	-10.6%	1.1%	4.4%	-1.7%	3.8%	-4.4%
Circuit Breaker	-0.4%	-0.9%	-2.6%	-1.7%	-0.8%	2.6%
Empire of the Sun	10.9%	-4.9%	-4.4%	-1.7%	-2.7%	0.6%
Fortress	11.0%	8.1%	12.3%	-0.7%	-2.6%	4.6%
Python	1.9%	1.1%	-0.5%	2.3%	-8.0%	1.6%

There’s no “overall” row because, after normalization, it’s just a row of zeroes. Also, as I mentioned, the sizes of the imbalances can’t be compared directly. A relative balance of -5% in the bot ZvP column (average balance 71%) doesn’t mean the same thing as -5% in the pro ZvP column (average balance 54.4%).

No convincing pattern is visible. The pro and bot columns have the same sign in 12 cases, which is not distinguishable from 50% (15 cases). Sometimes a pro map with a large imbalance has a large imbalance for bots too; sometimes not. Here’s a scatter chart with relative pro balance on the x-axis and relative bot balance on the y. Remember that the signs are arbitrary: We arbitrarily chose to compare TvZ rather than ZvT, so + and - were chosen arbitrarily. If your eyes think they see a pattern, flip one or two of the symbol sets around one axis or the other before you decide it’s real.

What does it all mean in practice? There are some maps with apparent imbalances, which means we should have map pools large enough that imbalances tend to average out. Most maps are not far from balanced, so 10 maps should be enough; the 5 maps of CIG 2016 do not seem enough. Other than that, there’s no reason to change how we select maps. We don’t know whether last year’s relative map balances will carry over to this year, when the skill of the top bots is greater and they are terran rather than zerg. The main conclusion is the same as the conclusion of all studies since the invention of science: More research is needed!

I wrote Ye Usualle Little Perl Script to calculate map balance in AIIDE 2015, based on the the detailed game results (the “plaintext” link on that page). The results do not tell us what race random UAlbertaBot got each game, so its results don’t count in the analysis. UAlbertaBot was the only random bot.

map	TvZ		ZvP		PvT
	wins	n	wins	n	wins	n
(2)Benzene.scx	18%	405	72%	315	64%	567
(2)Destination.scx	19%	405	73%	315	64%	567
(2)HeartbreakRidge.scx	24%	405	70%	315	61%	567
(3)Aztec.scx	19%	405	72%	315	64%	567
(3)TauCross.scx	20%	405	70%	315	64%	567
(4)Andromeda.scx	22%	405	69%	315	60%	567
(4)CircuitBreaker.scx	20%	405	69%	315	67%	567
(4)EmpireoftheSun.scm	16%	405	69%	315	65%	567
(4)Fortress.scx	19%	405	70%	315	69%	567
(4)Python.scx	22%	405	73%	315	66%	567
overall	20%	4050	71%	3150	64%	5670

In the table, n is the total number of games in the matchup, one of several crosschecks to make sure the analysis is right. The tournament had 5 zerg, 7 protoss, and 9 terran bots (plus random UAlbertaBot, which was not counted, making 22 participants). There were 90 rounds, each on one map (which over 10 maps means 9 times through the map pool). So for TvZ there should be 5*9*90 = 4050 games; for ZvP 5*7*90 = 3150 games; for PvT 7*9*90 = 5670 games. Good.

OK, from this exercise I learned more about race balance in this tournament than about map balance. Zerg came out on top because zerg bots won. Meanwhile terran bots were concentrated toward the bottom of the crosstable, while protoss were scattered throughout. Zerg crushed protoss 2:1 but annihilated terran 5:1. I had not realized that it was so extreme. The maps made small differences, the bots made big differences.

Bots analyze maps shallowly and try to play about the same on different maps. I had expected that that lack of adaptivity would cause maps to affect results strongly: Adapting means that the bot matters more; failing to adapt means that the map matters more. But if so, it’s not visible in this table. Maybe the maps are standardized enough that adaptation doesn’t matter at this level of play. Or maybe my original thinking is wrong, and adaptation is what allows the map to matter—Heartbreak Ridge has a narrow base entrance, so that you can easily block your enemy in or out, and high ground over the natural to proxy on, and I haven’t seen any bot take advantage of those features.

You can download the AIIDE 2015 map balance analysis script in a .zip file. I ran it on a *nix but it can probably be adapted to run under Windows with no more than a tweak or two.

Next: I’ll try to normalize the results and compare human map balance to bot map balance in relative terms. Though you can get an idea already by eyeballing the tables.

Here’s the map balance table for AIIDE 2015. As yesterday, these per-matchup statistics are for pro players and are copied from TLPD.

map	TvZ	ZvP	PvT
Benzene	64.1%	49.1%	48.7%
Destination	52.3%	57%	54.5%
Heartbreak Ridge	48.6%	56.6%	59.1%
Aztec	39%	50%	65.4%
Tau Cross	50%	50%	52%
Andromeda	42.7%	58.8%	57.6%
Circuit Breaker	52.9%	51.8%	53%
Empire of the Sun	64.2%	50%	51.1%
Fortress	64.3%	66.7%	51.2%
Python	55.2%	53.9%	45.8%
overall	53.3%	54.4%	53.8%

With 10 maps to average over, the balance looks close enough to be fair. Some individual maps have large imbalances, but they mostly even out over the map pool. They don’t completely even out, though, because imbalances are too consistent across maps; there aren’t enough counterbalancing maps.

Of these 10 maps, only 2 (Tau Cross and Python) overlap with the 5 CIG 2016 maps.

Human balance and bot balance should be different. Next: I’ll try to investigate the bot balance in practice, using the AIIDE 2015 game results. Per-matchup numbers can’t be deduced from any of the summary tables, so I’ll have to go back to the raw game results. Will human and bot balance be somewhat similar, or all different?

Map balance is hard.

Only about 5 competition maps have stats showing balance within a few percent of equal for all matchups. Seriously! That’s less than 2% of maps ever used in pro play! (Though to be fair, the total includes maps without enough games for us to know the balance.) The closest are the popular Fighting Spirit, Circuit Breaker, and Tau Cross, and the less-popular Arcadia 2 and Neo Aztec. If you want a balanced map pool beyond these 5 maps, you have to balance the maps against each other: “This one is T>P by 10%, so the rest should add up to P>T by 10%.” Of course those are human stats, and bot balance should be different, so you might want to balance using bot data.

The AIIDE and CIG rules both say that maps will be chosen at random from a larger pool. SSCAIT says its maps are selected from popular recent pro maps, and doesn’t mention balance. So I decided to look into it.

For today I calculated the balance of the CIG 2016 map pool, 5 maps randomly selected from a larger collection. Think of this as a first check to see how balance may come out when you’re not paying attention.

• (2)RideofValkyries1.0
• (3)Alchemist1.0
• (3)TauCross1.1
• (4)LunaTheFinal2.3
• (4)Python1.3

I used balance numbers from the TLPD map database, which gives statistics for pro games played from 1999 to 2012. It’s not a definitive current pro balance, but it should be pretty good and it was complete and easy to use. Alchemist is not often played (presumably because it is grossly Z>P; also, according to Liquipedia “Alchemist is mostly noted for being a poor attempt at an asymmetrical three-player map”) and its stats are based on only 53 games. The % number in each cell is the winning rate for the first race in the matchup over each column.

map	TvZ	ZvP	PvT
Ride of Valkyries	48.5%	67.1%	54.4%
Alchemist	55.6%	80%	62.5%
Tau Cross	50%	50%	52%
Luna the Final	53.2%	60.2%	60%
Python	55.2%	53.9%	45.8%
overall	52.5%	62.2%	54.9%

I’d say that’s a substantial Z>P imbalance.

The numbers from TLPD are raw outcomes, with no attempt to adjust for the strength of the players. That’s likely good enough; it should average out over the large number of games played on most of these maps. But if we want to compare the pro balance with the bot balance after the tournament is over, we may want to do some normalization of both data sets. I’m predicting that this tournament will be dominated by terran bots. A comparison might give the impression that the maps are T>P and T>Z for bots, when in fact the terran bots were playing better.

Tomorrow: AIIDE 2015 map balance.

I will never run a tournament. I don’t have the stomach for that much administrative work (and hats off to those who do!). So it’s perfectly safe for me to offer advice—I know I’ll never have to listen to it myself.

The way I see it, one goal of tournaments is to prod bots to improve; tournaments motivate. Another goal is to measure progress; tournament organizers are happy to include older bots that have competed in past tournaments, to see how they do against newer competition. There’s some tension between the two goals, but you don’t want to compromise either of them too much.

Earlier I suggested changing timeout rules to prod the winner to finish the game. Another way to prod bots is to make them play on new maps that present different challenges. Unfortunately, most of the concept maps that I talked about seem too hard for current bots (and the novelty maps are not suitable for competitions). Exception: The map Fantasy is not too hard, but it’s too subtle. Stepping down a level, I don’t know any current bot that can play on an island map. Even ignoring balance issues, a tournament would not want to include an island map like Charity, or even a semi-island map like Indian Lament, because it would break the goal of measuring progress. Bots that were made able to play the maps would likely score 100% against bots that could not.

There is a compromise. I suggest the map Namja Iyagi, a land map with 4 mineral-only islands (one in the corner behind each main base) and 2 mineral-and-gas islands. A bot with island skills would have a large advantage over a bot without island skills (the prod)—but not necessarily a decisive advantage. Two bots with no island skills could still play sound games against each other. If Namja Iyagi is only one map out of several, the tournament results remain a fair measure of progress.

The map Return of the King has 4 islands, so it might be a gentler prod.

Another prod that would be good is a map that promotes (but does not require) pushing through minerals or mineral-walking through obstacles, as in some of the concept maps. I’m not sure what a good choice would be, though.

A Team Liquid thread RFC: BW AI Bot Ladder proposes a much fancier attempt to encourage progress.

Tomorrow: Map balance.