AIIDE 2018 - race balance | Starcraft AI blog

AIIDE 2018 - race balance

This post is about race balance in AIIDE 2018, but first I want to show my version of the crosstable, because mine is minutely different from the official results. My version includes exactly one game that the official results skip, game 11821 in round 33 between AILien and CSE, recorded as a crash for CSE after 23400 frames. The only other games omitted from the official results are games that failed to start and are recorded as having a length of -1 frames. I have no idea why this one game was dropped when other crash games were kept. In any case, the effect is that my win rates for CSE and AILien are different from the official rates by a small amount, and that affects my other statistics, though the difference is only one game and it’s hardly noticeable.

	overall	SAID	Cher	CSE	Blue	Locu	ISAM	DaQi	McRa	Iron	ZZZK	Stea	Micr	Last	Tyr	Meta	Leta	Arra	Ecgb	UAlb	Ximp	CDBo	Aiur	Kill	Will	AILi	CUNY	Hell
SAIDA	95.91%		83% 83/100	93% 93/100	97% 97/100	96% 96/100	89% 89/100	95% 95/100	86% 86/100	98% 98/100	97% 97/100	97% 97/100	100% 100/100	100% 100/100	100% 100/100	96% 90/94	98% 98/100	100% 100/100	100% 100/100	87% 87/100	98% 98/100	96% 96/100	95% 95/100	96% 96/100	99% 99/100	100% 100/100	98% 94/96	100% 100/100
CherryPi	90.86%	17% 17/100		72% 72/100	86% 86/100	89% 89/100	96% 96/100	100% 100/100	85% 85/100	96% 96/100	91% 91/100	81% 80/99	91% 91/100	82% 82/100	96% 96/100	99% 92/93	98% 98/100	89% 89/100	99% 99/100	98% 98/100	99% 99/100	100% 100/100	99% 99/100	100% 100/100	100% 100/100	100% 100/100	100% 100/100	100% 100/100
CSE	87.08%	7% 7/100	28% 28/100		66% 66/100	68% 68/100	78% 78/100	84% 84/100	72% 71/99	98% 98/100	91% 91/100	98% 98/100	99% 99/100	100% 100/100	100% 100/100	94% 89/95	91% 91/100	100% 100/100	100% 100/100	98% 98/100	100% 100/100	99% 99/100	96% 96/100	100% 100/100	100% 100/100	99% 99/100	99% 97/98	100% 100/100
BlueBlueSky	81.48%	3% 3/100	14% 14/100	34% 34/100		61% 61/100	66% 66/100	92% 92/100	72% 72/100	98% 98/100	88% 88/100	97% 97/100	94% 94/100	100% 100/100	100% 100/100	64% 60/94	95% 95/100	92% 92/100	100% 100/100	89% 89/100	95% 95/100	96% 96/100	72% 72/100	97% 97/100	100% 100/100	100% 100/100	100% 92/92	100% 100/100
Locutus	81.01%	4% 4/100	11% 11/100	32% 32/100	39% 39/100		56% 56/100	76% 76/100	54% 54/100	88% 88/100	95% 95/100	97% 97/100	96% 96/100	100% 100/100	97% 97/100	94% 90/96	94% 94/100	97% 97/100	100% 100/100	94% 94/100	94% 94/100	100% 100/100	95% 95/100	98% 98/100	100% 100/100	98% 98/100	100% 90/90	100% 100/100
ISAMind	78.46%	11% 11/100	4% 4/100	22% 22/100	34% 34/100	44% 44/100		63% 63/100	49% 49/100	98% 98/100	89% 89/100	96% 96/100	93% 93/100	100% 100/100	95% 95/100	88% 79/90	92% 92/100	98% 98/100	100% 100/100	95% 95/100	93% 93/100	99% 99/100	82% 82/100	97% 97/100	100% 100/100	100% 100/100	100% 96/96	100% 100/100
DaQin	72.39%	5% 5/100	0% 0/100	16% 16/100	8% 8/100	24% 24/100	37% 37/100		42% 42/100	92% 92/100	87% 87/100	99% 99/100	96% 95/99	96% 96/100	99% 99/100	61% 58/95	95% 95/100	99% 99/100	100% 100/100	88% 88/100	73% 73/100	97% 97/100	81% 81/100	98% 98/100	100% 100/100	97% 97/100	93% 89/96	100% 100/100
McRave	65.74%	14% 14/100	15% 15/100	28% 28/99	28% 28/100	46% 46/100	51% 51/100	58% 58/100		55% 55/100	72% 72/100	63% 63/100	79% 79/100	96% 96/100	88% 88/100	51% 49/97	74% 74/100	99% 99/100	96% 96/100	72% 71/99	41% 41/100	100% 100/100	77% 77/100	58% 58/100	74% 74/100	83% 83/100	92% 89/97	100% 100/100
Iron	63.79%	2% 2/100	4% 4/100	2% 2/100	2% 2/100	12% 12/100	2% 2/100	8% 8/100	45% 45/100		41% 41/100	73% 73/100	86% 86/100	91% 91/100	94% 94/100	99% 85/86	66% 66/100	94% 94/100	100% 100/100	91% 91/100	99% 99/100	88% 88/100	97% 97/100	85% 85/100	100% 100/100	93% 93/100	91% 87/96	100% 100/100
ZZZKBot	51.13%	3% 3/100	9% 9/100	9% 9/100	12% 12/100	5% 5/100	11% 11/100	13% 13/100	28% 28/100	59% 59/100		59% 59/100	57% 57/100	55% 55/100	35% 35/100	80% 68/85	40% 40/100	83% 83/100	25% 25/100	86% 86/100	100% 100/100	72% 72/100	89% 89/100	58% 58/100	92% 92/100	73% 73/100	85% 77/91	99% 99/100
Steamhammer	50.99%	3% 3/100	19% 19/99	2% 2/100	3% 3/100	3% 3/100	4% 4/100	1% 1/100	37% 37/100	27% 27/100	41% 41/100		24% 24/100	25% 25/100	53% 53/100	57% 50/88	74% 74/100	57% 57/100	96% 96/100	89% 89/100	79% 79/100	84% 84/100	88% 88/100	91% 91/100	93% 93/100	83% 83/100	95% 91/96	100% 100/100
Microwave	50.46%	0% 0/100	9% 9/100	1% 1/100	6% 6/100	4% 4/100	7% 7/100	4% 4/99	21% 21/100	14% 14/100	43% 43/100	76% 76/100		57% 57/100	52% 52/100	72% 66/92	80% 80/100	68% 68/100	73% 73/100	81% 81/100	65% 65/100	78% 78/100	81% 81/100	91% 91/100	86% 86/100	51% 50/99	98% 90/92	100% 100/100
LastOrder	49.23%	0% 0/100	18% 18/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	4% 4/100	4% 4/100	9% 9/100	45% 45/100	75% 75/100	43% 43/100		57% 57/100	88% 87/99	75% 75/100	83% 83/100	4% 4/100	95% 95/100	85% 85/100	89% 89/100	95% 95/100	89% 89/100	35% 35/100	92% 92/100	96% 95/99	100% 100/100
Tyr	44.60%	0% 0/100	4% 4/100	0% 0/100	0% 0/100	3% 3/100	5% 5/100	1% 1/100	12% 12/100	6% 6/100	65% 65/100	47% 47/100	48% 48/100	43% 43/100		20% 20/98	61% 61/100	96% 96/100	46% 46/100	58% 58/100	98% 98/100	100% 100/100	50% 50/100	94% 94/100	49% 49/100	62% 62/100	94% 88/94	100% 100/100
MetaBot	44.42%	4% 4/94	1% 1/93	6% 6/95	36% 34/94	6% 6/96	12% 11/90	39% 37/95	49% 48/97	1% 1/86	20% 17/85	43% 38/88	28% 26/92	12% 12/99	80% 78/98		41% 32/78	62% 56/91	86% 79/92	55% 52/94	40% 38/94	68% 64/94	73% 62/85	49% 47/95	91% 85/93	65% 62/95	87% 80/92	99% 87/88
LetaBot	37.80%	2% 2/100	2% 2/100	9% 9/100	5% 5/100	6% 6/100	8% 8/100	5% 5/100	26% 26/100	34% 34/100	60% 60/100	26% 26/100	20% 20/100	25% 25/100	39% 39/100	59% 46/78		42% 42/100	76% 76/100	41% 41/100	78% 78/100	44% 44/100	36% 36/100	53% 53/100	23% 23/100	92% 92/100	89% 67/75	100% 100/100
Arrakhammer	37.24%	0% 0/100	11% 11/100	0% 0/100	8% 8/100	3% 3/100	2% 2/100	1% 1/100	1% 1/100	6% 6/100	17% 17/100	43% 43/100	32% 32/100	17% 17/100	4% 4/100	38% 35/91	58% 58/100		69% 69/100	43% 43/100	71% 71/100	77% 77/100	54% 54/100	59% 59/100	94% 94/100	63% 63/100	100% 95/95	100% 100/100
Ecgberht	36.72%	0% 0/100	1% 1/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	4% 4/100	0% 0/100	75% 75/100	4% 4/100	27% 27/100	96% 96/100	54% 54/100	14% 13/92	24% 24/100	31% 31/100		58% 58/100	79% 79/100	38% 38/100	57% 57/100	66% 66/100	58% 58/100	84% 84/100	95% 83/87	95% 95/100
UAlbertaBot	34.71%	13% 13/100	2% 2/100	2% 2/100	11% 11/100	6% 6/100	5% 5/100	12% 12/100	28% 28/99	9% 9/100	14% 14/100	11% 11/100	19% 19/100	5% 5/100	42% 42/100	45% 42/94	59% 59/100	57% 57/100	42% 42/100		48% 48/100	45% 45/100	64% 64/100	75% 75/100	41% 41/100	70% 70/100	83% 78/94	98% 98/100
Ximp	32.61%	2% 2/100	1% 1/100	0% 0/100	5% 5/100	6% 6/100	7% 7/100	27% 27/100	59% 59/100	1% 1/100	0% 0/100	21% 21/100	35% 35/100	15% 15/100	2% 2/100	60% 56/94	22% 22/100	29% 29/100	21% 21/100	52% 52/100		3% 3/100	68% 68/100	84% 84/100	96% 96/100	64% 64/100	84% 71/85	94% 94/100
CDBot	31.98%	4% 4/100	0% 0/100	1% 1/100	4% 4/100	0% 0/100	1% 1/100	3% 3/100	0% 0/100	12% 12/100	28% 28/100	16% 16/100	22% 22/100	11% 11/100	0% 0/100	32% 30/94	56% 56/100	23% 23/100	62% 62/100	55% 55/100	97% 97/100		23% 23/100	62% 62/100	78% 78/100	66% 66/100	89% 79/89	93% 93/100
Aiur	31.56%	5% 5/100	1% 1/100	4% 4/100	28% 28/100	5% 5/100	18% 18/100	19% 19/100	23% 23/100	3% 3/100	11% 11/100	12% 12/100	19% 19/100	5% 5/100	50% 50/100	27% 23/85	64% 64/100	46% 46/100	43% 43/100	36% 36/100	32% 32/100	77% 77/100		27% 27/100	61% 61/100	38% 38/100	76% 65/85	96% 96/100
KillAll	29.64%	4% 4/100	0% 0/100	0% 0/100	3% 3/100	2% 2/100	3% 3/100	2% 2/100	42% 42/100	15% 15/100	42% 42/100	9% 9/100	9% 9/100	11% 11/100	6% 6/100	51% 48/95	47% 47/100	41% 41/100	34% 34/100	25% 25/100	16% 16/100	38% 38/100	73% 73/100		39% 39/100	70% 70/100	98% 94/96	95% 95/100
WillyT	27.76%	1% 1/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	26% 26/100	0% 0/100	8% 8/100	7% 7/100	14% 14/100	65% 65/100	51% 51/100	9% 8/93	77% 77/100	6% 6/100	42% 42/100	59% 59/100	4% 4/100	22% 22/100	39% 39/100	61% 61/100		43% 43/100	91% 85/93	100% 100/100
AILien	27.04%	0% 0/100	0% 0/100	1% 1/100	0% 0/100	2% 2/100	0% 0/100	3% 3/100	17% 17/100	7% 7/100	27% 27/100	17% 17/100	49% 49/99	8% 8/100	38% 38/100	35% 33/95	8% 8/100	37% 37/100	16% 16/100	30% 30/100	36% 36/100	34% 34/100	62% 62/100	30% 30/100	57% 57/100		96% 87/91	100% 100/100
CUNYBot	9.84%	2% 2/96	0% 0/100	1% 1/98	0% 0/92	0% 0/90	0% 0/96	7% 7/96	8% 8/97	9% 9/96	15% 14/91	5% 5/96	2% 2/92	4% 4/99	6% 6/94	13% 12/92	11% 8/75	0% 0/95	5% 4/87	17% 16/94	16% 14/85	11% 10/89	24% 20/85	2% 2/96	9% 8/93	4% 4/91		95% 80/84
Hellbot	1.36%	0% 0/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	1% 1/100	0% 0/100	0% 0/100	0% 0/100	0% 0/100	1% 1/88	0% 0/100	0% 0/100	5% 5/100	2% 2/100	6% 6/100	7% 7/100	4% 4/100	5% 5/100	0% 0/100	0% 0/100	5% 4/84

getting back to the subject at hand...

Here is the overall race balance. The only random player was UAlbertaBot, so that row and column reflect only UAlbertaBot’s results.

	overall	vT	vP	vZ	vR
terran	52%		45%	60%	67%
protoss	57%	55%		63%	67%
zerg	43%	40%	37%		62%
random	35%	33%	33%	38%

The top winner was terran and the runner-up was zerg, but the Locutusoid mass still makes it look like a protoss world. As always, race balance reflects more who the entrants are than anything else.

Here is the more interesting table, how each bot performed against each race.

#	bot	race	overall	vT	vP	vZ	vR
1	SAIDA	terran	95.91%	99%	95%	97%	87%
2	CherryPi	zerg	90.86%	82%	93%	93%	98%
3	CSE	protoss	87.08%	79%	86%	91%	98%
4	BlueBlueSky	protoss	81.48%	79%	76%	88%	89%
5	Locutus	protoss	81.01%	77%	74%	89%	94%
6	ISAMind	protoss	78.46%	80%	67%	88%	95%
7	DaQin	protoss	72.39%	78%	54%	86%	88%
8	McRave	protoss	65.74%	63%	57%	76%	72%
9	Iron	terran	63.79%	67%	50%	74%	91%
10	ZZZKBot	zerg	51.13%	44%	43%	61%	86%
11	Steamhammer	zerg	50.99%	59%	39%	58%	89%
12	Microwave	zerg	50.46%	51%	37%	63%	81%
13	LastOrder	zerg	49.23%	25%	39%	70%	95%
14	Tyr	protoss	44.60%	32%	29%	65%	58%
15	MetaBot	protoss	44.42%	45%	44%	44%	55%
16	LetaBot	terran	37.80%	34%	33%	44%	41%
17	Arrakhammer	zerg	37.24%	45%	26%	46%	43%
18	Ecgberht	terran	36.72%	20%	28%	51%	58%
19	UAlbertaBot	random	34.71%	33%	33%	38%	-
20	Ximp	protoss	32.61%	28%	33%	33%	52%
21	CDBot	zerg	31.98%	42%	23%	35%	55%
22	Aiur	protoss	31.56%	35%	30%	31%	36%
23	KillAll	zerg	29.64%	28%	26%	35%	25%
24	WillyT	terran	27.76%	30%	21%	31%	59%
25	AILien	zerg	27.04%	18%	27%	32%	30%
26	CUNYBot	zerg	9.84%	7%	15%	5%	17%
27	Hellbot	protoss	1.36%	1%	1%	2%	2%

At the top, it is interesting that #1 SAIDA had a little trouble versus random #19 UAlbertaBot, which CherryPi steamrolled, and #2 CherryPi had a little trouble versus terrans, which SAIDA annihilated. The upper protoss bloc, the Locutusoids plus #8 McRave, crushed all competition below them, with the worst matchup at 54%. Everybody in the top half was strong against zerg, even the zergs. (#11 Steamhammer was the weakest of them at ZvZ due to the micro bugs that are fixed in version 2.1.) Four zergs are closely grouped around 50%, but had different paths to get there. #15 MetaBot was equally good against all races, which I think is a sign that its 3 heads are well balanced. #18 Ecgberht was strikingly stronger against zergs than other races, reflecting its infantry first strategy. There are not many surprises in the table.

Trackbacks

No Trackbacks

Comments

Jay Scott on Saturday, November 17. 2018:

By the way, the overall column in the small race balance crosstable includes mirror matchups. Since the mirrors are always 50%, that tends to push the averages a bit toward 50%.

Tully on Sunday, November 18. 2018:

When did Cherry Pi get so strong? What is the source of its game-winning power?

Jay Scott on Sunday, November 18. 2018:

There are some hints in their survey answers; they were clearly aiming for #1. And they’ve had a year to work on it. Anyway, I’ll be looking into what they did.

Ecgberht on Sunday, November 18. 2018:

Regarding my bot's results It suffered from broken learning, same bug that Java bots had on CIG but this time It only affected my bot as TyrP had a proper "run_proxy.bat" file.

With no learning my bot will only play its 1-2 base Bio push instead of choosing between the 6-7 different strategies that probably work better on most matchups (BioMech, PureMech, Cheese, 2PortWraith, etc.).

Overall Im a bit sad because I think my bot could have done much better even If It wasnt a top bot.

Also Im probably taking a break and Im actually considering entering in this year's SSCAIT.

Thomas on Sunday, November 18. 2018:

The only reason cherry Pi looks weaker vs terran is SAIDA. Pi has over 95% vs every other terran bot. The reason SAIDA did so well in TvT is that it is the only strong terran.

DaveChurchill on Sunday, November 18. 2018:

That game result was not included because one of the TM Clients did not report back a result, so it is deemed not completely trustworthy. Maybe some lost packets or something

Jay Scott on Sunday, November 18. 2018:

A rare random failure somewhere, OK. The detailed results format seems to change every year, so it would be nice if it came with a little documentation about how to interpret it.

McRave on Monday, November 19. 2018:

Another well run AIIDE tournament, I'm disappointed in my results but it is what it is! Good to see SH performing near the top of the Z pack, I think we're both in the same shoes and have some catching up to do.

Add Comment

Name*

Homepage

Comment*

In reply to

E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA