AIIDE 2015 Bayesian Elo ratings

Krasi0 asked me to calculate ratings for tournaments using Rémi Coulom’s excellent bayeselo program. Here are ratings for AIIDE 2015.

bayeselo does not calculate basic Elo ratings like my little code snippets. It can’t calculate an Elo curve over time. It assumes that the players are fixed and have one true rating, and it crunches a full-on Bayesian statistical analysis to not only find the rating as accurately as possible, but also a 95% confidence interval so you can see how accurate the rating is. The ratings for the bots that learn, which aren’t fixed in strength as bayeselo assumes, can be seen as measuring the average strength over the tournament—the tournament score is no different in that respect.

The last column of the table is the probability of superiority, bayeselo’s calculated probability that the bot truly is better than the bot ranked immediately below it. The last bot doesn’t get one, of course. (bayeselo calculates this for all pairs, but in a tournament this long it rounds off to 100% for most.)

	bot	score	Elo	95% conf.	better?
1	tscmoo	89%	2026	2002-2050	81.0%
2	ZZZKBot	88%	2011	1988-2035	99.9%
3	UAlbertaBot	80%	1895	1874-1916	61.2%
4	Overkill	81%	1890	1870-1911	99.9%
5	Aiur	73%	1784	1765-1803	99.9%
6	Ximp	68%	1712	1694-1731	99.9%
7	Skynet	64%	1666	1648-1684	50.7%
8	IceBot	64%	1666	1648-1684	88.4%
9	Xelnaga	63%	1650	1632-1668	81.4%
10	LetaBot	61%	1638	1620-1656	99.9%
11	Tyr	54%	1553	1534-1572	96.0%
12	GarmBot	52%	1531	1513-1549	100%
13	NUSBot	39%	1380	1362-1398	73.1%
14	TerranUAB	38%	1372	1354-1390	99.8%
15	Cimex	36%	1335	1316-1353	99.6%
16	CruzBot	32%	1299	1280-1317	99.9%
17	OpprimoBot	28%	1231	1211-1250	96.7%
18	Oritaka	26%	1205	1185-1225	84.0%
19	Stone	25%	1190	1170-1210	91.3%
20	Bonjwa	23%	1171	1151-1191	100%
21	Yarmouk	9%	913	885-939	95.0%
22	SusanooTricks	8%	882	853-910	-

In the official results, Overkill came in ahead of UAlbertaBot with a higher tournament score. bayeselo ratings are more accurate than score because they take into account more information, and bayeselo says UAlbertaBot > Overkill with probability 61%. As explained in the original results, it’s a statistical tie, but bayeselo says it’s not an even tie but a little tilted in a counterintuitive way.

Skynet looks dead even with IceBot in the rounded-off numbers above. bayeselo says that Skynet > IceBot with probability 50.7%, a hair off dead even. Even the large number of games in this tournament could not rank all the bots accurately.

Tomorrow: The same for CIG 2016.

Trackbacks

No Trackbacks

Comments

tscmoo on Monday, October 3. 2016:

This is cool. Maybe this is how future tournaments should be ranked?

Jay Scott on Tuesday, October 4. 2016:

It would be more accurate, but only for deep technical reasons that are hard to understand. So... I think it would be a good idea, but it might be hard to justify.

Jay Scott on Tuesday, October 4. 2016:

People could reasonably object: The program is complicated and poorly documented. If it has a bug, or the tournament organizers misunderstand how to use it, then the results might be wrong.

imp on Tuesday, October 4. 2016:

to my knowledge there is hardly any (or even not a single one) program that gets the tournament ranking calculation right for chess tournaments. The official ranking algorithm is described in prosa and the implementation is surprisingly difficult if you consider every single corner case.
Still it would make a lot of sense to use a tournament ranking system borrowed from chess.

Jay Scott on Tuesday, October 4. 2016:

The bayeselo program is actually specialized for chess. It corrects for an empirical 32 elo advantage for the white player and has specific knowledge about the draw rate in chess. The only point that makes it unsuitable is that it can’t take prior ratings into account, because it doesn’t know how much evidence they represent. Luckily it also has commands to change its settings, so we can use it for other games!

imp on Wednesday, October 5. 2016:

it is one thing to calculate the elo used to seed players in a Swiss system tournament. It is quite a different thing to calculate the "correct" ranking of players in a tournament. If you're interested in the topic, check out e.g. Wikipedia for "Swiss System tie-breaking" and "Buchholz" as well as:

https://www.fide.com/component/handbook/?id=187&view=article

Jay Scott on Wednesday, October 5. 2016:

None of these tiebreak systems can equal the accuracy and fairness of bayeselo performance ratings. Bayeselo makes mathematically sound maximum likelihood estimates based on all the tournament information—use of maximum likelihood is its weakest point, and even there it is hard to suggest an improvement.

Jay Scott on Tuesday, October 4. 2016:

So, in other words, bayeselo should be excellent for ranking one tournament. It can’t support an ongoing rating system by itself.

krasi0 on Thursday, October 6. 2016:

Agreed. I think we should use it in the case of next year's SSCAIT tournament as well

Add Comment

Name*

Homepage

Comment*

In reply to

E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA