Rob Bogie’s MaasCraft has map problems

What is it with Rob Bogie’s SSCAIT version of MaasCraft (originally written by Dennis Soemers)? Why does it do so much better on some maps and so much worse on others, with far more variation than the rest of the bots?

I looked up MaasCraft’s results in the AIIDE 2014 and CIG 2014 competitions (it didn’t play in other years). Both map pools overlap with the SSCAIT map pool. CIG 2014 had an extra wide and diverse map pool, with some nonstandard maps, and it only overlaps a little. The overall row is across all maps in that competition, not only the ones in the table.

map	CIG 2014 win %	AIIDE 2014 win %	SSCAIT Elo diff
Andromeda		63%	-361
Benzene		55%	+42
Circuit Breaker		52%	+246
Destination		59%	-313
Empire of the Sun		61%	-333
Fighting Spirit	50%		-418
Heartbreak Ridge		61%	-193
Icarus	50%		+291
Python		60%	-306
Tau Cross		57%	+365
overall	55%	59%	0

I’m comparing raw win percentages with Elo differences, but the pattern is clear—I mean, the lack of pattern. The 2014 tournament results look normal, the SSCAIT results look extreme, and the two are nothing alike.

CIG 2014 and AIIDE 2014 both released source code. The MaasCraft source is exactly the same in both. It looks as though some difference or bug is affecting only the SSCAIT version under Rob Bogie.

Later today: First thoughts on the AIIDE 2016 results.

Trackbacks

No Trackbacks

Comments

krasi0 on Friday, October 14. 2016:

Have you watched any replays played on the "problematic" maps? Perhaps it's due to some path blockage or perhaps the author has some knowledge hardcoded for the "good" maps and skipped the problematic ones?

Jay Scott on Friday, October 14. 2016:

Good idea. I’ll do that when I get some time.

Jay Scott on Friday, October 14. 2016:

It turns out that every Rob Bogie replay is on a “blue” map that Rob Bogie did well on. On the one hand, the bot is no Trump fan, so that’s good. But on the other hand—what the what??? The map selection is not looking random! What’s going on here?

Jay Scott on Friday, October 14. 2016:

Did it crash so badly on the other maps that no replay was recorded?

Jay Scott on Sunday, October 16. 2016:

Hypothesis verified. In the SSCAIT data file, for the 6 maps it loses on (Destination, Heartbreak Ridge, Andromeda, Empire, Fighting Spirit, Python) the scores are usually 0 for Rob Bogie and 0 for the opponent with a win for the opponent, which means that the game ended before the first battle. I assume Rob Bogie either overstepped the time limit or crashed. Occasionally Rob Bogie has a win with score 0 for itself and -100 for the opponent, which I take to mean that the opponent crashed on startup. On other maps,the scores look normal. MaasCraft does its own map analysis, and in the 2014 source code it has special cases hardcoded in for Andromeda and Fortress, to work around limitations in its map analysis. So it seems likely that the map analysis for Rob Bogie’s version crashes or is too slow on those maps.

krasi0 on Monday, October 17. 2016:

Yup, that explains it perfectly. Thanks! It would be nice for bots with inhouse pathfinding algorithms to be able fallback to something like BWTA(2) in the case of maps out of the "whitelist" (where the author's algorithm has been tested).

Add Comment

Name*

Homepage

Comment*

In reply to

E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA