random bots on BASIL
Today I revisited the BASIL stats page with its collected numbers. The random bots got me thinking.
Over the stats period, the random bots collectively scored 56.8%, making random the most successful choice of race. On BASIL, the server chooses the random race before the game starts so that the opponent knows it, so theoretically there should be no game play advantage to being random. There could still be an advantage in confusing the opponent by playing a wider range of strategies, each less often—or that could be a disadvantage, depending on how the learning of each side works. In practice, does playing random give an edge, or did stronger bots choose to play random?
It seems impossible to answer the question by comparing strength numbers between the randoms and the rest. Do the random bots have higher elo? Well, they scored better! The result says nothing about the direction of causation, if any.
Another line to answer the question is, do the random bots score better or worse than their component races playing independently? We can answer by averaging the elo ratings, because elo is designed so that strength differences can be added and subtracted. I compared PurpleDestiny, tscmoo, and ChimeraBot.
PurpleDestiny has elo 2273 at the moment, and I get 2333 for the mean elo of its 3 components. There is no evidence of an advantage for playing random while purple.
Tscmoo random is at 2348, while its components average to 2146. It may be that playing random is effective for cows (“moo!”). But it could also be that the apparent components aren’t actually part of tscmoo. They were updated at different times, and I expect that they are configured differently too, with different sets of strategies allowed. If so, the comparison may not mean anything.
ChimeraBot (made of Iron, McRave, and ZZZKBot by Chris Coxe) is at 2342, while its components average to 2330. But Iron and McRave have both been updated since ChimeraBot was assembled, and I believe the updates were improvements. If so, the true mean of the components should be lower, but in any case it is probably different. There is at best weak evidence of an advantage for playing random while chimeric.
The bottom line is that the result is inconclusive. I haven’t found enough information on BASIL to judge whether playing random is an advantage, or a disadvantage, or a wash.
By the way, the stats page says there are 18 random bots, but I count 11 on the ranking page (2 disabled). Does the ranking page not show all bots?
Trackbacks
No Trackbacks
Comments
Bytekeeper on :
* "Aggregate" bots are built using good "component" bots
* Random bots generally are made by authors with a lot of experience and strong "component" bots
* There are (almost?) no random bots done with weak bots, whereas there are plenty of single-race bots with bad performance
The stats page might be a bit confusing. It shows all bots that BASIL saw. The ranking page filters out bots that had no games at all. Maybe I should split up the table to reflect that?
What kind of information/statistics would you like to have for better judgement?
Jay Scott on :
Joseph Huang on :
Jay Scott on :
Of course BASIL hasn’t been around for that long yet, but if stats like this can be kept over years they become invaluable.
Jay Scott on :
Bryan Weber on :
If a randomhammer (when it happens to random T) outperforms randomhammer when it is locked as T, the gain in performance is attributable to being random. Release both into the basil ladder, run 2000 test games and find out.
Jay Scott on :
MarcoDBAA on :
2.Further, the most popular base bots (UAB and SH/RH) are random bots and at least mid tier (UAB) or higher.
3.There are weaker random bots, they just aren't active there . Sure, non random weak bots are also disabled. I think that the field (the same in SSCAIT) is a bit too elitist for new arrivals (which are not using established base bots especially) now anyway.
4. Random bots, that are just other bots (Chimera). Makes no sense the way I see it. If other people upload more of them...
5. There are possible mid tier random bots, that just aren´t uploaded as random (McRave) or disabled as random (MadMix).
I always thought, that the random bot option should be used to try crazy and/or really random strategies, instead of turning them into completely serious learning bots. RH does not count, because protoss and terran aren´t uploaded (yet?), but if you have 4 bots, the random one should be a "fun" bot really. Just entertain the people :P. Did not check out yet, if Purpledestiny does this maybe (because it is weaker)?
Dan on :
Joseph Huang on :