Starcraft AI blog

I want to take a day off from analyzing AIIDE data to join in a conversation. From comments to Dragon vs Ecgberht:

Tully Elliston: Using learning to track win:loss %, and having a risk rating for each build (if win rate is above this level, don’t select this build unless it has won at least 1 game against this opponent) could actually be a very useful tool.

You can throw in lots of ridiculous polarised builds, and still ensure they won’t get accidentally selected when beating down a 4pool bot 100 times in a row.

MarcaDBAA: Yes, or give these builds, you don´t want to use at first, some default pseudo-losses, so that they will only be selected after other builds fail to win.

In BananaBrain versus Ecgberht I had mentioned that BananaBrain’s few losses to Ecgberht were due to unnecessarily playing a risky build. The commenters suggest two ways of marking builds as risky.

It can be done automatically from data, in the same style as opening timing data. In fact, it’s on my to-do list. The first step is to keep track of how good each opening is on average: For each matchup, store each opening’s average win rate across all opponents. It can be done offline by adding up the numbers from the individual learning files of each opponent, or you could keep separate records. That already gives you an automatic way to select openings against opponents you have not yet learned about; there’s no more need for hand configuration.

The next step is to compare how well each opening does against strong opponents versus weak opponents. If it reaches its average by beating expectations against strong opponents and falling below against weak opponents, taking into account the opponent’s strength, then it is a risky opening. If the reverse, it is a solid opening and is to be preferred against weak opponents (and if you’re ranked high, also against unknown opponents). One natural way to determine riskiness is to fit a line to the dataset this-opening’s-wins versus opponent-strength as measured by your win rates. The slope of the line tells how risky or solid the opening is. (If you have a lot of data you could fit a more complicated curve. Just make sure it strongly smooths the data.)

The same goes for other data about openings. For example, you can track how well each opening does on each map, and at given starting positions, and against different opponent strategies that you recognize. All the data can fold into your opening selection, without any hand configuration.

BASIL was formerly an excellent forum to collect this kind of data. But now the BASIL pairings are strongly biased toward opponents close in elo, so it is no longer a good option. Look at the crosstable for the last 30 days and notice how the white cells are laid out; unless you rank right in the middle, you can’t get a full cross-section of opponents without a long run.