On 5/24/2023 9:09 PM, Simon Woodhead wrote:
> Rather than right or wrong, the /size/ of the error is what is
> most interesting (to me at least).
The book 'Backgammon Super Genius Quiz' scores things both ways,
but the official ranking was based on 1 for correct and 0 for
incorrect, rather than on equity. I don't entirely agree with
the justification given in the book, though, which among other
things assumes that it's not possible to come up with enough
discriminatory problems with large equity differences. (The
Othello quiz is full of counterexamples, after all!)
There is a lot to be said for scoring based on the size of the
error, but there are two problems I see with that.
1. It means that for match play, you pretty much have to use EMG
to measure error size, and EMG has well-known problems.
2. Size of error depends too sensitively on the rollout settings,
choice of bot, etc. The Othello quiz is carefully designed so that
the correct answer is robust to these variations. You can go back
to the earliest Othello quiz and use Snowie, GNU, BGBlitz, or XG,
on anything but the very weakest settings, and they will all agree
on what the top play is. But if you ask them for the size of the
errors then they may disagree rather significantly.