Actually, these things might be too small for FishTest to measure in a
reasonable amount of
time.
Which leads to another interesting and perhaps valuable idea:
"indirect measurement"!
I claim it is possible to measure some things much more accurately
than is feasible with FishTest.
1. we can measure speed very precisely, certainly 1 part per million,
probably billion. We know a relationship between elo and speed:
specifically, about 70 elo points per speed-doubling, i.e. about 1 elo
per 1% speed increase.
2. We can measure frequency of occurrence of various phenomena (such
as underpromotions, KBNk endings) pretty precisely. KBNk ending
allegedly occurs
1/5000 fraction of games. Underpromotions allegedly occur in about
1/1200 of games.
These numbers all are not due to me and I do not vouch for their accuracy.
So if SF could not mate with bishop & knight, that'd hurt it by about 1/10000 of
a full point per game, i.e. about 0.1 elo. If SF could not
underpromote (all types)
that would hurt it by somewhere around 0.5 elo. The speed benefit
from SF ignoring
all underpromotions would outweigh that if it were >0.5% speed gain.
We could measure the speed gain to much better accuracy than that.
I suspect about 0.1 to 1% of all searched moves are pawn-to-8th, and
hence the slowdown likely is of the correct order of magnitude to
outweigh underpromotion benefits, especially
if we only are talking about R&B underpromos.
So, this rough analysis has suggested, paradoxically, that knowing the
rules actually hurts stockfish. But the hurt is at most a few elo.
If a modification to SF impacts speed