crafty 12.6 evaluation strangeness
j...@yktvmv.watson.ibm.com
I played a blitz (40 moves per 5 minutes) game against crafty
12.6 in which crafty gave up a pawn early and then lost quickly. In
trying to see why crafty gave the pawn I found some strange behavior
for the evaluation function.
This is the position after I won the pawn with crafty's
evaluation.
......................................................................
Crafty v12.6
White(1): learn 0
book learning disabled
position learning disabled
White(1): in pscore3
White(1): setboard r1bq1rk/ppp1bppp/4p/4P/3PQ2P/4P/PPP3P/RNB1K2R/ b KQ
+---+---+---+---+---+---+---+---+
8 | *R| | *B| *Q| | *R| *K| |
+---+---+---+---+---+---+---+---+
7 | *P| *P| *P| | *B| *P| *P| *P|
+---+---+---+---+---+---+---+---+
6 | | | | | *P| | | |
+---+---+---+---+---+---+---+---+
5 | | | | | P | | | |
+---+---+---+---+---+---+---+---+
4 | | | | P | Q | | | P |
+---+---+---+---+---+---+---+---+
3 | | | | | P | | | |
+---+---+---+---+---+---+---+---+
2 | P | P | P | | | | P | |
+---+---+---+---+---+---+---+---+
1 | R | N | B | | K | | | R |
+---+---+---+---+---+---+---+---+
a b c d e f g h
Black(1): exit
Black(1): display
Black(1): score
clearing transposition table
clearing pawn hash tables
note: scores are for the white side
material evaluation................. 1.000
trade bonus/penalty................. 0.000
development......................... -0.250
pawn evaluation..................... -0.080
passed pawn evaluation.............. 0.000
passed pawn race evaluation......... 0.000
outside passed pawn evaluation...... 0.000
piece evaluation.................... -0.565
total evaluation.................... 0.105
.......................................................................
As you can see crafty thinks black has good compensation for
the pawn. This is questionable. However the really strange thing is
what happens if we move the white king from e1 to d1. This is the
new position with crafty's evaluation.
..................................................................
Black(1): in pscore3b
Black(1): setboard r1bq1rk/ppp1bppp/4p/4P/3PQ2P/4P/PPP3P/RNBK3R/ b
+---+---+---+---+---+---+---+---+
8 | *R| | *B| *Q| | *R| *K| |
+---+---+---+---+---+---+---+---+
7 | *P| *P| *P| | *B| *P| *P| *P|
+---+---+---+---+---+---+---+---+
6 | | | | | *P| | | |
+---+---+---+---+---+---+---+---+
5 | | | | | P | | | |
+---+---+---+---+---+---+---+---+
4 | | | | P | Q | | | P |
+---+---+---+---+---+---+---+---+
3 | | | | | P | | | |
+---+---+---+---+---+---+---+---+
2 | P | P | P | | | | P | |
+---+---+---+---+---+---+---+---+
1 | R | N | B | K | | | | R |
+---+---+---+---+---+---+---+---+
a b c d e f g h
Black(1): exit
Black(1): display
Black(1): score
note: scores are for the white side
material evaluation................. 1.000
trade bonus/penalty................. 0.000
development......................... -0.050
pawn evaluation..................... -0.080
passed pawn evaluation.............. 0.000
passed pawn race evaluation......... 0.000
outside passed pawn evaluation...... 0.000
piece evaluation.................... -0.365
total evaluation.................... 0.505
Black(1): quit
execution complete.
......................................................................
As you can see now crafty doesn't think so much of black's
compensation for the pawn. This seems really strange to me. Is
this working as intended or is there a bug somewhere?
James B. Shearer
Robert Hyatt
j...@yktvmv.watson.ibm.com wrote:
: I played a blitz (40 moves per 5 minutes) game against crafty
: Black(1): exit
the two things are unrelated. Look at the white king position. With
the King unmoved at e1, crafty evaluates the king safety as not so hot,
because the king-side is not very safe. But when you move the king to
d1, it is safer, because it is further away from the shredded kingside
pawns...
Also notice the -250 for a development penalty for not having
castled, but after you move the king, this is no longer applied.
The move that really moves the king gets a penalty during the
search, but after the search, it "disappears." This is part of
the "aggressive" behavior of Crafty. It is attempting to keep you
from castling. If it is successful, it might well win. If not,
the penalty for your weakened (or trapped in the center) king won't
mean a lot.
Also, notice that you are using 12.6, which clearly did have some
weak pawn bugs. I modified this code extensively in 12.7 and up,
although I wouldn't claim it is a "done deal" yet since I am still
working on it for Paris...
However, after looking at this game, if black plays f6, I'd rather have
the resulting black positionI think...
BTW, 13.0 thinks black is behind here, with an eval of -.6 or
so...
: As you can see now crafty doesn't think so much of black's
: compensation for the pawn. This seems really strange to me. Is
: this working as intended or is there a bug somewhere?
: James B. Shearer
As you noticed above, it was a simple combination of things, whether are
right or wrong is a topic for another discussion, of course. BTW, from
where you work, you ought to be picking on Deep Blue when you play. It
is probably right down the hall. :)
j...@yktvmv.watson.ibm.com
Following up on my own post, I have been looking at the source
to crafty 12.6 and I am somewhat puzzled.
It appears that the evaluation function depends on whether you
are in the opening phase of the game or not. It also appears that
positions input with setboard are never in the opening phase. This
would appear to mean that if I save a position from the opening phase
of a game (using savepos) and then read it in (using in) and analyze
it the results will be bogus because they will be using a different
evaluation function from that crafty was actually using in the game.
Is this correct?
Also the score command appears to always call the
EvaluateDevelopment subroutine although crafty actually only calls
this routine in the opening phase of the game. This would appear
to make the output of the score command misleading when you are not
in the opening phase (including for example any position input with
setboard). Is this correct?
James B. Shearer
Robert Hyatt
j...@yktvmv.watson.ibm.com wrote:
: Following up on my own post, I have been looking at the source
: to crafty 12.6 and I am somewhat puzzled.
: It appears that the evaluation function depends on whether you
: are in the opening phase of the game or not. It also appears that
: positions input with setboard are never in the opening phase. This
: would appear to mean that if I save a position from the opening phase
: of a game (using savepos) and then read it in (using in) and analyze
: it the results will be bogus because they will be using a different
: evaluation function from that crafty was actually using in the game.
: Is this correct?
At present, yes. I'm not sure why this is, but it will be fixed.
: Also the score command appears to always call the
: EvaluateDevelopment subroutine although crafty actually only calls
: this routine in the opening phase of the game. This would appear
: to make the output of the score command misleading when you are not
: in the opening phase (including for example any position input with
: setboard). Is this correct?
Yes. This has already been fixed in v13.0... and 13.1...
: James B. Shearer
j...@yktvmv.watson.ibm.com
Looking some more at the code, it appears to me that the
game phase (opening, middle, end) is determined at the root. Is
this correct? Doesn't this mean crafty is in fact doing some
root evaluation with the same potential problems you have been
citing for programs like fritz?
James B. Shearer
j...@yktvmv.watson.ibm.com
I had posted:
The above is wrong because I used setboard which meant that
crafty did not realize this is still the opening phase. If I use
read to enter the game then the score becomes:
note: scores are for the white side
material evaluation................. 1.000
trade bonus/penalty................. 0.000
development......................... -0.250
pawn evaluation..................... -0.150
passed pawn evaluation... .......... 0.000
passed pawn race evaluation......... 0.000
outside passed pawn evaluation...... 0.000
piece evaluation.................... -0.815
total evaluation.................... -0.215
So crafty 12.6 actually thinks it is ahead.
Robert Hyatt responded.
>the two things are unrelated. Look at the white king position. With
>the King unmoved at e1, crafty evaluates the king safety as not so hot,
>because the king-side is not very safe. But when you move the king to
>d1, it is safer, because it is further away from the shredded kingside
>pawns...
>
>Also notice the -250 for a development penalty for not having
>castled, but after you move the king, this is no longer applied.
>The move that really moves the king gets a penalty during the
>search, but after the search, it "disappears." This is part of
>the "aggressive" behavior of Crafty. It is attempting to keep you
>from castling. If it is successful, it might well win. If not,
>the penalty for your weakened (or trapped in the center) king won't
>mean a lot.
Actually if I use setboard to set the position (with white
to move) crafty will actually play kd1 as white at shallow depths.
This is because penalty for moving the king is only applied in the
opening phase which crafty does not realize it is in. However this
behavior would not occur in an actual game.
Robert Hyatt:
>However, after looking at this game, if black plays f6, I'd rather have
>the resulting black positionI think...
I'll take the pawn especially since crafty (at least 12.6)
does not play f6. It continued c5.
Robert Hyatt:
>BTW, 13.0 thinks black is behind here, with an eval of -.6 or
>so...
Perhaps I should get a later version. Is there one you think
is reasonably stable?
Robert Hyatt:
>As you noticed above, it was a simple combination of things, whether are
>right or wrong is a topic for another discussion, of course. BTW, from
>where you work, you ought to be picking on Deep Blue when you play. It
>is probably right down the hall. :)
Well, I like to win some of the time. Actually this is an
advantage for you. I expect Deep Blue would be stronger if more
people were allowed to try and pick on it.
James B. Shearer
Robert Hyatt
j...@yktvmv.watson.ibm.com wrote:
: Looking some more at the code, it appears to me that the
To a limited extent. But, quite simply, it simply turns off "evaluate
development"... but that procedure turns itself off too so it won't
behave as badly as it could. I shut it off at the root to avoid having
to discover, down in the search, that it should turn itself off...