Oakfoam early resignation issue in high handicap games?

[Hong Lu]

Here is new findings on  handicap games between  Oakfoam (DCNN) versus gnugo on 4 handicap. 

5 matches results were  3 - 2  that Oakfoam beat gnugo version 3.8 

This is unexpected that  Oakfoam (DCNN) version lost 2 games to gnugo.   Oakfoam testing with fuego or Pachi UCT 

indicators that Oakfoam probably is on par at DGS  2 kyu while  gnugo only ranks at 10 kyu. 

Pachi UCT can beat gnugo on 7 handicap 5  to 0.    

The poor performance of Oakfoam  under handicap games is 2 reasons:

 one game lost because of early PASS issue, which was already reported in other thread, known issue. 

The 2nd game Oakfoam resigned at move  142,  way too early to call quit based on sgf reviewing by myself. 

seed: 1486264397

WARNING! unexpected move color

WARNING! unexpected move color

WARNING! unexpected move color

[genmove]: r:-0.001 v:0.599 plts:10007 ppms:0.23 cnncs:118 cnnps:2.73 rd:-1.000 r2:1.00 fs:0.00 eq:0 eq2:0 fsd:0.00 un:0/1(-0) bs:0  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,145,2,54,0,0,0,34,331,331,0) ravepreset: -nan expand_num: 11.1102

White resigns!

Game 1: B+Resign

Pachi had same problem and can only be resolved by changing resignation threshold.  I am going to dig out Oakfoam code to see if such parameter exist. 

Hong

[Hong Lu]

Append below to beginning of nicego-cnn-06.gtp, still does not solve problem. 

param thread_count 1

param time_move_max 60

param resign_ratio_threshold 0.01

param resign_move_factor_threshold 0.55

in 10 game 9 handicap matches against gnugo 3.8,   Oakfoam resigned 8 games, won 2 games. 

In 8 games resignation,  in fact  Oakfoam  was ahead  winning, but still resigned.   If this bug is fixed,  the real score should 6 - 4  gnugo winning by small margin.  

Try some other ideas to fix the issue

[Hong Lu]

Latest update:

 One extreme bad case when Oakfoam actually won the game with W+1.5 points, but it did not know and actually resigned on a won game.   I used modified or appended parameter in  nicego-cnn-06.gtp  to address early PASS/RESIGN, but still not fully fixed the issue yet. 

This early RESIGN bug in CPU_ONLY mode appears to be random similar to the issue of early PASS.  When I re-run the same GTP code second time with different random seed,   the PASS or RESIGN goes away. 

So I got a workaround on this issue for an GTP engine mainly designed for turn-based go  with  a perl GTP wrapper around  Oakfoam,   if the GTP answer  is  "PASS"/"RESIGN"  prematurely,  quit the 

current session,  restart a new oakfoam GTP engine session with different random seed,  re-run again.   If this is time still generating "PASS"/"RESIGN"  prematurely,  then re-run  a UCT version 

of Oakfoam.   I tested and found that UCT version of oakfoam does not have this early PASS bug although UCT version of oakfoam is very weak on strength, and should be only used as 

last resort in this workaround. 

A  DGS bot is developed using this workaround for Oakfoam: 

http://www.dragongoserver.net/userinfo.php?uid=97704

The  Oakfoam Bot use below appended/modified parameter in nicego-cnn-06.gtp file:

param thread_count 2

param time_move_max 60

param resign_ratio_threshold 0.01

param resign_move_factor_threshold 0.61

param surewin_pass_bonus 0.1

param tree resign_ratio_threshold 0.01

Other parameters file are unchanged from the original file. 

The UCT version  gtp file that I used as last resort to work around early PASS/RESIGN bugs:

param thread_count 2

param time_move_max 20

param resign_ratio_threshold 0.01

param resign_move_factor_threshold 0.61

param move_policy uct

Hong

[Detlef Schmicker]

some short remarks:

if all this happens with 1 thread, than I do not have any idea, how it could be oakfoam code, but most probably caffe bug. Problem: the pass accurs, if the cnn gives all 0 probabilities. This seem to happen randomly. this means, it will most probably also happen in nodes deeper in the search tree, which must weaken playing strength (by a lot, as it sometimes will do a pass instead of a good answer move)

by the way, uct is on by default in ng-06, dont know what you changed :) move policy cnn means only cnn without search....

not understanding a 1.5 lead is a known problem of value networks:) the value network does 70% of the winning probability in ng-06. so it should not resign in this case, but play to end if the value says loosing and playouts winning it should play to end....

Detlef

> Am 06.02.2017 um 04:39 schrieb Hong Lu <hong...@gmail.com>:
>
> 01

[Hong Lu]

I was not able to fix the early pass/resignation issue directly in Oakfoam code.  These two issues significantly weakened the Oakfoam in CPU_ONLY mode. 

Clarifications: 

My workaround only works for DGS like turn based go, not live go because my perl code will feed all the gtp command (play white,  play back) from beginning to end, then last 

"genmove"  to get answer from Oakfoam, then quit.  waiting for next move could be hours or days in corresponding games so that turn based go is fine with this workaround. 

Obviously,  this does not work in live mode games like in KGS.  On  1 minute per move when  Oakfoam CNN early pass/resign  bug happens, my perl wrapper spend another minute

 to redo the same thing in another session, if the answer is still buggy,  then redo again UCT in third session.  In other word,  my turn based perl wrapper could spend 3 minutes maximum per move when the early PASS/RESIGN bug hits.  In live mode games,  my perl wrapper would lose on time out easily.

2 commands the perl wrapper used :

/usr/local/bin/oakfoam --nobook -c /usr/local/share/oakfoam/configs/nicego-cnn-06.gtp

last resort (early pass/resign bug free):

/usr/local/bin/oakfoam --nobook -c /usr/local/share/oakfoam/configs/uct.gtp

My perl wrapper which powers the DGS Oakfoam Bot account is significantly stronger than the default Oakfoam code with 2 bugs. 

with 2 bugs,  Oakfoam lost in  5 games match against GNUGO 3.8  on 9 handicap stones,  won 2, lost 3

the perl wrapper beat gnugo 3.8 on 9 handicap stones in 10 games match,  won 8, lost 2. 

Hong