Oakfoam early pass bug?

[Hong Lu]

Hi  Detlef, 

I have found quite a severe bug in the  Oakfoam  caffe  CPU_ONLY version running in my dual core Ubuntu 14.04 machine.  Oakfoam seems to issue PASS in the middle of game, 

at least 2 games I reviewed all have this.  This would dramatically changed the game results. In fact before the Oakfoam issuing the PASS,  Oakfoam was ahead against Fuego.  

Obvious past the PASS,  fuego could reverse the results by re-living a dead group in the battle.  

I wrote a fuego gtp wrapper to let fuego also issue PASS once the other party did that so that I can debug the results.  Here is the game last log file before the PASS 

black: Oakfoam

white: Fuego

I0201 22:21:45.555395  3543 net.cpp:746] Ignoring source layer power_soll3

I0201 22:21:45.555402  3543 net.cpp:746] Ignoring source layer loss6

Attention, fixed coded as caffe support broken?!   --->>> !!!!!!!!!!!!!!!!!!!!!!!! 0 shape 20

seed: 1485950985

loading opening book from 'book.dat'... done

[genmove]: r:0.341 v:0.989 plts:10007 ppms:0.42 cnncs:14 cnnps:0.59 rd:-0.338 r2:1.00 fs:26.31 eq:0 eq2:0 fsd:0.52 un:1/1(-0) bs:1  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,91,0,118,13,23,0,0,0,38,357,357,0) ravepreset: -nan expand_num: 10.2142

play Black pass

play White pass

sgf log file are attached here for your review.  

The gtp command configuration is below 2 lines appended to the beginning of  your original  nicego-cnn-06.gtp  file: 

param thread_count 2

param time_move_max 60

Thanks, 

Hong

[Hong Lu]

Another game where black (Oakfoam) issued a PASS early in the game in the fight.  This  fuego took the PASS chance, and won the game with resignation. 

Here is log where MOVE 119   Oakfoam issued PASS:

 Attention, fixed coded as caffe support broken?!   --->>> !!!!!!!!!!!!!!!!!!!!!!!! 0 shape 20

seed: 1485951277

loading opening book from 'book.dat'... done

[genmove]: r:0.180 v:0.000 plts:10007 ppms:1.04 cnncs:4 cnnps:0.42 rd:-0.180 r2:1.00 fs:-13.98 eq:0 eq2:0 fsd:0.53 un:1/1(-0) bs:1  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,98,0,93,6,16,0,0,0,51,394,337,0) ravepreset: -nan expand_num: 8.24979

play Black pass

The whole game SGF is attached here

[Hong Lu]

Third case,  in fact  black (Oakfoam) issued 2 premature PASS.  Still this game Oakfoam win by resignation against Fuego. 

Log file  where 2 PASS happened: 

Attention, fixed coded as caffe support broken?!   --->>> !!!!!!!!!!!!!!!!!!!!!!!! 0 shape 20

seed: 1486548573

loading opening book from 'book.dat'... done

[genmove]: r:0.045 v:0.000 plts:2307 ppms:0.04 cnncs:206 cnnps:3.42 rd:-0.017 r2:1.00 fs:-64.27 eq:0 eq2:0 fsd:2.07 un:1/1(-0) bs:1  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,99,0,106,3,36,0,0,0,47,381,325,0) ravepreset: -nan expand_num: 11.0437

play Black pass

Attention, fixed coded as caffe support broken?!   --->>> !!!!!!!!!!!!!!!!!!!!!!!! 0 shape 20

seed: 1486517843

loading opening book from 'book.dat'... done

[genmove]: r:0.175 v:0.000 plts:10006 ppms:0.98 cnncs:8 cnnps:0.79 rd:-0.170 r2:1.00 fs:-15.37 eq:0 eq2:0 fsd:0.46 un:1/1(-0) bs:1  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,104,0,90,7,14,0,0,0,57,388,336,0) ravepreset: -nan expand_num: 9.62488

play Black pass

[Detlef Schmicker]

hmm, this does not look as ng-06 playing fuego ...


At move 119 I get

genmove b
best move cannot change! (current 0.264 playratio 1.204 timedratio 0.000 calcmax 0.000 time used 20.843 total 8309 newtotal 8309)
[genmove]: r:0.463 v:0.000 plts:8314 ppms:0.40 cnncs:687 cnnps:32.90 rd:0.067 r2:4.21 fs:-3.67 eq:0 eq2:0 fsd:1.01 un:9/14(-0) bs:1 pv:(S8,Q10,P11,Q11,O12,P12,O11,T14,Q12,S13,P13,S11,S9,R3,S3,Q3)  un:(P11,O12,L12,D18,Q10,S4,R3,Q9,R8,T14,P13,P19,L6,J5)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,97,0,95,6,16,0,0,0,50,392,340,0) ravepreset: -nan expand_num: 11.0102
= r8

if you look at un: you get the moves taken into account during playouts. They are preselected by the neural net and in your case the neural net does not seem to offer any move probability abough the

param CNN_pass_probability 0.001




So I am not sure what happens, but you do get total different cnn values from me :(

by the way, you seem to load the opening book? this should not be done with cnn, but should not cause this kind of problems....


Detlef

[Hong Lu]

Yes,  I did load the fuego openbook data.  Sure I can test more matches without loading the openbook data to see if I see the same PASS bug. 

[Detlef Schmicker]

just a follow up:

I added cpu only switch and turned it on in my version:

move 119:

genmove b
best move cannot change! (current 0.301 playratio 1.277 timedratio 0.000 calcmax 0.000 time used 143.146 total 7830 newtotal 7845)
[genmove]: r:0.478 v:0.000 plts:7850 ppms:0.05 cnncs:653 cnnps:4.55 rd:0.054 r2:3.39 fs:-2.72 eq:0 eq2:0 fsd:0.90 un:9/14(-0) bs:1 pv:(S8,S6,S7,S4,Q9,R3,R9,S2,L12,G16,E18,C18)  un:(P11,O12,L12,D18,Q10,S4,R3,Q9,R8,T14,P13,P19,L6,J5)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,96,0,95,6,16,0,0,0,49,393,340,0) ravepreset: 3.60588e+08 expand_num: 11.0015
= r8

[Hong Lu]

Due to the low end hardware,  I add this 1 minute time limit, which could cause the difference between your results versus my results. 

Without this time limit,  I noticed some moves can take 3 to 5 minutes per move, which is not practical.  I think 1 minute is probably longest allowed time for 

turn based bot like DGS

[Detlef Schmicker]

dont think this is the problem, you cnn gives wrong numbers.

If you use Gogui you can use the

Show Probability CNN

command and get a picture of the moves suggested by cnn.

by the way the first move on an empty board should look like this

[genmove]: r:0.516 v:0.000 plts:10005 ppms:0.39 cnncs:831 cnnps:32.00 rd:0.008 r2:1.09 fs:4.11 eq:0 eq2:0 fsd:2.27 un:3/13(-0) bs:0 pv:(R3,R17,R5,R14,E3,C5,E5,C7,C17,E17,C15,E15,C13,C18,B18,D17,C16)  un:(R4,R3,C3,R17,R5,C17,R16,C16,R6,C4,C6,D3,Q17)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,109,0,75,2,15,0,0,0,62,416,317,0) ravepreset: -7.45269e-15 expand_num: 11
= c3

[Hong Lu]

OK,  after disabling openbook,  here is one game first move by Oakfoam: 

!! 0 shape 20

seed: 1486035432

[genmove]: r:0.529 v:0.000 plts:2839 ppms:0.05 cnncs:237 cnnps:3.93 rd:-0.001 r2:1.02 fs:7.58 eq:0 eq2:0 fsd:3.38 un:7/8(-0) bs:1 pv:(R4,P4,Q3,P3,R6,P6,C3,Q8,S8,C5)  un:(R4,R3,C3,R17,R5,C17,R16,C16)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,110,0,75,2,15,0,0,0,62,416,317,0) ravepreset: -nan expand_num: 10.9578

play Black r16

[Hong Lu]

Another game the first move: 

Attention, fixed coded as caffe support broken?!   --->>> !!!!!!!!!!!!!!!!!!!!!!!! 0 shape 20

seed: 1485886887

[genmove]: r:0.522 v:0.000 plts:2551 ppms:0.04 cnncs:212 cnnps:3.53 rd:0.013 r2:1.09 fs:0.50 eq:0 eq2:0 fsd:3.05 un:3/8(-0) bs:0 pv:(R4,P4,R6,Q3,R3,P6,R8,R2,S2,R17,P8,M4,R15,P17)  un:(R4,R3,C3,R17,R5,C17,R16,C16)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,110,0,75,2,15,0,0,0,62,416,317,0) ravepreset: -nan expand_num: 10.967

play Black c3

[Detlef Schmicker]

please send me the config file exactly as you use it,

my first move looks different, even with exactly the same number of playouts...

Detlef

--
You received this message because you are subscribed to the Google Groups "Oakfoam" group.
To unsubscribe from this group and stop receiving emails from it, send an email to oakfoam+u...@googlegroups.com.
To post to this group, send email to oak...@googlegroups.com.
Visit this group at https://groups.google.com/group/oakfoam.
For more options, visit https://groups.google.com/d/optout.

[Detlef Schmicker]

sorry it took me so long,

the problem is the komi!!

5.5 is not supported by the net,

so sorry you spend so much time on this, I should add a warning!

line 757 in engine.cc 

0.5 6.5 7.5 is supported,

I will change this to

<5.5

5.5-6.5

>6.5

this is not perfect, but at least the value net will not give 0 all the time!

Detlef

[Hong Lu]

I see.  It is twogtp.pike that set komi at 5.5.  

komi 5.5 is not common, but still a reasonable number to use.  I am glad that you are able to catch the issue and fix the code. 

Let me know after you commit the new code and I will do another test. 

Hong

[Hong Lu]

I think Oakfoam support for 0.5  6.5  7.5 komi  is adequate.  I will be fine if your new code simply treat komi 5.5 as if equal 6.5 in your code.  I don't think the results will be 

that much different.  some games may have different results on different komi,  but Oakfoam should play the same way as it should. 

[Hong Lu]

Here is history of komi in go game: 

History of Komi

Although there were some games played with compensation in the 19th century, more substantial experiments came in the first half of the 20th century. Several values were experimented with, until a value of 4.5 became the standard from the 1940's onward. game results from the next two decades showed that 4.5 komi still favored black, so a change was made to 5.5 komi, which was mostly used for the rest of the century in both Japan and China. At the start of the 21st century, the komi was increased yet again, to 6.5 in Japan and to 7.5 in China.

[Hong Lu]

I implemented your proposal for the fix on Engine.cc as below:

if (col==Go::BLACK) {

  if (komi < 5.5) komiplane=0;

  if (komi >= 5.5 && komi <=6.5) komiplane=1;

  if (komi > 6.5) komiplane=2;

}

if (col==Go::WHITE) {

  if (komi < 5.5) komiplane=3;

  if (komi >= 5.5 && komi <=6.5) komiplane=4;

  if (komi > 6.5) komiplane=5;

}

recompiled and re-test on more matches.  This should catch all komi situatation, lets see how it goes. 

Here is  first move stats after the fix.  Because none of the game I am running starts with c3,  I just picked one of game first move stats here:

seed: 1484994699

[genmove]: r:0.513 v:0.507 plts:3016 ppms:0.05 cnncs:240 cnnps:3.98 rd:0.032 r2:1.06 fs:4.53 eq:0 eq2:0 fsd:3.34 un:3/9(-0) bs:1 pv:(D4,Q17,D16,F17,C14,K16,R14,Q15)  un:(Q16,D16,Q4,D4,C4,D3,R16,Q3,Q17)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,110,0,75,2,14,0,0,0,64,417,315,0) ravepreset: -nan expand_num: 10.9625

play Black q4

another game first move:

Attention, fixed coded as caffe support broken?!   --->>> !!!!!!!!!!!!!!!!!!!!!!!! 0 shape 20

seed: 1485688646

[genmove]: r:0.526 v:0.509 plts:2712 ppms:0.05 cnncs:214 cnnps:3.56 rd:0.016 r2:1.28 fs:2.76 eq:0 eq2:0 fsd:3.24 un:4/8(-0) bs:0 pv:(D16,R4,Q16,R14,O17,Q10,O3,P4)  un:(Q16,D16,Q4,D4,C4,D3,R16,Q3)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,110,0,75,2,15,0,0,0,64,417,314,0) ravepreset: -nan expand_num: 10.9533

play Black d4

[Detlef Schmicker]

This looks promising :)

genmove b
[genmove]: r:0.538 v:0.507 plts:3016 ppms:0.25 cnncs:241 cnnps:20.03 rd:0.033 r2:1.29 fs:8.10 eq:0 eq2:0 fsd:2.88 un:3/9(-0) bs:1 pv:(D4,Q17,D16,F17,C14,K16,R14,Q15)  un:(Q16,D16,Q4,D4,C4,D3,R16,Q3,Q17)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,110,0,75,2,14,0,0,0,64,417,314,0) ravepreset: -nan expand_num: 10.9544
= q4


the value net gives the same wining probability
v: 0.507

and the unpruned moves are the same too
un:(Q16,D16,Q4,D4,C4,D3,R16,Q3,Q17)


There are sometimes problems with passing, but usualy the opponent passes and oakfoam does too. than it is a problem in cgos counting, that there might be some uncaptured stones which are counted alive.

I did not put too much work in this kind of stuff at the moment, as my main interest is getting oakfoam strong at the moment!

A ladder reader with complex moves in between is also missing :( it can play a ladder to the end, but can not see what to do if in between there are two possible ladder moves...

anyway, I hope you are on a good track now. It should have a very high winrate against 10k fuego at least:

This is the play of only the neuronal net on cgos (the actual version might be slightly higher) without any playouts. This means it always would play the first unpruned move, thus Q16 as the first move.


you can set this in the analytic command (Gogui) Parameters(General) -> move policy to cnn
and in the Parameters(CNN) (cnn random for  only cnn) you can choose from the how many best moves it is shuffling ....

[Hong Lu]

Perfect, the matching results look good. 2 resignation wins.  One is not sure, I think also Oakfoam win against fuego. 

There are still early pass,  but I feel is ok like below: 

seed: 1485999120

[genmove]: r:0.290 v:0.787 plts:10006 ppms:1.12 cnncs:6 cnnps:0.67 rd:-0.290 r2:1.00 fs:12.06 eq:0 eq2:0 fsd:0.65 un:1/1(-0) bs:1  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,106,0,93,3,16,0,0,0,47,402,330,0) ravepreset: -nan expand_num: 9.16651

play Black pass

This early pass is probably because Oakfoam think the winning probability is 787%  (v: 0.787) that so high, it is sure win. 

[Detlef Schmicker]

this does not look correct, no other moves.

I have a little bit the feeling, that the caffe net is not stable over the game in CPU mode.

maybe caffe bug?

Detlef

[Hong Lu]

Maybe  time limit kickin ?   I set 1 minute per move maximum.  Not enough time,  then PASS

[Hong Lu]

Just double checked the folder,  no core dumped  No sign of software crash or anything causing this early PASS. 

[Hong Lu]

The same game had 2 early PASS,  after the first pass, the second pass is below: 

seed: 1485994823

[genmove]: r:0.313 v:0.863 plts:10007 ppms:1.31 cnncs:4 cnnps:0.52 rd:-0.313 r2:1.00 fs:16.69 eq:0 eq2:0 fsd:0.53 un:1/1(-0) bs:1  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,102,0,101,4,17,0,0,0,45,389,338,0) ravepreset: -nan expand_num: 8.24979

play Black pass

[Hong Lu]

Despite some games still have early pass,  this seem to only affect one game result. 

The 10 games against fuego,  5 white, 5 black,   Oakfoam  won 9 to 1

The one game it lost, it was due to early pass as below:

seed: 1485651382

[genmove]: r:0.289 v:0.938 plts:10007 ppms:0.43 cnncs:67 cnnps:2.88 rd:-0.288 r2:1.00 fs:7.16 eq:0 eq2:0 fsd:0.61 un:1/1(-0) bs:1  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,110,0,118,5,22,0,0,0,29,372,342,0) ravepreset: -nan expand_num: 10.8806

play White pass

White (Oakfoam) pass at move  168, thinking its own winning rate is 93.8%.   White was ahead W+28.5 at this point.  Major blunder,  it created a ko,  Black eventually won this ko and 

killed a large group of white.   If it did not pass, the group would have lived by making 2 eyes. 

The game evetually B+ 5.5, fuego win.  

There were other early pass in other games, but they did not change final results.  

Hong

[Detlef Schmicker]

Hi,

I found 5 free minutes :)

If you want to try to track down this problem (I can not at the moment, mainly because my PC is used to train a new neural net at the moment...)

1. Try to open a log file from GoGui and check if oakfoam with the configuration file unprunes (un:) also only the PASS move. If my test yesterday was ok, this should not be the case meaning that the bug is not reproduceble from the position.

2. As a first test set thread_count to 1 and try to see, if it still happens. If it is a problem in oakfoam the most risky part is the multi threading I think.

If this helps try to live with it and tell me, I will try to track it down, as probably this will affect the gpu version as well, only with lower probability, as the timing of the cnn evaluation is much lower. I do not see early passing in selfplay and on CGOS, but who knows.


If this does not help: Hmm, this is bad. My first guess would be it is a bug in caffe, but I would try to track it down. First I would check Tree.cc

line 2225 and 2229

for some reason line 2225 gives alway 0 in your case. so I would try to do some dumping in case all

CNNresults[size*x+y] are smaller than the CNN_pass_probability and try to find the problem :)


sorry I can not reproduce at the moment, as I do not have resources to run a version of caffe
(another bug in caffe makes it reserve GPU memory also in CPU mode if compiled with GPU support, 
and I need all my GPU memory for training :)

[Hong Lu]

Hi Detlef, 

No problem with your plan on this issue.   I think at this moment,  Oakfoam version with NG06 setting is very solid even with this  "early pass" issue in low end hardware. 

I will test your idea of the tracking the "early pass issue" as you suggested below, probably next week or in the weekend.  My hardware is tied up in more testing now, mainly focusing on high 

handicap games against GNUGO /Pachi to make sure   Oakfoam is OK under high handicap environment.    Once my resource is freed up,  I will try out GoGui as you suggested. 

I only had success with twogtp.pike  in the past after trying out many different testing control software as most testing software did not work well under handicap games mode.   It will take some time for me to setup GoGui successfully. 

Happy Chinese New Year and have a great weekend!

Hong

[Detlef Schmicker]

sounds great! I have no idea on high handicap games :)

I would be afraid, that the early pass issue can cause weak moves too, depending what the problem is ...

I dont know how strong dcnn pachi is?! at least if you run both with the same number of playouts oakfoam should be much stronger, as it is as strong as aya with 10000 playouts.

So I would set thread_count to 1 and see if early pass goes away...

Detlef

[Detlef Schmicker]

by the way: I use gomill, at least the documentation says it supports handicap!

And it supports run several games at once! so thread_count 1 is no problem and run 2 competitions. I would choose a number of playouts for oakfoam which is ok for your timing and keep it to get stable results.

If you use pachi: by default it does pondering, to get stable fair competitions this should be turned of, otherwize it is taking computational power from its opponent.

Detlef

[Hong Lu]

Tested with param thread_count 1 to see whether this premature early PASS issue is caused by multi-thread trouble. 

Unfortunately,  the early pass bug still exist here.  Most cases, the win rate is high at > 0.70,  some cases it passes very early within 50 moves, and the win rate is very low like below:

[genmove]: r:0.037 v:0.102 plts:10007 ppms:0.51 cnncs:2 cnnps:0.10 rd:-0.463 r2:1.00 fs:-84.60 eq:0 eq2:0 fsd:0.66 un:1/1(-0) bs:1  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,105,0,81,2,22,0,0,0,55,405,326,0) ravepreset: -nan expand_num: 5.49972

play White pass

It looks that Oakfoam  will issue PASS when it feels that it surely wins,  or  when it feels it surely lose.  

Strange behavior.  

Hong

[Hong Lu]

Latest update:

This early PASS/RESIGN bug in CPU_ONLY mode appears to be random.  When I re-run the same GTP code second time with different random seed,   the PASS goes away. 

So I got a workaround on this issue for an GTP engine mainly designed for turn-based go  with  a perl GTP wrapper around  Oakfoam,   if the GTP answer  is  "PASS" prematurely,  quit the 

current session,  restart a new oakfoam GTP engine session with different random seed,  re-run again.   If this is time still generating "PASS" prematurely,  then re-run  a UCT version 

of Oakfoam.   I tested and found that UCT version of oakfoam does not have this early PASS bug although UCT version of oakfoam is very weak on strength, and should be only used as 

last resort in this workaround. 

A  DGS bot is developed using this workaround for Oakfoam: 

http://www.dragongoserver.net/userinfo.php?uid=97704

Hong

[Hong Lu]

A more detailed description of this early PASS bug: 

It happens to either thread 1 or 2.  

The early PASS typically happens in 2 places: 

(1)  a very early stage in high handicap games, say 9 handicap stones games, within the first 10 moves, 

Oakfoam issued PASS with win rate very low, at 0.1  

(2) Or late stage when win rate is very high, at 0.7 0.8  or 0.9.   This happens a lot in even games, which was the first phenomena I observed.  initially I thought 

this was issue of "SUREWIN" related code until I saw the high handicap games issue (1). 

Hong

[Hong Lu]

Early PASS bug happened in real-time to Oakfoam Bot at DGS
http://www.dragongoserver.net/game.php?gid=1148086

at move 244,  Oakfoam Bot issued PASS, which was premature because border was not closed and the fight was still ongoing.

My perl wrapper workaround did not catch this because my perl wrapper only check for  first 224 GTP commands (about 215 moves).  After 225  gtp commands,  Oakfoam is allowed to
issue PASS in order to allow reasonable PASS at end game.  

[Detlef Schmicker]

game ending is low priority for me at the moment, on CGOS usally it is played to end or resigned.


I run a test oakfoam against gnugo (3.8)
 'gnugo'     : Player("gnugo --mode=gtp --chinese-rules "
                      "--capture-all-dead --positional-superko --level=0"),


with 4 H for gnugo and only 100 playouts for oakfoam with CPU_ONLY switched on, but caffe compiled with gpu support

gnugo v oakfoam (24/1000 games)
board size: 19   handicap: 4 (free)   komi: 0.5
                   wins                                  avg cpu
gnugo           2       8.33%   (black)   10.37
oakfoam     22     91.67%   (white)  233.88


on game was really lost, one was misjudged by oakfoam I think.

[Hong Lu]

Below game is finished,  Oakfoam Bot lost,  B+0.5 

This is a loss of half point with Chinese rule.  If the early pass would not have happened, and Black did not gain extra 1 point, then may be Oakfoam may have won.  

[Detlef Schmicker]

I can not reproduce early pass bug here. I set cpu_only in oakfoam but have gpu code compiled into caffe...

I let it play a lot of handicap games, did not analyze all but I did not see your kind of problem?!

at the moment I am running 9h games against gnugo and it is about 50% wins with 100 playouts (resign threshold 0.05) 

Detlef

[Hong Lu]

Oakfoam Bot at DGS in my setting can easily beat GNUGO version 3.8 in 9 handicap settings.  But my setting is fixed handicap, maybe you have free handicap?

[Detlef Schmicker]

Hi,

very interessting: I got an early pass :)

I got it with cpu_only caffe compiled with mkl library!

It seems to play significantly weaker too, I have it on cgos running...
www.yss-aya.com/cgos/19x19/SGF/2017/03/17/199877.sgf

I did not have this kind of problem with atlas and openblas (openblas is 2 times faster and mkl three times faster than atlas on my machine)

Did you compile caffe with mkl?


On Thursday, February 2, 2017 at 4:59:07 AM UTC+1, Hong Lu wrote:

Hi  Detlef, 

I have found quite a severe bug in the  Oakfoam  caffe  CPU_ONLY version running in my dual core Ubuntu 14.04 machine.  Oakfoam seems to issue PASS in the middle of game, 

at least 2 games I reviewed all have this.  This would dramatically changed the game results. In fact before the Oakfoam issuing the PASS,  Oakfoam was ahead against Fuego.  

Obvious past the PASS,  fuego could reverse the results by re-living a dead group in the battle.  

I wrote a fuego gtp wrapper to let fuego also issue PASS once the other party did that so that I can debug the results.  Here is the game last log file before the PASS 

black: Oakfoam

white: Fuego

I0201 22:21:45.555395  3543 net.cpp:746] Ignoring source layer power_soll3

I0201 22:21:45.555402  3543 net.cpp:746] Ignoring source layer loss6

Attention, fixed coded as caffe support broken?!   --->>> !!!!!!!!!!!!!!!!!!!!!!!! 0 shape 20

seed: 1485950985

loading opening book from 'book.dat'... done

[genmove]: r:0.341 v:0.989 plts:10007 ppms:0.42 cnncs:14 cnnps:0.59 rd:-0.338 r2:1.00 fs:26.31 eq:0 eq2:0 fsd:0.52 un:1/1(-0) bs:1  un:(PASS)st:(0,0,0,0,0,0,0,0,0,0,0,0,0,91,0,118,13,23,0,0,0,38,357,357,0) ravepreset: -nan expand_num: 10.2142

play Black pass

play White pass

sgf log file are attached here for your review.  

The gtp command configuration is below 2 lines appended to the beginning of  your original  nicego-cnn-06.gtp  file: 

param thread_count 2

param time_move_max 60

Thanks, 

Hong

[Detlef Schmicker]

sorry, false alarm, was another reason :(

[Detlef Schmicker]

OK,

now I think I might have the same issue as you have:

When playing on cgos I switch between two configurations. So oakfoam is reloaded some times and after one of the reloads I had a lot of passes!

Possibly it is a bug loading the neural net, maybe some kind of race condition. If I understood you correctly your "work around" loads oakfoam again in a case of pass?

Do you log this kind of reloads? If so, it would help me debugging if you could say, if it happens in gpu version too and if you could give me a rough number on how often it happens!


Thanks a lot

Detlef

[Hong Lu]

Sure,  I will change my perl wrapper code at DGS to log this bug incidence.   The current GPU version of DGS  oakfoam no longer reload same NG06 again.  Instead, it 

reload below UCT version once to minimize the loss and move forward: 

 oakfoamUCT.sh   ( I will add a log in this shell script to catch all the early bug incident ):

/usr/local/bin/oakfoam --nobook -c /usr/local/share/oakfoam/configs/uct.gtp

content of file uct.gtp:

param thread_count 2

param time_move_max 20

param resign_ratio_threshold 0.01

param resign_move_factor_threshold 0.61

param move_policy uct

[Hong Lu]

Added date and log of move for both early pass/resign bug in my  DGS  oakfoam bot perl wrapper. 

Will report here when the incident of early pass/resign bug happens in DGS bot. 

The oakfoam bot at DGS run very solid at 4 dan level. 

Hong

[Detlef Schmicker]

thanks a lot!!! I think I tracked down the issue, very ugly a copy and paste error :)

https://bitbucket.org/dsmic/oakfoam/commits/033fe148ae1bb9ebcff9f056fae561f939dae3c2

you might also change

https://bitbucket.org/dsmic/oakfoam/commits/86de3d51909c7821a076fe6520b0e7b5b69dd238

which is also a bug fix and should increase playing strength slightly (I did not give the version a new number though).
It is wrong to put a high value on salfatari moves in playouts...

[Hong Lu]

So far the log file has no early pass bug yet.  THere is early resign issue logged, but  I would not necessarily call this as "bug". 

For example: 

http://www.dragongoserver.net/game.php?gid=1156491

My log file says that oakfoambot wanted to resign as early as step 157,  a little too early in my taste and my code forced the bot to continue to play.  Eventually it quit at move 191

[Detlef Schmicker]

Tanks a lot, I think initializing the variable depends on the linked libs, so maybe I was just lucky with the gpu version, anyway, I think the bug is fixed, as I do not have any problems with the two versions running on cgos (lloading oakfoam before every game) at the moment.

resigning is controled by two parameters

param resign_move_factor_threshold 0.300

which is multiplied by the number of fields on the board (361 on 19x19)
thus before move 0.3*19*19=108 it should not resign

the other is

param resign_ratio_threshold 0.15

which means it should have a winrate less than 15% in default. This is user friendly but usually it is quite right with resigning, but maybe not with high handicap...
This second rule became worse due to the value network, before there was a decreasing dynamic komi for the handicap stones, but the value network does not support dynamic komi....

Anyway, thanks again on running oakfoam! I am still fighting training a bigger policy network :)

Detlef