Script for deep analysis of position

[zeFresk]

Hello,

Some times ago someone mentioned creating pre-computed opening book for Leela. Yesterday I wanted to analyze deeply a given position (the starting position of the Ruiz Lopez) to see what Leela thought about it.

What I wanted was to use Multi-PV to create a tree which would contains the top moves after each parent moves.

For example given starting position we could end up with this (multiPV set to 2 and depth set to 2 as well) :  

Sadly I couldn't find anything like that for free...

So I created a script which generate such trees from any fen or epd and export them as pgn. It works for any engine but I mostly use it with Leela.

The repository can be found here : https://github.com/zeFresk/deep-position-analysis and I think the README is a good tutorial.

I don't know if I can post this here but I thought we could use this to compare different networks and engines.

Please tell me what you think about this.

PS: English is not my mother tongue, if I did any mistake in this post or in my repository please tell me :)

[gvergh...@gmail.com]

Very interesting...thanks !!

Will try it soon.

About exporting the tree, what format does the tree use ?

[gvergh...@gmail.com]

Tried it...it's quite fast.

But the output pgn is not readable by both Fritz and Chessbase.

[Event "DeA using Stockfish 011218 64 BMI2 at 1000000 per move, 5 ply-depth, of r2qkb1r/p3pppp/2n2n2/1p1p1b2/P1pP4/1P2PN1P/2P1BPP1/RNBQK2R b KQkq - 0 8"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "Stockfish 011218 64 BMI2"]
[Black "Stockfish 011218 64 BMI2"]
[Result "*"]
[FEN "r2qkb1r/p3pppp/2n2n2/1p1p1b2/P1pP4/1P2PN1P/2P1BPP1/RNBQK2R b KQkq - 0 8"]
[SetUp "1"]

8... a6 { -0.04 } ( 8... e6 { -0.04 } 9. O-O { 1.22 } ( 9. axb5 { 0.0 } 9... Nb4 { 0.0 } ( 9... cxb3 { -1.08 } 10. cxb3 { 1.92 } ( 10. Na3 { 0.62 } 10... Nb4 { -0.33 } ( 10... Bxa3 { -0.4 } 11. Bxa3 Bxc2 12. Qc1 Na5 13. Ne5 Nc4 14. O-O Nxa3 15. Nc6 Qc7 16. Rxa3 Ne4 17. Qb2 O-O 18. Rfa1 Nd6 19. Rxa7 Rxa7 20. Rxa7 Qb6 21. Ra6 Qb7 ) 11. cxb3 Bd6 12. O-O O-O 13. Bd2 Ne4 14. Nb1 Qe7 15. Ne5 Nxd2 16. Qxd2 Rac8 17. Rc1 Rc7 18. Rxc7 Qxc7 19. Nc3 Rc8 20. Na2 Qa5 21. Rc1 Rxc1+ 22. Nxc1 ) 10... Nb4 { -1.69 } ( 10... Nb8 { -1.83 } 11. O-O Nbd7 12. Bb2 Bb4 13. Ra4 Bd6 14. Ra6 Ne4 15. Nc3 Nb6 16. Nxe4 Bxe4 17. Bd3 O-O 18. Bxe4 dxe4 19. Ne5 Bxe5 20. dxe5 Nd5 21. Qd4 ) 11. O-O ) 10. Na3 { 0.0 } ( 10. O-O { -1.3 } 10... Bxc2 { 2.09 } ( 10... Nxc2 { -0.34 } 11. Ra2 cxb3 12. Rb2 Bd6 13. Ne5 Rc8 14. Rxb3 O-O 15. Bd2 Bxe5 16. dxe5 Nd7 17. f4 Be4 18. Nc3 ) ) 10... Ne4 { 0.0 } ( 10... cxb3 { -0.65 } 11. cxb3 ) 11. bxc4 Nc3 12. Qd2 Ne4 ) 9... Nb4 { -1.03 } ( 9... b4 { -1.45 } 10. bxc4 { 1.55 } ( 10. Ne5 { 0.59 } 10... Nxe5 { -0.42 } ( 10... Rc8 { -0.65 } 11. bxc4 dxc4 12. Nxc4 Be7 13. Nbd2 O-O 14. Bd3 Nd5 15. Bxf5 exf5 16. a5 Qd7 17. Bb2 Rfd8 18. Qf3 Bf6 19. Nb3 Qe6 ) 11. dxe5 ) 10... Be7 { -1.45 } ( 10... dxc4 { -1.45 } 11. Bxc4 Bd6 12. Nbd2 O-O 13. Bd3 Bxd3 14. cxd3 Nd5 15. Nc4 Nc3 16. Qe1 Na5 17. Bd2 Nb3 18. Bxc3 bxc3 19. Qxc3 Nxa1 20. Rxa1 Kh8 ) 11. cxd5 ) 10. Na3 { 1.13 } ( 10. Ne1 { -0.04 } 10... a6 { 0.18 } ( 10... Bd6 { -0.95 } 11. c3 ) 11. Ba3 Ne4 12. Nf3 Bd6 13. c3 Nc6 14. Bxd6 Nxd6 15. Nbd2 O-O 16. Ne5 Nxe5 17. dxe5 Ne4 18. Nxe4 Bxe4 19. f3 Bg6 ) 10... Ne4 { -1.04 } ( 10... Rc8 { -1.51 } 11. c3 Nd3 12. axb5 Ne4 13. Bd2 Nb2 14. Qc1 Nd3 15. Bxd3 cxd3 16. c4 Nxd2 17. Qxd2 Bxa3 18. Rxa3 dxc4 19. bxc4 Rxc4 20. Rxa7 O-O 21. Ne5 Rc2 ) 11. Bd2 Bd6 12. Be1 cxb3 13. Bxb5+ Kf8 14. c4 Nc2 15. Nxc2 bxc2 16. Qe2 dxc4 17. Ne5 Kg8 18. Rc1 c3 19. Qxc2 Bxe5 ) 9. Bb2 { -0.3 } ( 9. O-O { -0.22 } 9... e6 { 0.69 } ( 9... Ne4 { -0.14 } 10. axb5 { 0.55 } ( 10. Nfd2 { 0.18 } 10... Nxd2 { -0.45 } ( 10... Nb4 { -0.5 } 11. Nxe4 Bxe4 12. axb5 axb5 13. Rxa8 Qxa8 14. Qd2 Na2 15. Ba3 h5 16. h4 Bg6 17. Bc5 e6 18. Bxf8 Kxf8 19. bxc4 dxc4 20. Nc3 Nxc3 21. Qxc3 Kg8 ) 11. Nxd2 ) 10... axb5 { -0.25 } ( 10... Nb4 { -3.79 } 11. bxa6 ) 11. Rxa8 Qxa8 12. Nfd2 Nf6 13. g4 Be6 14. Bf3 h6 15. Bg2 cxb3 16. Nxb3 h5 17. g5 Ne4 18. Nc5 Nxc5 19. dxc5 b4 20. Bb2 Qa2 ) 10. axb5 { -0.27 } ( 10. Nh4 { -0.37 } 10... Bb4 { 0.09 } ( 10... Be4 { 0.0 } 11. f3 Bg6 12. Bd2 Bh5 13. axb5 axb5 14. Rxa8 Qxa8 15. Nc3 Qb7 16. bxc4 bxc4 17. Qa1 Bb4 18. Rb1 Qb8 ) ) 10... axb5 { 0.47 } ( 10... cxb3 { -2.29 } 11. cxb3 axb5 12. Rxa8 Qxa8 13. Bxb5 Nd7 14. Bd2 Bd6 15. Nc3 O-O 16. Qa1 Qxa1 17. Rxa1 Rc8 18. Rc1 Ncb8 19. Nh4 Be4 20. f3 Bg6 21. Nxg6 hxg6 ) 11. Rxa8 Qxa8 12. Bd2 Ne4 13. Nh4 Bg6 14. Be1 Qa1 15. Nxg6 hxg6 16. bxc4 bxc4 17. f3 Nd6 18. Bd2 Be7 19. Nc3 Qxd1 20. Rxd1 Nb4 21. Rc1 ) 9... e6 { 0.49 } ( 9... Nb4 { 0.01 } 10. Na3 { 0.01 } ( 10. e4 { -2.05 } 10... Bxe4 { 2.23 } ( 10... Nxe4 { 1.32 } ) 11. Na3 cxb3 12. cxb3 bxa4 13. O-O e6 14. Bc3 Bd6 15. bxa4 O-O 16. Qd2 Nc6 17. Nc2 Qc7 18. Nb4 Bxf3 19. Bxf3 Rfc8 20. Rfc1 Bf4 ) 10... e6 { 0.42 } ( 10... cxb3 { 0.37 } 11. cxb3 bxa4 12. bxa4 e6 13. O-O Be4 14. Bc3 Bd6 15. Bxb4 Bxb4 16. Nb1 O-O 17. Nbd2 Bd6 18. Nxe4 Nxe4 19. Bd3 Nc3 ) 11. O-O ) 10. axb5 { -0.51 } ( 10. O-O { -0.59 } 10... Bd6 { 0.51 } ( 10... Nb4 { 0.32 } 11. Na3 cxb3 12. cxb3 bxa4 13. Nh4 Be4 14. f3 Bg6 15. Nxg6 hxg6 16. Nc2 Qc7 17. Nxb4 Bxb4 18. Rxa4 Rb8 19. f4 Ne4 20. Bxa6 ) 11. g4 ) 10... axb5 { 0.78 } ( 10... Nb4 { 0.39 } ) 11. Nc3 Rb8 12. O-O Bd6 13. Nh4 Be4 14. f3 Bg6 15. Ra6 Qc7 16. f4 O-O 17. Nxg6 hxg6 18. bxc4 bxc4 *

=========================================================================

However, Aquarium reads it fine, which is surprising.

Usually, Fritz and Chessbase read almost any format of pgn.

[Event "DeA using Stockfish 011218 64 BMI2 at 1000000 per move, 5 ply-depth, of r2qkb1r/p3pppp/2n2n2/1p1p1b2/P1pP4/1P2PN1P/2P1BPP1/RNBQK2R b KQkq - 0 8"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "Stockfish 011218 64 BMI2"]
[Black "Stockfish 011218 64 BMI2"]
[Result "*"]
[FEN "r2qkb1r/p3pppp/2n2n2/1p1p1b2/P1pP4/1P2PN1P/2P1BPP1/RNBQK2R b KQkq - 0 8"]
[SetUp "1"]

8... a6 (8... e6 9. O-O (9. axb5 Nb4 (9... cxb3 10. cxb3 (10. Na3 Nb4 (10...
Bxa3 11. Bxa3 Bxc2 12. Qc1 Na5 13. Ne5 Nc4 14. O-O Nxa3 15. Nc6 Qc7 16. Rxa3
Ne4 17. Qb2 O-O 18. Rfa1 Nd6 19. Rxa7 Rxa7 20. Rxa7 Qb6 21. Ra6 Qb7)11. cxb3
Bd6 12. O-O O-O 13. Bd2 Ne4 14. Nb1 Qe7 15. Ne5 Nxd2 16. Qxd2 Rac8 17. Rc1 Rc7
18. Rxc7 Qxc7 19. Nc3 Rc8 20. Na2 Qa5 21. Rc1 Rxc1+ 22. Nxc1)10... Nb4 (10...
Nb8 11. O-O Nbd7 12. Bb2 Bb4 13. Ra4 Bd6 14. Ra6 Ne4 15. Nc3 Nb6 16. Nxe4 Bxe4
17. Bd3 O-O 18. Bxe4 dxe4 19. Ne5 Bxe5 20. dxe5 Nd5 21. Qd4)11. O-O)10. Na3
(10. O-O Bxc2 (10... Nxc2 11. Ra2 cxb3 12. Rb2 Bd6 13. Ne5 Rc8 14. Rxb3 O-O
15. Bd2 Bxe5 16. dxe5 Nd7 17. f4 Be4 18. Nc3))10... Ne4 (10... cxb3 11.
cxb3)11. bxc4 Nc3 12. Qd2 Ne4)9... Nb4 (9... b4 10. bxc4 (10. Ne5 Nxe5 (10...
Rc8 11. bxc4 dxc4 12. Nxc4 Be7 13. Nbd2 O-O 14. Bd3 Nd5 15. Bxf5 exf5 16. a5
Qd7 17. Bb2 Rfd8 18. Qf3 Bf6 19. Nb3 Qe6)11. dxe5)10... Be7 (10... dxc4 11.
Bxc4 Bd6 12. Nbd2 O-O 13. Bd3 Bxd3 14. cxd3 Nd5 15. Nc4 Nc3 16. Qe1 Na5 17.
Bd2 Nb3 18. Bxc3 bxc3 19. Qxc3 Nxa1 20. Rxa1 Kh8)11. cxd5)10. Na3 (10. Ne1 a6
(10... Bd6 11. c3)11. Ba3 Ne4 12. Nf3 Bd6 13. c3 Nc6 14. Bxd6 Nxd6 15. Nbd2
O-O 16. Ne5 Nxe5 17. dxe5 Ne4 18. Nxe4 Bxe4 19. f3 Bg6)10... Ne4 (10... Rc8
11. c3 Nd3 12. axb5 Ne4 13. Bd2 Nb2 14. Qc1 Nd3 15. Bxd3 cxd3 16. c4 Nxd2 17.
Qxd2 Bxa3 18. Rxa3 dxc4 19. bxc4 Rxc4 20. Rxa7 O-O 21. Ne5 Rc2)11. Bd2 Bd6 12.
Be1 cxb3 13. Bxb5+ Kf8 14. c4 Nc2 15. Nxc2 bxc2 16. Qe2 dxc4 17. Ne5 Kg8 18.
Rc1 c3 19. Qxc2 Bxe5) 9. Bb2 (9. O-O e6 (9... Ne4 10. axb5 (10. Nfd2 Nxd2
(10... Nb4 11. Nxe4 Bxe4 12. axb5 axb5 13. Rxa8 Qxa8 14. Qd2 Na2 15. Ba3 h5
16. h4 Bg6 17. Bc5 e6 18. Bxf8 Kxf8 19. bxc4 dxc4 20. Nc3 Nxc3 21. Qxc3
Kg8)11. Nxd2)10... axb5 (10... Nb4 11. bxa6)11. Rxa8 Qxa8 12. Nfd2 Nf6 13. g4
Be6 14. Bf3 h6 15. Bg2 cxb3 16. Nxb3 h5 17. g5 Ne4 18. Nc5 Nxc5 19. dxc5 b4
20. Bb2 Qa2)10. axb5 (10. Nh4 Bb4 (10... Be4 11. f3 Bg6 12. Bd2 Bh5 13. axb5
axb5 14. Rxa8 Qxa8 15. Nc3 Qb7 16. bxc4 bxc4 17. Qa1 Bb4 18. Rb1 Qb8))10...
axb5 (10... cxb3 11. cxb3 axb5 12. Rxa8 Qxa8 13. Bxb5 Nd7 14. Bd2 Bd6 15. Nc3
O-O 16. Qa1 Qxa1 17. Rxa1 Rc8 18. Rc1 Ncb8 19. Nh4 Be4 20. f3 Bg6 21. Nxg6
hxg6)11. Rxa8 Qxa8 12. Bd2 Ne4 13. Nh4 Bg6 14. Be1 Qa1 15. Nxg6 hxg6 16. bxc4
bxc4 17. f3 Nd6 18. Bd2 Be7 19. Nc3 Qxd1 20. Rxd1 Nb4 21. Rc1) 9... e6 (9...
Nb4 10. Na3 (10. e4 Bxe4 (10... Nxe4)11. Na3 cxb3 12. cxb3 bxa4 13. O-O e6 14.
Bc3 Bd6 15. bxa4 O-O 16. Qd2 Nc6 17. Nc2 Qc7 18. Nb4 Bxf3 19. Bxf3 Rfc8 20.
Rfc1 Bf4)10... e6 (10... cxb3 11. cxb3 bxa4 12. bxa4 e6 13. O-O Be4 14. Bc3
Bd6 15. Bxb4 Bxb4 16. Nb1 O-O 17. Nbd2 Bd6 18. Nxe4 Nxe4 19. Bd3 Nc3)11. O-O)
10. axb5 (10. O-O Bd6 (10... Nb4 11. Na3 cxb3 12. cxb3 bxa4 13. Nh4 Be4 14. f3
Bg6 15. Nxg6 hxg6 16. Nc2 Qc7 17. Nxb4 Bxb4 18. Rxa4 Rb8 19. f4 Ne4 20.
Bxa6)11. g4) 10... axb5 (10... Nb4) 11. Nc3 Rb8 12. O-O Bd6 13. Nh4 Be4 14. f3
Bg6 15. Ra6 Qc7 16. f4 O-O 17. Nxg6 hxg6 18. bxc4 bxc4 *

==============================================================

Feature request

============

Could we have the option of feeding a pgn and the analysis starts from the last move ?

And the final output has the complete pgn and not just the analysis ?

For example, input pgn is...

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "New game"]
[Black "?"]
[Result "*"]
[PlyCount "9"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 *

So, the analysis should start from black's 5th move.

And the final pgn output is...

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "New game"]
[Black "?"]
[Result "*"]
[PlyCount "25"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 xxx 6.xxx xxx 7. xxx xxx 8. xxx xxx etc...

[Jupiter]

Thanks for sharing.

One thing I notice is that at the end of the requested depth, you append the pv analysis of the engine as the continuation but in Fritz DPA it follows the pv length option and is generated like engine plays against itself where each move is analyzed. I think Fritz's way is better as every move is analyzed whereas if the pv analysis is just appended, there is a high chance that the sequence of moves is not optimal. So I would like to request an option to turn off the appending of pv engine analysis at the end of the requested depth.

[zeFresk]

Thanks a lot for taking the time to test and for making suggestions !

I implemented all of your suggestions in version 1.1 :

• Added support of pgn files as suggested by @gvergh
• Added option '--no-appending' as suggested by @jupiter
• Output filename is now shorter and clearer

About Fritz not being able to read the output pgn I can't really try to reproduce it since I don't have either Fritz or Aquarium :(

I tried with Arena and some online pgn readers, they all worked fine, even with a +100ko pgn so it's really strange. (EDIT : I don't know why but I managed to see the bug with arena on old generated pgn, I didn't managed to reproduce it since v1.1)

EDIT : Why can't I edit my own message ? I had to delete it to post it again...

[Jupiter]

Just upgraded to latest v0.24.0 of python-chess.

There is error.

......

Completed position analysis 1 of 1 from game.pgn in 0 hours 0 minutes 14 seconds.

Saving result.

Traceback (most recent call last):

  File "dpa.py", line 344, in <module>

    main()

  File "dpa.py", line 320, in main

    append_variations(tree, last_node, args.depth)

  File "dpa.py", line 203, in append_variations

    append_variations_rec(tree[i], pgn, depth - 1)

  File "dpa.py", line 193, in append_variations_rec

    node = pgn.add_variation(move, str(cp))

TypeError: add_variation() takes 2 positional arguments but 3 were given

[gvergh...@gmail.com]

@ZeFresk

Thanks for the update.
The pgn feature works well !!

When I use --depth 15 --time=15, I get this...

Exploring position 1 of 1 : [rnbr2k1/1p3pp1/7p/p7/1nP5/4RN2/PP1B1KPP/RB6 b - - 1 16]...

>Analysing variation 1 of 357913941, estimated time remaining : 55h 22m...
>> 100% @ 542nodes/s : N8c6 (-0.53)

>Analysing variation 2 of 357913941, estimated time remaining : 737769h 42m...
>> 100% @ 600nodes/s : a3 (+0.84)

>Analysing variation 3 of 357913941, estimated time remaining : 897263h 53m...
>> 100% @ 235nodes/s : Na6 (-0.66)

What is your recommendation for good accurate analysis (depth 15+) with MultiPV of 4 ?

[Makis Nikas]

First of all thanks for sharing!

I second the erratic behavior of dpa.py along with python-chess 0.24.0

I've tested it with 0.23.10 and it works.

One other thing now. In the engine output when we have "score mate" instead of "score cp" dpa.py gives error [fatal].

I understand that this is highly unlike to occur in the opening, however when it happens the execution loop is broken and we cannot 

take the pgn output....

By the way (@zeFresk) have you thought handling somehow: 

    a. Transpositions and/or repeatitions 

    b. Backpropagating leaf node evals to root node by some kind of minimax approach? 

[Jupiter]

node = pgn.add_variation(move, str(cp))

 

That line has error, I got it fix with. 

node = pgn.add_variation(move, comment=str(cp))

same with other lines with add_variation()

[Jupiter]

Another good feature to add is descending multipv value, as tree gets deeper.

If position is white to move, set multipv to 4

Black to move, set multipv to 3

White to move, set multipv to 2

Black to move, set multipv to 1

If there is still depth, just use multipv to 1 to expand.

So at first level use 4, second level use 3, 3rd level use 2, 4th level use 1

4, 3, 2 and 1 is settable. This is also how Fritz DPA was done.

On Monday, December 3, 2018 at 6:34:09 AM UTC+8, zeFresk wrote:

[Makis Nikas]

Another one (misbehavior?) @line 101 [dpa.py]

..............................

    # get all moves in an array

    moves = []

    for i in range(1, pv + 1):

        with info_handler:

.........................................

what is happening when wanted pv is GREATER than available variations in position? 

[zeFresk]

Thank you for taking the time to test and report bugs ! It really helps !

@Jupiter : Thank you for bringing up and fixing this bug ! I will add your patch on the next release, I wasn't expecting python-chess to release a new version so soon !

Also I think you solved another bug I couldn't understand and reproduce at will so THANK YOU !! :)

I will also implement your suggestion, I wish I could have thought of it earlier as it would solved the problem of exponential growth !

@gvergh : Sadly it is normal and it comes with the growth rate of the exponential :(

When generating a tree of 4 moves per nodes with a depth of 15 you're in fact trying to create a tree containing... 1 + 4 + 4^2 +4^3 + ... + 4^14 nodes !

So about 357'913'941 nodes ! And since your exploring each node for 15 seconds it would indeed take more or less one hundred years of computation !

@Makis Nikas : Thank you for reporting these bugs, I found about them yesterday as well and I will correct them asap !

I will work on all of this as soon as possible but sadly I don't think I will able to do it before Friday :(

[gvergh...@gmail.com]

@zefresk
Thanks for the very lucid explanation !!
And it's my pleasure to test your creations !
A hundred years...hmmm...
I guess it's a choice of life vs chess and sadly, I'll have to choose life :(

[Jupiter]

I temporarily solve this with

legal_moves = board.legal_moves.count()

for i in range(1, min(pv, legal_moves) + 1):

zeFresk may have a general solution to this as it seems like there are other parts that needs to be updated.

[Jupiter]

There is also a general issue on getting the score. I am sure you already figured this out, but have to tell it anyway.

Under uci protocol there are 2 kinds of score, score cp and score mate. It seems like Lc0 only supports score cp value, but not score mate d. Currently your program would fail on score mate info, usually coming from A/B engines like Sf.

I don't know if this optimal but currently typically this is what I have.

try:

    score = int(info_handler.info["score"][1].cp)/100

except TypeError:

    score = mate_to_cp(info_handler.info["score"][1].mate)/100

Also add the mate to cp conversion. This the one used to adapt to Sf score.

def mate_to_cp(d):

    """ Returns score in cp based from mate in d """

    MATE_SCORE = 32000

    

    if d == 0:

        print('Warning!! the position is mate in 0')

        return 0

    

    if d < 0:

        value = -2*d - MATE_SCORE

    else:

        value = MATE_SCORE - 2*d + 1

        

    return value

Another issue that you may need to address and probably already figured this out is when a user specifies a depth that is longer than the optimal number of moves to mate.

1k6/4Q3/1K6/8/8/8/8/8 w - - 0 1

py -3 dpa.py -p sf10.exe --pv 2 --depth 3 --time 1 pos.epd --no-appending

Lets say the position is mate in 1 and the user requested depth = 3. There is no need to seach for the next depth if the first multipv chosen by the engine is a mate in 1, because the other side is already mated. I think python-chess has a method to determine terminal nodes.

Another feature that could be added is called evaluation threshold. At evaluation threshold of 80, if the best move has a score of 100 and the 2nd bestmove has a score of 50, so 100-50 = 50, since 50 < 80, the 2nd bestmove will be analysed further. If 3rd bestmove has a score of 10. 100-10 = 90, since 90 > 80, the 3rd bestmove will no longer be analyzed further and its move is not saved in pgn.

[Makis Nikas]

@jupiter

Bingo!

Exactly what I was thinking of...

 

Another feature that could be added is called evaluation threshold. At evaluation threshold of 80, if the best move has a score of 100 and the 2nd bestmove has a score of 50, so 100-50 = 50, since 50 < 80, the 2nd bestmove will be analysed further. If 3rd bestmove has a score of 10. 100-10 = 90, since 90 > 80, the 3rd bestmove will no longer be analyzed further and its move is not saved in pgn.

So after all technical bugs dust is settled, one IMHO should concentrate on effective ways to store output variations.

PGN format is known for its limitations to handle "degeneracies" like transpositions and/or repetitions.

So ideally one would need some kind of tree storage format (book: .ctg, .bin other?) to interact with...

@zeFresk && @jupiter do you think that this is feasible?

[gvergh...@gmail.com]

Got this error with lc0...is it linked to the 3-Fold move rule ?

Traceback (most recent call last):
  File "dpa.py", line 344, in <module>
    main()

  File "dpa.py", line 278, in main
    tree = explore_rec(board, engine, args.pv, args.depth, args.nodes, msec, args.appending)
  File "dpa.py", line 119, in explore_rec
    ret += [[(mo, cp), explore_rec(new_board, engine, pv, depth-1, nodes, msec, appending, tot)]]
  File "dpa.py", line 119, in explore_rec
    ret += [[(mo, cp), explore_rec(new_board, engine, pv, depth-1, nodes, msec, appending, tot)]]
  File "dpa.py", line 119, in explore_rec
    ret += [[(mo, cp), explore_rec(new_board, engine, pv, depth-1, nodes, msec, appending, tot)]]
  [Previous line repeated 3 more times]
  File "dpa.py", line 105, in explore_rec
    score = 0 if info_handler.info["score"][i].cp == None else info_handler.info["score"][i].cp # to avoid null cp at few nodes
KeyError: 3

Feature request

============

While the score threshold is a good addition to curtail needless expansion, I would also like an adjustable Max and Min node expansion score.

So, using the POV of the starting analysis and having a max of 300cp and a Min of -50cp, would result in this...

   An analysis session startng with black, will continue to get expanded as long as the POV node score is over -50cp and under 300cp.

   I do not need a mate evaluation, as most engines would convert 300cp positions.

 

[Jupiter]

PGN format is known for its limitations to handle "degeneracies" like transpositions and/or repetitions.
So ideally one would need some kind of tree storage format (book: .ctg, .bin other?) to interact with...

@zeFresk && @jupiter do you think that this is feasible?

Although the output of DPA is a single game with variations, this can broken down into separate individual game. Once done you can then remove the duplicates. These individual games with further manipulations like adding game results can be used for building a book like polyglot, arena and ctg books.

EPD's can also be generated out of this individual games, then duplicates can also be removed. A custom book can be created out of this EPD'S too. Like NN you can have

[epd1] [bm1, pr1] [bm2, pr2] ..[bmn, prn] [pos_pr]

[epd2] [...]

...

and so on

where pr1 to prn are the move probabilities, and pos_pr is the win probability of the current EPD position. These pr's can be calculated from the score cp output of the analyzing engine.

Single game

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start after that node } 6. Bc4 { 0.77 } ( 6. Bd3 { 0.77 } 6... g6 { 0.29 } ( 6... Nc6 { 0.29 } 7. Nxc6 { 0.67 } ( 7. Bc4 { 0.67 } ) ( 7. Be2 { 0.67 } ) ) ( 6... Nbd7 { 0.29 } 7. O-O { 0.52 } ( 7. a4 { 0.52 } ) ( 7. h3 { 0.52 } ) ) 7. Nf3 { 0.45 } ( 7. h3 { 0.45 } ) ( 7. O-O { 0.45 } ) ) ( 6. Bg5 { 0.77 } 6... e6 { 0.19 } ( 6... Nbd7 { 0.19 } 7. Be2 { 0.83 } ( 7. Bh4 { 0.83 } ) ( 7. Be3 { 0.83 } ) ) ( 6... Nc6 { 0.19 } 7. Qd3 { 0.74 } ( 7. Be2 { 0.74 } ) ( 7. Bc4 { 0.74 } ) ) 7. Qf3 { 0.67 } ( 7. f4 { 0.67 } ) ( 7. g3 { 0.67 } ) ) 6... e6 { -0.1 } ( 6... e5 { -0.1 } 7. Nde2 { 0.74 } ( 7. Nf5 { 0.74 } ) ( 7. Nf3 { 0.74 } ) ) ( 6... Qc7 { -0.1 } 7. Bb3 { 0.63 } ( 7. Be2 { 0.63 } ) ( 7. Bd3 { 0.63 } ) ) 7. a4 { 0.71 } ( 7. O-O { 0.71 } ) ( 7. Be3 { 0.71 } ) *

Individual game.

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bc4 { 0.77 } 6... e6 { -0.1 } 7. a4 { 0.71 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bd3 { 0.77 } 6... g6 { 0.29 } 7. Nf3 { 0.45 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bd3 { 0.77 } 6... Nc6 { 0.29 } 7. Nxc6 { 0.67 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bd3 { 0.77 } 6... Nc6 { 0.29 } 7. Bc4 { 0.67 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bd3 { 0.77 } 6... Nc6 { 0.29 } 7. Be2 { 0.67 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bd3 { 0.77 } 6... Nbd7 { 0.29 } 7. O-O { 0.52 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bd3 { 0.77 } 6... Nbd7 { 0.29 } 7. a4 { 0.52 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bd3 { 0.77 } 6... Nbd7 { 0.29 } 7. h3 { 0.52 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bd3 { 0.77 } 6... g6 { 0.29 } 7. h3 { 0.45 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bd3 { 0.77 } 6... g6 { 0.29 } 7. O-O { 0.45 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bg5 { 0.77 } 6... e6 { 0.19 } 7. Qf3 { 0.67 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bg5 { 0.77 } 6... Nbd7 { 0.19 } 7. Be2 { 0.83 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bg5 { 0.77 } 6... Nbd7 { 0.19 } 7. Bh4 { 0.83 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bg5 { 0.77 } 6... Nbd7 { 0.19 } 7. Be3 { 0.83 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bg5 { 0.77 } 6... Nc6 { 0.19 } 7. Qd3 { 0.74 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bg5 { 0.77 } 6... Nc6 { 0.19 } 7. Be2 { 0.74 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bg5 { 0.77 } 6... Nc6 { 0.19 } 7. Bc4 { 0.74 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bg5 { 0.77 } 6... e6 { 0.19 } 7. f4 { 0.67 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bg5 { 0.77 } 6... e6 { 0.19 } 7. g3 { 0.67 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bc4 { 0.77 } 6... e5 { -0.1 } 7. Nde2 { 0.74 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bc4 { 0.77 } 6... e5 { -0.1 } 7. Nf5 { 0.74 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bc4 { 0.77 } 6... e5 { -0.1 } 7. Nf3 { 0.74 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bc4 { 0.77 } 6... Qc7 { -0.1 } 7. Bb3 { 0.63 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bc4 { 0.77 } 6... Qc7 { -0.1 } 7. Be2 { 0.63 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bc4 { 0.77 } 6... Qc7 { -0.1 } 7. Bd3 { 0.63 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bc4 { 0.77 } 6... e6 { -0.1 } 7. O-O { 0.71 } *

[Event "?"]

[Site "?"]

[Date "?"]

[Round "?"]

[White "?"]

[Black "?"]

[Result "*"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 { Deep analysis start

after that node } 6. Bc4 { 0.77 } 6... e6 { -0.1 } 7. Be3 { 0.71 } *

You can use pgn-extract to split the single file into individual games.

[gvergh...@gmail.com]

"A custom book can be created out of this EPD'S too. Like NN you can have

[epd1] [bm1, pr1] [bm2, pr2] ..[bmn, prn] [pos_pr]

[epd2] [...]

...

and so on"

@Jupiter

I'm curious -- which opening book can be made from epd's ?

[Jupiter]

I'm curious -- which opening book can be made from epd's ?

Polyglot books.

[gvergh...@gmail.com]

On Friday, December 7, 2018 at 2:34:41 PM UTC-6, Jupiter wrote:

I'm curious -- which opening book can be made from epd's ?

Polyglot books.

You are very wrong.

Epd's are single-point references, which cannot be used for anything else other than analysis.

I guess you think you can convert the epd's to pgn and use polyglot.exe, or some other book making program.

How will a program know where to insert the epd in a tree ?

I think the only way would be with python-chess, but that's highly doubtful.

Even if such an algorithm was created, it would probably output rubbish.

[Jupiter]

The idea is to convert the epd to pgn, then pgn to polyglot book. I did this before. If you don't believe, go asks some experts.

[gvergh...@gmail.com]

On Friday, December 7, 2018 at 3:30:47 PM UTC-6, Jupiter wrote:

The idea is to convert the epd to pgn, then pgn to polyglot book. I did this before. If you don't believe, go asks some experts.

I don't know what to say to you.

Polyglot doesn't read fen positions, while making a book.

Maybe you should try it before talking about experts.

And to say you did it before, you're quite simply a liar, posting wrong information.

[zeFresk]

And... I'm back !

First of all I just wanted to say that I'm really happy to see so much people interested in my work !

I read all of your suggestions and I will start implementing them after fixing all the bugs you reported.

For the moment I fixed the most obvious one, when the program crashed if the engine provided less PV than requested by the user.

I also sped things up at low nodes count.

So right now the script should handle everything well, except when there is a mate (no 'cp') or when the node end in a draw.

I will fix that tomorrow but I wanted to fix the other bug first.

Thanks again to everyone for your support !! :)

[Jupiter]

Here is a hint. Go here http://talkchess.com/forum3/index.php, and ask the epd to polyglot book methods, some computer experts are posting here.

[Jupiter]

Welcome back !! Hope you are doing fine.

[gvergh...@gmail.com]

@Jupiter

You need to learn a lot.
Use that link for your benefit.
Also, read up on the manpage of polyglot.

I'm done with you but do keep arguing if you wish to.

[Makis Nikas]

Ok back on track....

Please split these loooong lines of code..... :)

Also for applying score thresholds it might be convenient to convert engine evaluations from side Point of view to more common White Point of view...

[Jupiter]

@gvergh 

Now you are backing off. But that is fine. Just try to check what I said, you will be able to build a polyglot book using epd. It is not just straightforward but possible.

[Jupiter]

Tried to look for a way to plot the tree output, but could not find it. Maybe you have ideas on how to visualize the tree output.

Here is a sample output using networkx.

If you can output below format that would be good too. The plot of this output is below.

Basically a python dict. The edges are the links, source, target connects the nodes, move is the attribute of the edge. Somehow the eval of the position should be near the node. Will look into it later.

tree.json file

{"directed": true, "multigraph": false, "graph": {}, "nodes": [{"id": 1}, {"id": 2}, {"id": 3}, {"id": 4}, {"id": 5}, {"id": 6}, {"id": 7}], "links": [{"move": "Nf3", "source": 1, "target": 2}, {"move": "d4", "source": 1, "target": 3}, {"move": "d5", "source": 2, "target": 4}, {"move": "Nf6", "source": 2, "target": 5}, {"move": "e6", "source": 3, "target": 6}, {"move": "d5", "source": 3, "target": 7}]}

[Jupiter]

The score of the move can be added in the edge attribute move using newline char \n

The json out that it can read.

{"directed": true, "multigraph": false, "graph": {}, "nodes": [{"id": 1}, {"id": 2}, {"id": 3}, {"id": 4}, {"id": 5}, {"id": 6}, {"id": 7}], "links": [{"move": "Nf3\n0.10", "source": 1, "target": 2}, {"move": "d4\n0.07", "source": 1, "target": 3}, {"move": "d5\n0.15", "source": 2, "target": 4}, {"move": "Nf6\n0.20", "source": 2, "target": 5}, {"move": "e6\n0.05", "source": 3, "target": 6}, {"move": "d5\n0.25", "source": 3, "target": 7}]}

[zeFresk]

And... I'm back again, after a bit (a lot ?) of work done since my last post.

I fixed both major bugs:

• Script crashing when score is given in "mate in X"
• Script crashing when position is mate or drawn.

I added a bug catching feature which generate a detailed report, so if you have a crash you can just post the report here :)

I also refactored and optimized the core exploration function. It is now almost 100x times faster at low nodes count (I can analyze 3k+ positions for 10k nodes in less than a minute using SF10).

I don't think the exploration function can be optimized much more without switching to C++, and I don't think it's needed.

I plan on refactoring the "main()" function later but it's not my priority.

My priority is to add all the features you mentioned, which is going to be way easier now that the code is more flexible.

You can see more in the changelog of the last release (v1.2)

@Jupiter : I already thought about using such tree format but it quickly become a mess, even with low pv and depth. Maybe I will add it later or create a separate script for converting an existing tree later on.

As usual, thanks a lot to everyone for helping me !!

[Jupiter]

I got an error when running v1.2

This is my command line:

python dpa.py -p sf10.exe --pv 2 --depth 2 --time 2 pos.epd

pos.epd contains:

1k6/5Q2/1K6/8/8/8/8/8 w - - 0 1

The position is mate in 1 and the depth specified is 2. I think I already reported this before.

The text below is from my screen. bug.log is empty.

Parsing args...

Setting-up engine

Exploring position 1 of 1 : [1k6/5Q2/1K6/8/8/8/8/8 w - - 0 1]...

>Analysing variation 1 of 3, estimated time remaining : 0h 0m...

>> 0% : ###

                                        

>> 1% @ 390.5knodes/s : Qb7# (+M1)

                                        

>> 1% @ 390.5knodes/s : Qb7# (+M1)

Congratulations, you found a bug ! A bug report is generated in bug.log

Please help me correct it by linking the report to your message :)

Traceback (most recent call last):

  File "dpa.py", line 150, in _explore_rec

    tot -= worst_case_tree_nodes(self.pv, depth-1)*(self.pv - self.info_handler.info["multipv"]) # We need to update its value because less nodes need to be explored

UnboundLocalError: local variable 'tot' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "dpa.py", line 439, in <module>

    main()

  File "dpa.py", line 373, in main

    tree = exp.explore(board, engine, args.pv, args.depth, args.nodes, msec, args.appending)

  File "dpa.py", line 106, in explore

    return self._explore_rec(board, depth)

  File "dpa.py", line 159, in _explore_rec

    self.log_bug("bug.log", board, depth, sys.exc_info())

  File "dpa.py", line 168, in log_bug

    fbug.write("bug in fen = [{:s}] with \"{:s}\", PV={:d} NODES={:d} DEPTH={:d}\n####\n{:s}\n\n".format(board.fen(), self.engine.name, self.pv, self.nodes, depth, exception_str))

TypeError: unsupported format string passed to NoneType.__format__

[zeFresk]

@ Jupiter : I'm dumb... I forgot to test again after refactoring...

Fixed in v1.2..1 thank you !

As for the second bug it should be fixed but I can't reproduce it, what version of Python are you using ?

[Jupiter]

C:\Windows\system32>python -V

Python 3.6.3

[zeFresk]

I coded all day and I'm happy to announce that v1.3 is out !

Here what's new :

Finally some more functionalities !

• Score is now normalized
• Added --threshold option
• Now supporting less common fen formats
• Fixed minor bugs
• Refactored most of the code

 I tried to add an option to decrease MultiPV every X plies but I wasn't happy with my implementation and decided to restart from scratch tomorrow.

Next I will try to be able to generate books in Polyglot format.

Do you have other ideas of functionalities I should add besides improving the tree format ?

To thank you all I added a deep analysis of the Petroff using Leela w 11250 at 800k nodes :)

PS : If you use Leela for deep analysis be careful with smart-pruning, I got some strange results wit it before.

Thank you again for you support !!

[Makis Nikas]

Houston we might have a small problem....

@zeFresk your coding skills are extraterrestrial compared to mine of a poor ex-sysadmin.

I barely can follow the logic of this object oriented magics...

Fortunatelly, you have realized your ideas in previous versions of dpa.py...

So here is my point.

In extending the root node at depths >= 5ply the so called GHI (Graph History Interaction) problem

would appear. (eg. Campbell "The graph-history interaction: on ignoring position history", 1985)

Going deeper than the 5th ply, transpositions and/or repeatitions will appear in the tree and 

the engine would eval and extend the same positions again and again. 

This clearly would cost time and resources.

So some kind of hash machinery is needed not to extend/eval the same positions.

A naive index would be the FEN representation of the position.

The implementation is all yours. Probably you have to store all fens and compare every 

new move fen with the ones stored....

---------------------------------------------------------------

Also you can split the pgn output to lines of certain length. 

Some pgn parsers can not understand well very very long lines.

Have a good night

[Makis Nikas]

Something is not backwards compatible...

python3 ./dpa.py -p "./cfs" --pv 6 --depth 3 --time 1   o-hg.pgn

Setting-up engine
PGN input detected we will only analyze from last position(s) reached.

Traceback (most recent call last):

  File "./dpa.py", line 550, in <module>
    main()
  File "./dpa.py", line 509, in main
    position_str = extract_fen(position_str)
  File "./dpa.py", line 188, in extract_fen
    return match[1]
TypeError: '_sre.SRE_Match' object is not subscriptable

I'm runing python 3.5.3 in debian stretch.

[Makis Nikas]

Around line 390 

   

 def get_pv_score(self, board, i):
        """Returns score associated to i-th PV formatted as a string. !!! WE SUPPOSE HANDLER IS LOCKED !!!"""
        return normalized_score_str(board, self.info_handler.info["score"][1].cp, self.info_handler.info["score"][i].mate)

Probably info["score"][1].cp should become info["score"][i].cp   ???

Else threshold cutoffs have unexpected results...

[jessi]

Jupiter, zeFresk

great JOB!

Good to see someone who cares about the graphical representation of a chess trees!
Imho this is one of the most desperately missing features in open source computer chess, whereas engines, data bases and tournament software are doing quit well.
Plots of chess trees would be a valuable tool for NN and also human training or for writing opening books and chess (e-)books.
Promising projects like chesstree.net got stuck somewhere in the middle (I don't see significant further development in the last years)

As I am not an experienced programmer my contribution will be limited to collaboration for a kind of duties record book or beta testing and feed back
My vision is a scalable high resolution graphic tree representation with some of the following features:
* showing >100 nodes on one page (don't waste space)
* zooming tree and text size, if text size get to small: omit
* toggle show / hide the text (move, eval) with 1 key stroke
* representing the eval by the color of the node and link (e.g. -5: red +5: green, soft transition)
* alternative: xy-plot: eval on one axis (left black wins, right white wins) and move number on the other axis with automatic scaling / magnification of the x-axis around the selected node
* focus on a selectable node with magnification of the surrounding part of the tree
* hover over node gives a board representation
* automatic joining of transpositions
* start with a platform form high resolution monitors (not mobiles)
* collapsing subtrees
* rerooting
* collaboration with visualization projects and statistic software

as an example here are the features of "dendroscope" http://ab.inf.uni-tuebingen.de/software/dendroscope/

    Large trees with hundreds of thousands of taxa can be easily displayed, browsed and edited
    Multiple trees and networks from a single file can be displayed together in an m by n grid
    Novel magnifying features for zooming detailed views (see screenshots);
    Find and replace tool bar that uses regular expressions;
    Subtrees can be collapsed and colored;
    All labels (leaves/inner nodes and edges) can be edited;
    Trees can be rerooted;
    Seven different views are available, including a rectangular, slanted, circular and radial view;
    Input formats: Newick and Nexus, extended-Newick (for rooted phylogenetic networks) and Dendroscope;
    Multiple graphic export formats: .eps, .svg, .png, .jpg, .gif, .bmp, .pdf;
    Trees and networks can be copied and pasted between different windows
    Platform independent (Java, installers for common operating systems available)
    Consensus trees and rooted phylogenetic networks can be computed from a set of trees
    Hybridization networks and tanglegrams for multifurcating trees on unequal taxon sets
    Commandline mode
    
For sure we should start simple and look for further development as more people participate.
Atm the main issue is to fill some devs with enthusiasm for an open source project for a sophisticated chess tree visualization!

[Henrik Skov Midtiby]

I would suggest you to look at xdot.py (https://github.com/jrfonseca/xdot.py) for making a highresolution plot of the search tree, that you can navigate in a GUI.

This is able to draw graphs defined in the Graphiz dot language.

[gvergh...@gmail.com]

@zeFresk

Thanks for the update and including my request !!

The threshold limits work fabulously :)

[gvergh...@gmail.com]

This is strange -- v1.3 doesn't output more than one pgn for each position analysed ?

[zeFresk]

Quick update :

I fixed 3 bugs in v1.3.1, thank you @Makis Nikas for reporting them !

@jessi and @Henrik Skov Midtiby : Until now I wasn't sure about generating graphical representations, your messages made me change my mind !!

Sadly you will have to wait for a major rework to be done first :(

First I'm planning to add one last feature (decreasing PV as depth increase), because I think it's very much needed to be able to analyze deeper.

Then I would like to do a major refactoring and switch from internally generating a tree to generating a map. The map would then need to be reconverted to a tree as to be able to generate a pgn.
Doing this would allow me to take care of repetitions and transpositions. It would then be much easier to export it as a book or as graphical tree, killing two birds with one stone. The cherry on top is that the script would be much faster.

But sadly this rework will take a lot of time and I'm busy until Saturday :(

Again I want to thank you all for testing and submitting new ideas, that helps me a lot and I'm really happy to see so much people interested !

[zeFresk]

Ahaha, @gvergh you found the third bug 5 minutes before I pushed the commit fixing it !

It is patched in v1.3.1 as well :)

[Makis Nikas]

First I'm planning to add one last feature (decreasing PV as depth increase), because I think it's very much needed to be able to analyze deeper.

 

@zeFresk

 

Needless to say, please make everything tunable (as you do)  since the main objective here is not ELO performance but visualization. 

For example, the absolute value in threshold constraints might be a little unflexible depending on the instability/asymmetry of the chess position

and the stage of game (opening, middle, end). Perhaps upper-lower bounds is desirable. However, this choice is not crucial...

Pruning in PV should be done very very carefully since otherwise you amplify the risk of excluding "tunnel variations" (variations that lead to win/advantage beyond engine's horizon). 

which is a major issue in opening stage.

Going deeper is not nessecary a one way proceess but you can always shift-displace root node due to eg human intuition reasoning.

Good night to all

 

[jessi]

zeFresk,

good news, thx for re-thinking graphics.

A quick way would be to translate PGN trees in the dendroscope-input format: Newick

PGN:

1.e4 e5 ( 1...c5 2.Nf3 ) ( 1...e6 2.d4 ) ( 1...c6 2.d4 ) 2.Nf3 Nf6 *

=> Newick format

(((Nf6)2.Nf3)e5,(2.Nf3)c5,(2.d4)e6,c6)1.e4;

tree graphic can be tested e.g. at http://etetoolkit.org/treeview/ but not some cute as dendroscope (needs installation)