Testing Watcher...

[Andrea Gatti]

Hi,

I'm playing a bit with the Tutor and I'm a bit confused regarding the output that I get. Please refer to the attached image that I took while doing a test. Of course, please disregard the game quality itself, I started playing some moves around to trigger the Tutor.

Here is what I don't understand:

1) I played 10.Nxe4 and the Tutor commented that this was a very bad move, however, it was giving me a clear advantage. Also the evaluation still says +2.28 after black's move.

2) In the Watcher window, I can't understand the meaning of what's written. Shouldn't we have a black move following the "after:"?

The way I read it, for example for watcher's first suggestion is:

"Playing 9.Nb3 you lost 101 centipawns because of: [what black's move??]. New eval is -0.13; the best move was Bxe4".

Am I missing anything here?

[Johan Sjöblom]

Good testing. I will try a quick explanation here. Just ask more if needed.

1. If you have debug logging there should be more information in the logs/picochess.log around that 10th move with a full listing of the best moves seen by the tutor. The challenge with the tutor (watcher) is that it starts for white immediately after black has moved, but if the next white move arrives too quickly the tutor might not have had enough time to get to a depth of 17 which it needs for a good evaluation.

I think the learning here is that I should add some depth number or other information that shows the level of confidence of the watcher. If the depth is below 10 we should probably not add anything to the watcher list as its not based on good enough depth information.

2.  "Playing 9.Nb3 you lost 101 centipawns because of: [what black's move??]. New eval is -0.13; the best move was Bxe4".
It means: Playing 9. Nb3 you lost 101 centipawns compared to the best move found by the tutor Bxe4. The eval is for the situation after your white move. The centipawn loss should be the diff between the best move eval and the chosen move eval. The quality again of this line depends on if the tutor reached a 17 depth. The tutor calculates 50 different lines, so it takes a while to reach the 17th depth. The 17 and 50 factors are found in tutor_constants. If you think you always chose one of the best 10 moves you can reduce the 50 to 10. Then the tutor will probably be faster.

Tutor definitely needs more testing. If you have a log from your test I can analyse it and find out if tutor actually made a mistake or if it just had too low quality.

[Andrea Gatti]

Thank-you. This evening I will look into it an try to post here the log file. Maybe I can also try to suggest a different wording (just my opinion) for the watcher's analysis.

In my opinion, the Tutor fuction does need a bit of additional test and maybe tweaking. Reason is that I feel there is like a sync issue between the tutor's voice comments is, and the eval's of the engine in use and that of the engine used by the tutor. In a few occasions I heard Luigi's voice claiming my move was very bad, but indeed it was good, and also the engine's evaluations were in my favor..

Maybe that part of the issue is related to a timing factor? I mean that Luigi's verbal comments take approx 5 seconds. Five seconds for an engine is like an eternity. An eval that was minus whatever five seconds ago, can be whatever different after five seconds. But again I could be totally wrong - I'm just thinking loud.

I don't feel I can trust the Tutor for the time being. Hopefully my tests and your coding abilities will fix it :-)

Thanks!

AG

[Johan Sjöblom]

We have not been testing the tutor heavily for a while. Its still the same evaluation algorithm as in V3 but now we can see it on screen in the WATCHER window. There has never been any quality gate before. I now added one. If the depth of the tutor best line, and the line chosen by the user is below depth 12 it will not show anything in the WATCHER window. This makes tutor results more accurate.

I also show the depth at the end of the WATCHER line, so d12 is maybe not so good, d17 is the target of the tutor. Below d12 we dont show anything any more.

I checked your PGN move list, and the move 10. Nxe4 is correctly flagged as a significant loss of centipawns as the move 10. Nxd5 is so much better. I think you can trust the tutor, especially now as I added a quality gate of minimum depth of tutor analysis to be d12.

I also improved the words on the WATCHER line, but feel free to suggest more improvements.

Before you continue testing, please update your picochess using the install-picochess script, or by update picochess from the menu.

Notice that you can make the tutor 3 best lines visible on your move-list window by chosing Display -> PGN Lines from the menu.

-- Johan

[Andrea Gatti]

Johan, super. I will update this evening and start some more testing. I will then report back to you. I'm willing to contribute to it via my test sessions.

Thanks

AG

[Andrea Gatti]

Btw, what engine is used by the various Pico Tutor functions? Can I configure what engine to use, and what UCI params shall this engine use (like threads, hash, etc?)

[Johan Sjöblom]

Nice. Grazie Mille! We definitely need more testing. It was maybe a year since I last analysed the picotutor, but I am trying to analyse it again now as I have not touched it for a long time. I noticed the current setting is to analyse 30 best moves in picotutor_constants.py (ROOT_MOVES). It means as soon as the engine has moved it asks the tutor engine to analyse 30 top lines... and it takes a few seconds to reach depth 17. If it does not reach 17 the quality might not be good enough.

And we might have some strange audio delays ...

Happy testing, let me know what you find. I will continue to analyse the picotutor to see if it is calculating as it should.

-- Johan

[Johan Sjöblom]

I think in picochess.ini you can configure any engine to be used as tutor... That engine will use the settings from its uci file.

You could use the latest stockfish, but we have been conservative and used the last non_NNUE version 16 as the default tutor. But you can change it. It should be better if it can use the latest Stockfish. LC0/Maia are not good for line evaluations so I dont recomend LC0 as tutor.

I updated the picotutor code slightly again, so if you have time, update picochess once more before continuing testing.

-- Johan

[Andrea Gatti]

Sure, will do thanks!