What might be the reason behind Leela's draws in TCEC...

[Vassilis]

I have watched all Leela's games so far in divP, and there seems to be a pattern common to all.
After the opening, Leela plays fairly fast (every move in about 1min more or less). However she manages to reach a better and sometimes a winning position. But in the critical moment, with plenty of time in clock, she plays even faster, only to reach a dead drawn endgame. It is there where Leela starts thinking long, but it's already too late. Apparently something is wrong with Leela's time management!

Note that when I say fast, I don't mean moves that has already precalculated and stored in cash, but moves that has to think after an unexpected response by her opponent.

What is your opinion on this?

[Stephen Timothy McHenry]

You may have something there they need to look into on the shell version side of things, but as far as the network goes, 32194 is still just a baby and I would need to see a mature 30net do the same things consistently before making any changes to a future Leela version (.22? or something)

[Tony Asdourian]

Post this idea on Discord.  I think they have already been talking about Leela's TM in TCEC, but it couldn't hurt to repost there just in case.

[Vassilis]

Hi Stephen.

I also agree about the infancy of 32194. But most of Leela's inaccuracies are made in the first few seconds while oscillating her decisions between two or more possible moves, only to choose the wrong one because of her fast play. I believe that if she were thinking longer, probably her evaluation would have being stabilized to the correct move. The same behavior is observed in other engines too. But they think longer and make their move when their evaluation is not probable to change again.

Hi Tony.

How would I do that?

[123]

 Vassilis:

The reason behind the draws is, that Leela has a very weak tactical level.

This can easily be improved to 3000 or more tactical elo, but it isn't done yet, because they won't use the chess 960 positions to train Leela.

But you can do a new net which is training from the chess 960 start positions and you will see after 100.000 training games it is much stronger than our Leela after 100.000 training games.

Cost you only 3 training days with an RTX 2080 Ti.

[Vassilis]

Hi 123 :)

I agree with you, that Leela is perhaps the weakest tactical engine, among the competition.
And surely, this has an major impact, on the quality of her games!
But this could be because of her search algorithm and not necessarily because of her training...

I also agree with you, and is worth trying, that starting from a randomly chosen chess 960 position, instead of the initial one, would improve Leela's understanding about the game, and would have led to more powerful play more quickly. But she would still suffer from tactical issues, because of the way she does the search.

I only hope that the developers read this, and try to implement your idea, in one of their future tests!

Vas

[Vassilis]

I'm mostly convinced, that Leela's tactical weakness, is a search issue!
Otherwise how do I explain the fact she can't find a mate in 9 when her search depth reaches above 19?
Just see her last loss against Stockfish, at the TCEC bonus match...

[Tryfon Gavriel]

In TCEC Bobus, She is currently leading in this match:

https://tcecbonus.club/

10.5 to 9.5 - with 3 wins 2 losses and 15 draws which is pretty impressive for that particular ID LCZero v20-rc2-32329 against Stockfish 18122721

So perhaps we just need to try and get a network more fully evolved to be more tactical etc.

There is a need to be more tactically incisive against the engines of Premier. One win would give her 3rd equal - two wins would be second equal. So at the level of the premier division it is pretty tough to win any games, but also very tough not to lose any games. She is managing not to lose any games so far which is also impressive for that even lower ID for the main premier division.

[Vassilis]

[...There is a need to be more tactically incisive against the engines of Premier. One win would give her 3rd equal - two wins would be second equal. So at the level of the premier division it is pretty tough to win any games, but also very tough not to lose any games. She is managing not to lose any games so far which is also impressive for that even lower ID for the main premier division.]

Couldn't agree more, Tryfon :)

Like you, I've closely watched every Leela's game, and in most of them (with one exception against Houdini, if i recall) she gains the advantage quickly, sometimes reaching a winning position, and then she misses some good moves, and equalizes! Chess is a tactical game, and if Leela has reached such a strength due to her positional understanding, one can imagine where she would be, if tactically weren't so "weak"

Cheers, be well, and keep doing all these instructive (and inspiring) videos for us!

I follow them closely :)

Regards...

Vas

[Gökcan AKALIN]

It is possible that selfplay games quality need to be upgraded, when the NN eval function gets strong enough; because there is no way to disprove her illusions other than having deeper selfplay games instead of 800 nodes. As the quality of games increases, she may come to correct some of her illusions; because chess is a tactical game which sometimes requires precise long sequences; with 800 nodes she will always have to depend on her search algorithm. Perhaps it is best to increase the number of nodes in her self play in order to increase the quality of her games, when she reached this level of strength. Leela already learned the most about chess, but she needs to learn the nature of precise long sequences which sometimes decide the fate of the game. It is obvious she is blind to some unlikely tactical situations, due to her training experiences in shallow selfplay games.

30 Aralık 2018 Pazar 11:51:46 UTC+3 tarihinde Vassilis yazdı:

[Vassilis]

Expect some drop in performance after this change, though.
It's because you change the learning policy, so the NNs have to readjust their weights to other direction. But in the end, sure, she will improve more, no doubt.

[M MUSTERMANN]

Gökcan AKALIN:

Leela already learned the most about chess, but she needs to learn the nature of precise long sequences which sometimes decide the fate of the game. It is obvious she is blind to some unlikely tactical situations, due to her training experiences in shallow selfplay games.

30 Aralık 2018 Pazar 11:51:46 UTC+3 tarihinde Vassilis yazdı:

[...There is a need to be more tactically incisive against the engines of Premier. One win would give her 3rd equal - two wins would be second equal. So at the level of the premier division it is pretty tough to win any games, but also very tough not to lose any games. She is managing not to lose any games so far which is also impressive for that even lower ID for the main premier division.]

Couldn't agree more, Tryfon :)

Like you, I've closely watched every Leela's game, and in most of them (with one exception against Houdini, if i recall) she gains the advantage quickly, sometimes reaching a winning position, and then she misses some good moves, and equalizes! Chess is a tactical game, and if Leela has reached such a strength due to her positional understanding, one can imagine where she would be, if tactically weren't so "weak"

Regards...

Vas

That Leela already learned most about chess is completely wrong and also not possible.

Wrong because in theorie you need to learn from 100 books to increase you strength from 1000 to 2000 elo. 

But later at Leela's level (let's call it the 3000 books level) she needs to learn from 3000 new books to improve by 100 elo. 

So the fact is that Leela must at least double her chess knowledge and that means: Leela is far away from already learned most about chess.

Sooner or later Leela will be on much higher level and then to improve by 5 elo she will need to double her chess knowledge.

Not possible because she is learning from 1 initial position. - To learn from 1 initial position sounds like a joke by such a complex game like chess.

You can increase this factor by 960 using the chess960 initial positions. Lots of this positions leads to extrem difficult tactics, which are even to difficult for Stockfish dev. and Houdini tactical version.

This would easily boost Leela's tactical level from ~2300 elo to above 3500 elo.

The positional and strategical skills will also be improved.

[Vassilis]

Well, M.MUSTERMAN, learning from the starting position only is not a joke, as long as the learning process, allows random deviations, early on for better exploration of the game space. Every training game played is unique and follows its own path. Besides you can see by yourself the results of this training :)

This is just a small remark... I totally agree with everything else you said.
Especially for the chess 960 training idea. I'm sure the developers have thought of that also.
Sure some rules are different in chess 960, for example castling, but that's irrelevant with the learning process, as Leela will learn about different, "exotic", positions, not rulles.

Maybe they haven't done it because there is something we are missing here...
Or maybe they just haven't done it... yet!

Cheers
Vas

[Markus Kohler]

Still real endgames are rare and the self play does not play until the end, which means for Leela a lot of positions in mid game are won, but she only rarely learns how to really end the game

[Ingo Weidner]

Today against Fire 7.1 Lc0 ID 32194 finally scored the first win and now with 5.5 points after 10 games is at 4th place, 1.5 points ahead of Fire 7.1 at 5th place and 0.5 points behind Komodo which is at 2nd place:

https://tcec.chessdom.com/live.html

Stockfish with 7.5 points after 10 games (and 5 wins) seems to play iin it's own league so the question is which engine will reach 2nd place.

[Lothar Jung]

Komodo got his only two wins from his little brother which crash twice.

[Alexey Eromenko]

If it can hold a draw against the big three (Stockfish, K and H), then it's not a baby anymore.