Leela's impressive non-win at TCEC

[Jim Glass]

https://tcec.chessdom.com/

Round 76, second place +28 -0 Leela versus last place +1 -36 Johnny.

For 30 moves Leela considers herself up at least 5 pawns, repeatedly 10, and as much as 19.

Meanwhile, Johnny calls forced mate for her again and again - by my count more than 33 times!

Result: Draw.

Even with table bases Leela didn't find even one of the 30+ forced mates called by Johnny.

Sometimes she reminds me of my own old tournament playing days.

I haven't examined the game, maybe someone who does will find something interesting about her play that explain this.

[Ivan Ivec]

Leela is tactically weak. She wins on strong positional style.

For some combinations very high depth is needed.

Leela needs cupercomputers to show full streength.

[Alexey Eromenko]

Cuper-puper-duper-computers will not help, since Leela will simply migrate to a bigger neural network. 40x512?

What is needed is to find a way to combine a small network, say 6x64 and a big network for analysis.

[Todd Rohrer]

The way I understand neural networks work, it means that leela started out with just the rules and after 250 million games she played against herself was finally "learned" enough to defeat stockfish the six year reigning world champ. After this tournament it appears leela , since she didnt lose a single game, might not ever lose another game because the only one who would pose a threat to her is stockfish, and since neural net engines get better and better, and she already beat stockfish, in theory she should never lose another chess game. 

So Leela gets better and better and the other engines are kind of stuck where they are and so Leela will eventually have to just play against herself to get better.

I'm not sure how the other engines are going to get better.

So tournament 15 Leela did lose some games but now after this one she didnt lose any games, the only engine that didnt lose and so tournament 15 was the last tournament Leela lost in and now should never lose another game, unless it some sort of short timed game or something... She was winning in that last game of the tournament easily but the "over 50 moves" rule went into effect.

[Deep Blender]

Leela is for sure going to lose games in the future to "traditional" chess engines. The tactical weaknesses it has are clearly a lack of training for those kinds of positions or insufficient capacity. I personally assume it could be improved with a more focused training on those positions, but of course, there is no way for me to prove that.

At one point, Leela is going to miss a sharp move from an opponent and will lose because of it. Resolving that is definitely not going to be trivial.

[Deep Blender]

I agree with you that a combination of different networks is likely the way to go. Unfortunately, that's an open research question and it is not even known how to do that for way simpler problems.

[123]

Ivan Ivec:

Leela is tactically weak. She wins on strong positional style.

For some combinations very high depth is needed.

Tactic can be improved very easily.

LC0 is trained with our main start position this is one problem.

Chess960 positions should be used, because they have much more tactics, maybe 100000-500000 times more.

LC0 would get also a little more new positional and strategical positions.

For some combination very high depth is needed, that's true but also in this case it's somehow nonsense.

Feel free to show me a depth, 20 moves tactical combination.

Nearly every weak 1800 elo player from my chess club can calculate the tactics which LC0 missed. I'm pretty sure they can't calculate 20 tactical moves.

They are calculating only 1-6 moves and can see what LC0 missed. So depth is no problem!

LC0 can avoid this problem also by using a 40x512 instead of 40x256 or 20x256 network.

Again depth is no problem.

Even if we would have depth 50 in every position, someone would say it's not enough and they want more, because some other person have depth 60 due to better hardware.

If someone believe they have not enough depth then it's time to buy a stronger gpu or to buy one of the newest gpus or to buy a second, third... gpu!

And not to cry every time: oh I want to use the newest things and I know I need some other things to run it with more power but I do not want to pay money xDDD. Find a way to give me more power without paying for it.

It's like: oh I want to drive a tesla but I do not want to pay so much money.

[Shah]

Let's not forget: Despite her severe tactical weakness... Well, Leela is the ONLY engine that did not lose a single game in this event...

(-:

[Deep Blender]

Do you have evidence for that or is it a guess?

[Ivan Ivec]

Feel free to show me a depth, 20 moves tactical combination.

I'm not a position collector, but most mate in 20+ positions start with unique move.

You can find some of them in 7-men tablebases.

[Jim Glass]

Nobody has any thoughts on the game itself?

[Robert Filter]

Sorry Jim, I have shown to be too weak and too slow to use the search at TCEC.

Anyhow, I think we should simply reconsider that "strength" is actually combined by a lot of things such as openings, positional understanding, endgame handling, tactics, etc. Leela is just extremely good at certain things while others are a bit weaker. Since the distinction to traditional engines is so clear, imho a combination of two kinds of engines will be the way to go, at least for some time.

[Stephen Bell]

In case there is someone who wants to see the game but can't find it, it's here: https://cd.tcecbeta.club/archive.html?season=15&div=rapid&game=546

[Shah]

This endgame, which is a close variant of the Q vs QR, is a sure win.
But it requires a very accurate and a very long line of moves. Any small deviation from this line, and the 50 move rule kicks in. I.e. you must play "perfect play" to win against a strong opponent. Leela, is not built to play perfect game. She is all about approximations.
What's even worse for Leela, is that along this long line, no positional improvement can be hoped to be seen.
(E.g. it is not like in each move you restrict the space of the opponent king)
So Leela is a blind in the dark here...
So it should not come as a surprise that Leela can't win the Q vs QR nor that she missed this particular "sure win".
But on a personal note, those endgame are more math than they are chess.
If TB adjudication would have been used, Leela would win this.
She is still the best chess player, and we still love her. (-;

[Jesse Jordache]

I have thoughts on the game.  It's not a result of weak tactical training or weak endgame training.

It's the search method: MCTS will never, ever find the win.  It's almost like a trick position - it's like when you fool A/B engines into misevaluating something by putting up a pawn fence and making sure that anything decisive occurs beyond their horizon.  You can do the same thing to MCTS engines by giving them nothing but heavy pieces and then clearing out the board.  How is Leela supposed to find the winning method?  Step one is to bring the White pieces up and position the rook on the seventh rank.  But the reason that's necessary isn't at all obvious.  The problem is all moves are reasonable, and as aggressively as MCTS prunes back a position, an engine with that search engine will never find it, keeping in mind that any positional change is about 15 moves off.

btw, 5 centipawns doesn't mean anything to an NN.  She calculated the position as winning for White - that's all you need to know.  The irony is that under her old evaluation, it would have been a win by adjudication.  Also, for the sake of completeness, you ought to mention that Leela was in incremental time for the entire endgame.  But I thought at the time - and I still think - that an hour on the position wouldn't be enough.  But I haven't put that to the test.

Anyway, I think people expecting a computer engine that doesn't have a width-first algorithm to solve that sort of position needs to learn what different approaches do well and what they don't do well.

 

On Thursday, June 27, 2019 at 10:33:24 PM UTC-4, Jim Glass wrote: