Observations about Leela's play (NN 200)

[Stephen Frost]

I finally got around to setting up Arena and worked out how to set up Leela to play some other engines.  Played a few matches using NN 200 against the bundled Rybka engine.  Without exception, Rybka beat Leela at all time controls.

My setup is a simple laptop with 8GB RAM and 4-core latest gen Intel CPU.  I allocated 2 CPU cores to each engine.  Didn't change much in the other settings and just let them play.  Tried 15+5 and also 30+10 time controls.

I'm only a mid-strength chess player, Elo around 1650, so obviously I don't have any master-level insights.  But Leela at this level/hardware seems to suffer from a few shortcomings which she will need to overcome in the next month or two if we want to see her rating/ability climb higher.

1.  she seems far too often to accept a damaged pawn structure (e.g. doubled and isolated pawns) for herself, under-estimating the impact when an endgame arrives

2.  Leela seems to over-estimate the importance of advanced pawns in the middle-game (e.g. I've seen her sacrifice a piece in the middle-game in order to get two passed pawns on the 6th and 7th rank, but her opponent simply sat their king in front of them and they were impotent)

3.  she also seems to have an issue with valuing pieces/pawns in exchanges (e.g. a number of times she has given up R + 2P for B+N) and this might not be so bad, except she then falls into the next item #4

4.  she seems to drop pawns far too often, which when combined with the previous item, has seen her fighting with B+N against R+3P or similar, or eventual B vs 4Ps

5.  Leela consistently assesses her position as more positive than the opponent's assessment of it (not sure what's going on there, but having an overly optimistic view of your position will kill you in chess!)

I know this is an exploratory exercise and its amusing for me, just an average punter, to be pointing out stuff like this, when Leela is far stronger than me already.  Still, I do know a bit about the game and I would really like to see Leela continue to improve.  

To leave something positive:  

One of the games Leela played yesterday against Rybka, she actually got to a potentially winning position.  Rybka was assessing millions and millions of positions; Leela barely 5000 per move.  Both engines assessed it as a clear advantage to White.  But her pawn structure was damaged badly (see #1 above) and her assessment of +0.8 wasn't shared by Rybka (see #5 above) and unless she could find a tactical win, I could see that the endgame would likely end in tears.  Leela had an opportunity to repeat the position for a three-fold repetition draw, but then chose a sub-optimal move and tried to continue the game.  She was then outplayed and eventually lost.

I'll be running Leela vs Rybka mini-tournaments once a week and will watch for any changes, to results, but also to tactics and positional play.

[Stephen Frost]

One more fun thing which is encouraging, is seeing Leela playing some different openings, sometimes with colours reversed.

Yesterday in the games she played against Rybka, I saw Leela with White play a Pirc classical system (1. e4 d6 2. d4 Nf6 3. Nc3 g6 4. Nf3 O-O 5. h3).  Now she ended up playing e5 (see #2 above) and then supported it with Bf4 (when you normally would see maybe Bf3 or perhaps Bg5?) but it was nice to see that sort of thing appear on the board.

I've also seen some Benoni-type structures appear with colour reversed.

[gravity_well]

curious if you know your laptop's graphics capabilities?  is it a card? or integrated?

[Stephen Frost]

On Sunday, April 29, 2018 at 7:34:05 AM UTC+10, Stephen Frost wrote:

Yesterday in the games she played against Rybka, I saw Leela with White play a Pirc classical system (1. e4 d6 2. d4 Nf6 3. Nc3 g6 4. Nf3 O-O 5. h3).  Now she ended up playing e5 (see #2 above) and then supported it with Bf4 (when you normally would see maybe Bf3 or perhaps Bg5?) but it was nice to see that sort of thing appear on the board.

Typo there.  "Bf3" should read "Be3" of course. 

[Dietrich Kappe]

Do have a gpu? Running on a cpu, Leela will be much weaker than Rybka.

[Tadeusz R]

Stephen, you are running Leela on very low hardware, she needs decent GPU. See my short tournament few posts below: LCZ vs Rybka.

[Stephen Frost]

On Sunday, April 29, 2018 at 7:35:40 AM UTC+10, gravity_well wrote:

curious if you know your laptop's graphics capabilities?  is it a card? or integrated?

The laptop is a not-very-expensive Dell Inspiron 5570 with Intel 4-core i7-8550U CPU @ 1.8GHz.

Being a laptop, I expect the graphics hardware is all on the motherboard and/or in the CPU.  Don't really want to take the cover off and double-check as the laptop is still under warranty.  The graphics drivers show two items:

Intel UHD Graphics 620

(see:  https://www.notebookcheck.net/Intel-UHD-Graphics-620-GPU-Review-Benchmarks-and-Specs.239936.0.html)

-- GPU boost speed from 300MHz up to 1150MHz

Radeon 530

(see: https://www.notebookcheck.net/AMD-Radeon-530-GPU-Benchmarks-and-Specs.214534.0.html)

Basically entry-level stuff, not a lot of grunt, low power to preserve battery life.

[Stephen Frost]

On Sunday, April 29, 2018 at 7:48:10 AM UTC+10, Tadeusz R wrote:

Stephen, you are running Leela on very low hardware, she needs decent GPU. See my short tournament few posts below: LCZ vs Rybka.

True.  I don't have such hardware though (I'm not a gamer), but I do like to tinker.  It will be interesting to see whether, in time, the new iterations of the NN will improve sufficiently that she can overcome that impediment regardless. 

[Dietrich Kappe]

If you buy a gamer laptop with an NVIDIA card like a gtx1060 or better, you should be able to get Leela to perform. Neural nets use the massively parallel linear algebra processing of GPU’s to get performance. With the hardware you describe, you won’t be able to compete with Rybka.

[Jesse Jordache]

Leela NEEDS the gpu.  If you've worked with linear algebra to solve two-or-three (or more) dimensional geometry, or else used it to model changes over time, you'll see why a GPU is more appropriate than a CPU.
If not, basically what she's doing is a HUGE amount of comparatively simple parallel operations on variables within a giant multi-dimensional array, some which affect each other, some which don't..  While the operations themselves are simple, but they have to all be done before she can take the next step, and there are a lot of them.  Computationally it's exactly like what computers do when they're rending objects in three dimensional space: it's why she runs off of the graphics card.

Compare that to a traditional chess engine, which is a multi-multi-variable algorithm that's incredibly complex and has to be performed sequentially.  That's the CPU's job.

[Rudolf Posch]

I too played test games against the bundled engines installed with the chess GUI Arena 3.5.1.

I show here 2 test tournaments 5 min +1 s/move against weaker engines  on my Intel i5 - 3570K PC with a fully trained GPU (GeforceGTX660) LCZero  NN Id 208:

LCZero 0.7 Id 208 versus Ruffian 1.05 : 11-6-3  (11 win - 6 draws - 3 lost)      20 games  (Ruffian 1.0.1  has in the Swedish SSDF ELO-List  an ELO of 2677)

LCZero 0.7 Id 208 versus Nejmet  3.07:  4-0-0   (I stopped after 4 games)

 Greetings Rudolf from Austria

[Vennotius]

Because Leela has a handicap when running on a cpu against another engine, you can always handicap the other engine to even things out. Set rybka to use 10% of its time, for example.

The people who have got good gpu's would be able to give a better percentage based on the difference they see between leela running on their cpu vs gpu.

[Rudolf Posch]

Sorry, error in my previous post:  LCZero vs Ruffian: 11 wins 3 draws and 6 lost instead of 11 wins 6 draws and 3 lost 

[Stephen Frost]

I'm trying a few different things at the moment.  Longer time controls may help, so I am trying a tournament with 60min+10sec.  Have also found where to tweak the engine parameters in Arena, so have set LC Zero NN200 and NN210 with SlowMover=100 to see if that also make any difference.