Lc0 for correspondence chess analysis

1,257 views
Skip to first unread message

Martin Cacheiro

unread,
May 31, 2019, 4:31:51 AM5/31/19
to LCZero
Hi,

I'm using lc0 for correspondence chess analysis. I've got a RTX 2070 graphics card and a Ryzen 7 processor with 8 cores (16 threads). My settings are:

cudnn-fp16
MaxCollisionEvents 1024
Minibatch Size 1024
NNCacheSize 999999999
3 threads

I've used 20x256 and 40x256 networks but I have the same problem with both. The speed at the beginning is around 25-30 kN/s but after a couple of hours it starts decreasing (maybe because it stops analysisng and the average goes down) until 2-3 kN/s. The depth of analysis also stops, or at least goes much, much slower. It can go up to 20-25 very quickly but then it stays at this depth or increases very slightly (1 or 2 plies in more than 4-6 hours).

Any ideas/suggestions to optimize lc0 for correspondence chess analysis (settings, networks, etc...)?

Thanks a lot in advance for your help. 

John D

unread,
May 31, 2019, 11:31:14 AM5/31/19
to Martin Cacheiro, LCZero
My inclination is that leela is particularly suited for relatively quick analysis at the root, its search is so inefficient that hours-long thinks are probably a waste. 

That said if you haven’t done so already I’d use TCEC settings as your baseline. They’re in the PGNs of all the SuFI games, tho hopefully someone can list them here. After a few million nodes I’d just clip the results and if theres anything interesting check it in depth with SF (but I’m not a corr player, so take that fwiw)

 
From: lcz...@googlegroups.com on behalf of Martin Cacheiro <martinc...@gmail.com>
Sent: Friday, May 31, 2019 3:31 AM
To: LCZero
Subject: Lc0 for correspondence chess analysis
 
--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/323f79f8-6ee5-43ad-95b8-8b99c10da153%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Daniel Rocha

unread,
May 31, 2019, 12:16:33 PM5/31/19
to LCZero
Isn't it cheating to use a computer to play chess against other humans? I am not making a judgment here, I really don't know.

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/323f79f8-6ee5-43ad-95b8-8b99c10da153%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Daniel Rocha - RJ

Mattchess

unread,
May 31, 2019, 7:49:29 PM5/31/19
to LCZero
In ICCF it is accepted to use engines - think of ICCF as the "academia lab environment" of the chess world.

Regarding use of Leela, I use Leela to seed additional lines for analysis - but I use Stockfish for IDEA and minmax.  I think it is interesting when you see a major difference between what LC0 thinks of a position and what Stockfish thinks of a position and that is an area worth more investigation. 


On Friday, May 31, 2019 at 9:16:33 AM UTC-7, Daniel Rocha wrote:
Isn't it cheating to use a computer to play chess against other humans? I am not making a judgment here, I really don't know.

Em sex, 31 de mai de 2019 às 05:31, Martin Cacheiro <martinc...@gmail.com> escreveu:
Hi,

I'm using lc0 for correspondence chess analysis. I've got a RTX 2070 graphics card and a Ryzen 7 processor with 8 cores (16 threads). My settings are:

cudnn-fp16
MaxCollisionEvents 1024
Minibatch Size 1024
NNCacheSize 999999999
3 threads

I've used 20x256 and 40x256 networks but I have the same problem with both. The speed at the beginning is around 25-30 kN/s but after a couple of hours it starts decreasing (maybe because it stops analysisng and the average goes down) until 2-3 kN/s. The depth of analysis also stops, or at least goes much, much slower. It can go up to 20-25 very quickly but then it stays at this depth or increases very slightly (1 or 2 plies in more than 4-6 hours).

Any ideas/suggestions to optimize lc0 for correspondence chess analysis (settings, networks, etc...)?

Thanks a lot in advance for your help. 

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lcz...@googlegroups.com.

Mattchess

unread,
May 31, 2019, 7:51:59 PM5/31/19
to LCZero
I am currently using the network from TCEC 256x20.T8-swa-610000

Mattchess

unread,
May 31, 2019, 7:53:39 PM5/31/19
to LCZero
https://github.com/LeelaChessZero/lc0/wiki/Third-Party-Nets

Sorry for multiple posts - could not figure out how to edit my old post

Stuart Cracraft

unread,
May 31, 2019, 9:41:53 PM5/31/19
to LCZero

Who are the primary authors of Leela and what is their preferred contact method?


M MUSTERMANN

unread,
Jun 1, 2019, 3:10:57 AM6/1/19
to LCZero
Martin Cacheiro:
Sounds like there is at least one bug.
I heard from some people, they have the same problem. 
Some of them have two gpus and also the same problem. Like having 60 or 70 or 80 kn/s and losing 25% in the next ~5 minutes and altogether 50% after some more minutes.
This is possible because RTX gpus are new gpus and we have no experience.

Ivan Ivec

unread,
Jun 1, 2019, 5:17:39 PM6/1/19
to LCZero
Try to reduce NNCacheSize. I think that you run out of memory...

Alexander Lyashuk

unread,
Jun 1, 2019, 5:45:12 PM6/1/19
to Ivan Ivec, LCZero
Lc0 keeps entire search tree in memory, so the longer it thinks, the more memory it takes.
Roughly it takes 250 bytes per node in tree, plus 350 bytes per NN cache item.
NN cache of 999999999 items requires ~325GB of RAM.

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/00403a9d-30a8-471b-9246-b00945dcd5ee%40googlegroups.com.

Daniel Rocha

unread,
Jun 1, 2019, 8:09:13 PM6/1/19
to Alexander Lyashuk, Ivan Ivec, LCZero
So, when Leela crashes, does it decrease the performance?


To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/CACGjms%2Bw4B_yx1puaLaMoKA42XFwvX%2BJhh97uge6kjxOYCevsg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

David Larson

unread,
Jun 2, 2019, 2:42:17 AM6/2/19
to LCZero
I don't think this is true. Lc0 caches the search tree and the cache has a maximum size. There is a perf hit for a cache miss so it's usually best to maximize the cache size.


On Saturday, June 1, 2019 at 2:45:12 PM UTC-7, Alexander Lyashuk wrote:
Lc0 keeps entire search tree in memory, so the longer it thinks, the more memory it takes.
Roughly it takes 250 bytes per node in tree, plus 350 bytes per NN cache item.
NN cache of 999999999 items requires ~325GB of RAM.

On Sat, Jun 1, 2019 at 11:17 PM Ivan Ivec <ivan...@gmail.com> wrote:
Try to reduce NNCacheSize. I think that you run out of memory...

Dana petak, 31. svibnja 2019. u 10:31:51 UTC+2, korisnik Martin Cacheiro napisao je:
Hi,

I'm using lc0 for correspondence chess analysis. I've got a RTX 2070 graphics card and a Ryzen 7 processor with 8 cores (16 threads). My settings are:

cudnn-fp16
MaxCollisionEvents 1024
Minibatch Size 1024
NNCacheSize 999999999
3 threads

I've used 20x256 and 40x256 networks but I have the same problem with both. The speed at the beginning is around 25-30 kN/s but after a couple of hours it starts decreasing (maybe because it stops analysisng and the average goes down) until 2-3 kN/s. The depth of analysis also stops, or at least goes much, much slower. It can go up to 20-25 very quickly but then it stays at this depth or increases very slightly (1 or 2 plies in more than 4-6 hours).

Any ideas/suggestions to optimize lc0 for correspondence chess analysis (settings, networks, etc...)?

Thanks a lot in advance for your help. 

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lcz...@googlegroups.com.

Alexander Lyashuk

unread,
Jun 2, 2019, 3:32:16 AM6/2/19
to David Larson, LCZero
In Lc0 cache works like kind of transposition table, so if the same position occurs as a result of different sequence of moves, it doesn't have to reevaluate that position in NN.
The tree itself is always kept completely, and it doesn't depend on cache settings.
In MCTS you have to do that. There may be some techniques to prune branches (which Lc0 currently doesn't have) to squeeze some more visits, but it won't be anywhere as close as A/B engines where total number of nodes is basically unlimited.

To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/848e8f11-ff23-4e7c-a5a9-5598879090b2%40googlegroups.com.

Martin Cacheiro

unread,
Jun 2, 2019, 4:54:38 AM6/2/19
to lcz...@googlegroups.com, Alexander Lyashuk

Ok, I understand. I have 64 Gb RAM so I’ll reduce NNCache to around 200000000. But I guess the same will happen, won’t it? As son as it fills that RAM it will stop. Is the conclusión that for a fixed RAM there is a fixed time for which you can run Leela for analysis?

Martin Cacheiro

unread,
Jun 2, 2019, 4:55:49 AM6/2/19
to lcz...@googlegroups.com

In the end, what would be your settings to maximize correspondence analysis with a computer with 64 GB of RAM?

 

De: lcz...@googlegroups.com <lcz...@googlegroups.com> En nombre de Alexander Lyashuk
Enviado el: domingo, 2 de junio de 2019 9:32
Para: David Larson <itsallth...@gmail.com>
CC: LCZero <lcz...@googlegroups.com>
Asunto: Re: Lc0 for correspondence chess analysis

 

In Lc0 cache works like kind of transposition table, so if the same position occurs as a result of different sequence of moves, it doesn't have to reevaluate that position in NN.

Ivan Ivec

unread,
Jun 2, 2019, 6:07:34 AM6/2/19
to LCZero
For 16GB I set 8000000. I had similar problems with 10000000 or more.

So, for 64GB i will try 32000000, at most 40000000. Also, in your GUI reduce Hashtable size to 16GB.

I think these two caches should be together below free memory available.

--
You received this message because you are subscribed to the Google Groups "LCZero" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lcz...@googlegroups.com.

Ivan Ivec

unread,
Jun 2, 2019, 12:04:38 PM6/2/19
to LCZero

Ivan Ivec

unread,
Jun 2, 2019, 4:03:55 PM6/2/19
to LCZero
Martin,

there is an option RamLimitMb.

Something like RamLimitMb=60000 should do fine on your system.

Ivan Ivec

unread,
Jun 2, 2019, 4:30:22 PM6/2/19
to LCZero
Uhhh, reaching the limit stops the search.

Unfortunately, there is no infinite analysis without infinite memory. I hope this could be solved in near future...

Otherwise, correspondence chess players will need 1TB of RAM.
As I expected, this sport will become more expensive than ever...
Reply all
Reply to author
Forward
0 new messages