Creating yet another Lc0 benchmark on gpu/cpu

[Jupiter]

Trying to compile yet another benchmark. Generally interested to get the nps of v0.17.0 using network id 11198, using start position with go movetime 120000 command.

I am aware of the other benchmark done in google sheet, but I like to have a cleaner table.

Procedure:

1. Setup lc0 v0.17.0 to use default settings

You may use/revise lc0.config and define the network id.

Just use 2 threads first, you may run other settings if you still have time and post results here.

Example entry of your lc0.config file

weights=C:\chess\engines\lczero\v0.17.0\weights_11198.txt.gz

2. Use network id 11198

Lets use this id first to establish a baseline. Other id's maybe added if you are still interested to post your data.

3. Run or double click the lc0.exe

4. In console type 

uci

isready

(wait for the engine to be ready)

ucinewgame

position startpos

go movetime 120000

5. Wait for around 2 minutes and remember/record the nps after the search.

6. I am interested on the following info and kindly post here.

a. GPU name

b. GPU count or number of gpu you used

c. CPU name

d. Threads used (this is 2 by default)

e. Username that you like to use

f. lc0 exe type (cuda, opencl or blas)

g. Other info that you think is important

The priority right now is the nps using nvidia gpu, but you can still post non-cuda exe if you like, it will be recorded at site later.

I have created a site

https://lc0bench.netlify.com/

to show typcial data. Right now the data, are just dummies, but the cuda cores and memory are correct based on a given gpu (press the green + at left of row). I will update the table

once I get some data from you.

In the future I will be expanding this to include test sets as this table is filtereable and seachable thanks to datatables.

https://datatables.net/

[acepoint]

Am Montag, 3. September 2018 11:52:19 UTC+2 schrieb Jupiter:

6. I am interested on the following info and kindly post here.

a. GPU name

Nvidia Geforce GTX 1080

 

b. GPU count or number of gpu you used

1

c. CPU name

Intel i8700k

d. Threads used (this is 2 by default)

2

e. Username that you like to use

acepoint

f. lc0 exe type (cuda, opencl or blas)

0.17 cuda

6069 nps

Ciao

acepoint

[Alexander Lyashuk]

Notes:

1. Due to smart pruning (a.k.a futile-search-aversion), it will stop search earlier than 2 minutes, and that saved time may differ by a lot because of pure luck. It's better to use `--futile-search-aversion=0` command line parameter.

2. It was shown that startpos is not very indicative for game nps. I recommend picking some early middlegame position instead.

It would be nice if someone implemented benchmark command for lc0, to run as './lc0 benchmark` which would do that automatically, probably also trying several positions and outputting more data.

I've added benchmark.lczero.org as an alias to lc0bench.netlify.com, but something has to be done at server side to recognize that host name.

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/c6d958a2-ea11-46c8-ac2a-0046088898b9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[terrylee...@gmail.com]

a. GTX1060

b. 1

c. i7-7700HQ

d. 2

e. jtsbad

f. cuda

3111nps

[terrylee...@gmail.com]

GTX1060 6GB

0.17 cuda

[Jupiter]

1. Due to smart pruning (a.k.a futile-search-aversion), it will stop search earlier than 2 minutes, and that saved time may differ by a lot because of pure luck. It's better to use `--futile-search-aversion=0` command line parameter.

Thanks for this info. 

It may stop the search or not does not matter because the main idea is to bench the default of v0.17.0 on nid 11198 using startpos with command go movetime 120000, whatever is the reported nps will be recorded.

  

2. It was shown that startpos is not very indicative for game nps. I recommend picking some early middlegame position instead.
It would be nice if someone implemented benchmark command for lc0, to run as './lc0 benchmark` which would do that automatically, probably also trying several positions and outputting more data.

This is true indeed, because startpos is far from middle and too far from ending.

I actually started writing a program to bench lc0, this is external not inside the lc0, using different positions.

lc0bench.exe --engine lc0.exe --inputbench lc0bench.pgn --movetime 120000 --output lc0benchresult.csv

I am in the process of collecting positions.

Collection of bench positions.

(1) Opening/EarlyMiddle Phase

Famous game from Kasparov.

r2qr1k1/5ppp/p4n2/1pbP1bB1/1n6/N1N2B2/PP1Q1PPP/3R1RK1 b - - 1 16

Position after whites 16th move.

[Event "Karpov - Kasparov World Championship Match"]

[Site "Moscow URS"]

[Date "1985.10.15"]

[EventDate "?"]

[Round "16"]

[Result "0-1"]

[White "Anatoly Karpov"]

[Black "Garry Kasparov"]

[ECO "B44"]

[WhiteElo "?"]

[BlackElo "?"]

[PlyCount "80"]

1.e4 c5 2.Nf3 e6 3.d4 cxd4 4.Nxd4 Nc6 5.Nb5 d6 6.c4 Nf6 7.N1c3

a6 8.Na3 d5 9.cxd5 exd5 10.exd5 Nb4 11.Be2 Bc5 12.O-O O-O

13.Bf3 Bf5 14.Bg5 Re8 15.Qd2 b5 16.Rad1 Nd3 17.Nab1 h6 18.Bh4

b4 19.Na4 Bd6 20.Bg3 Rc8 21.b3 g5 22.Bxd6 Qxd6 23.g3 Nd7

24.Bg2 Qf6 25.a3 a5 26.axb4 axb4 27.Qa2 Bg6 28.d6 g4 29.Qd2

Kg7 30.f3 Qxd6 31.fxg4 Qd4+ 32.Kh1 Nf6 33.Rf4 Ne4 34.Qxd3 Nf2+

35.Rxf2 Bxd3 36.Rfd2 Qe3 37.Rxd3 Rc1 38.Nb2 Qf2 39.Nd2 Rxd1+

40.Nxd1 Re1+ 0-1

(2)

EarlyMiddle/Middle

(3)

MIddle/LateMiddle

(4)

LateMiddle/EarlyEnd

(5)

EarlyEnd/MiddleEnd

(6)

MiddleEnd/LateEnd

(7)

LateEnd

Famous rook ending by Capablanca.

5k2/p1p4R/1pr5/3p1pP1/P2P1P2/2P2K2/8/8 w - - 0 35

Position after black's 34th move.

[Event "New York"]

[Site "New York, NY USA"]

[Date "1924.03.23"]

[Round "6"]

[White "Jose Raul Capablanca"]

[Black "Savielly Tartakower"]

[Result "1-0"]

[WhiteElo "?"]

[BlackElo "?"]

[ECO "A40"]

[EventDate "1924.03.16"]

[PlyCount "103"]

1. d4 e6 2. Nf3 f5 3. c4 Nf6 4. Bg5 Be7 5. Nc3 O-O 6. e3 b6 7. Bd3 Bb7 8.

O-O Qe8 9. Qe2 Ne4 10. Bxe7 Nxc3 11. bxc3 Qxe7 12. a4 Bxf3 13. Qxf3 Nc6 14.

Rfb1 Rae8 15. Qh3 Rf6 16. f4 Na5 17. Qf3 d6 18. Re1 Qd7 19. e4 fxe4 20.

Qxe4 g6 21. g3 Kf8 22. Kg2 Rf7 23. h4 d5 24. cxd5 exd5 25. Qxe8+ Qxe8 26.

Rxe8+ Kxe8 27. h5 Rf6 28. hxg6 hxg6 29. Rh1 Kf8 30. Rh7 Rc6 31. g4 Nc4 32.

g5 Ne3+ 33. Kf3 Nf5 34. Bxf5 gxf5 35. Kg3 Rxc3+ 36. Kh4 Rf3 37. g6 Rxf4+

38. Kg5 Re4 39. Kf6 Kg8 40. Rg7+ Kh8 41. Rxc7 Re8 42. Kxf5 Re4 43. Kf6 Rf4+

44. Ke5 Rg4 45. g7+ Kg8 46. Rxa7 Rg1 47. Kxd5 Rc1 48. Kd6 Rc2 49. d5 Rc1

50. Rc7 Ra1 51. Kc6 Rxa4 52. d6 1-0

I can make the program feed all the sequence of moves up to the specified bench position if that matters.

Collect nps at 3 different phases

1. Opening

2. Middle

3. Ending

That is only one criteria (famous positions from famous games) in selecting positions, We may include, positions with pins, ep, castle, promote (with minor promotions), mate in 1, 2 and others.

But first lets bench this startpos.

[Jupiter]

Site updated with 2 entries so far.

https://lc0bench.netlify.com/

Thanks for data contributors.

[acepoint]

Forgot my iMac ;-)

Nvidia GTX 775M 2GB RAM

GPU 1

Intel i7 3.5 GHz (iMac late 2013)

Threads 2

lc0 0.17 cuda (self compiled)

nps 784

Ciao

acepoint

[Naphthalin]

On Monday, September 3, 2018 at 10:05:38 PM UTC+2, Georg Schröter wrote:

6. I am interested on the following info and kindly post here.

a. GPU name

GTX 760 4 GB

b. GPU count or number of gpu you used

single GPU

c. CPU name

i7 4770K

d. Threads used (this is 2 by default)

default

e. Username that you like to use

Naphthalin

f. lc0 exe type (cuda, opencl or blas)

CUDA

g. Other info that you think is important

as recommended above i used the --futile-search-aversion=0 option

 1049 nps

[Jupiter]

Thanks, but I will not enter this in the table at this time as I need nps from default.

[Jupiter]

This is added.

Better still if you can identify the processor, this is not urgent.

[Naphthalin]

I reran it with default settings, getting 1029 nps as a final result now.

[brian]

a. Titan V

b. 1 

c. e5-2699v4 ES

d. 2

e. asdffdsa

f. lc0 cuda-9.2

g. benchmarked on linux

11221 nps

[Jupiter]

Site updated with this info.

[Jupiter]

Site updated with this info.

[Jupiter]

1. Due to smart pruning (a.k.a futile-search-aversion), it will stop search earlier than 2 minutes, and that saved time may differ by a lot because of pure luck. It's better to use `--futile-search-aversion=0` command line parameter.

A bit off topic, I have a comment regarding this futile-search-aversion option, I read the source code at https://github.com/LeelaChessZero/lc0/blob/master/src/mcts/search.cc

and if I am right, this option is related to kAggressiveTimePruning if this is so, I would suggest that this time pruning should be disabled when a user uses

go movetime

command, to follow the uci specs as much as possible.

Relevant code, just saw it, am not familiar what UpdateRemaniningMoves() is for, have not searched deeper.

void Search::UpdateRemainingMoves() {

  if (kAggressiveTimePruning <= 0.0f) return;

[...]

}

UCI specs on movetime

* movetime 
    search exactly x mseconds

Lc0 may not be able to search exactly the time specified in movetime, but perhaps would also try not to use any time pruning when searching. Is the time pruning used so that Lc0 would not exceed the time specified in movetime? or any other commands related to time such as go wtime x btime y etc?

Link to uci specs:

http://wbec-ridderkerk.nl/html/UCIProtocol.html

On my slow computer using cpu only, tried testing the difference between default and not.

Lc0 v0.17.0 blas

position startpos

go movetime 120000

(1)

futile-search-aversion=0

time 12261

diff = 122611 - 120000 = +2611

(2)

futile-search-aversion=1.33 (default)

time = 117412

diff = 117412 - 120000 = -2588

So using futile-search-aversion=0 exceeds the time limit of 120000 by 2611 ms.

Whereas using futile-search-aversion=1.33 (default) searches lesser time than the specified 120000 by 2588 ms.

Default is more preferable because it does not exceeds the time limit.

[Jesús Jareño]

Hi all,

 

a. NVIDIA GeForce GTX 870M (GK104M)

b. 1

c. Intel Core i7-4810MQ

d. 2

e. jarechu

f. lc0 exe cuda  (cudnn)

g. nps 922

[acepoint]

Am Montag, 3. September 2018 12:24:29 UTC+2 schrieb acepoint:

Am Montag, 3. September 2018 11:52:19 UTC+2 schrieb Jupiter:

6. I am interested on the following info and kindly post here.

a. GPU name

Nvidia Geforce GTX 1080

I've noticed that other users also mention their GPU RAM: here it is 8 GB.

Ciao

acepoint

[Naphthalin]

In my case, search stopped after 85s with standard settings while taking extra ~200ms (so 120200ms) with smart pruning disabled. It's not about not exceeding the given time, it's about cutting the search early when the highest priority move won't change in the remaining x seconds. If you want to do a comparable GPU benchmark across positions and nets you should definitely decide to use `--futile-search-aversion=0`.

[Jupiter]

Std Memory config of gpu is not necessary, I search it myself. along with cuda cores.

[Jupiter]

Your info just added. Thanks.

https://lc0bench.netlify.com/

[tun nay]

Hello,

a. NVIDIA GeForce GTX 1060 , 3GB 

b. 1

c. Intel Core i5 , 7400, 3gh

d. 2

e. Nay Tun

f. lc0 exe cuda  (cudnn)

g. nps 3281

Regards,

Nay 

[Jupiter]

Site updated with your data. Thanks.

https://lc0bench.netlify.com/

[Jupiter]

For gtx 1060, user should supply gpu memory as there are 2 flavors in it, 6gb, and 3gb, same with gtx 1050 also has 3gb and 2gb, but for gtx 1080 no problem, I can search this in web. That is what I mean in my first reply.

So if the user has gtx 1060, or gtx 1050, he should supply the gpu memory.

On Tuesday, September 4, 2018 at 8:22:11 PM UTC+8, acepoint wrote:

[tun nay]

Hehe, nice to see my 1060, 3GB  performs better than 1060,6 GB

Additional info--, Nvidia card setting and windows 10 were tweeted a lot for maximum performance. (no microsoft bloatwares and services etc) 

[Andreas Strangmüller]

a. NVidia GeForce GTX 980M
b. 1
c. Intel i7-6820HK
d. 2
e. fastgm
f. Lc0 v0.17.0 cuda

2573 nps

Andreas

[Jupiter]

Data added. Thanks.

Changes:

* To compare nps in the list, added nps ratio, the value of nps/max_nps in the list

* Show user name instead of hiding it in the child rows so that when there are more than 1 type of gpu, it can be identified direcly by user name.

https://lc0bench.netlify.com/

[Robert Clark]

1. NPS:  74
2. GPU:  AMD Radeon HD 5570
3. Count of GPU used: 1
4. GPU Overclock: [Yes, 1800MHz]
5. CPU name: i7-2600K  4.2GHz
6. Number of threads used by Lc0: 2
7. rlc33
8. Lc0.exe type: opencl

You can see why I am not bothering to run training games .....

--

You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/86517f64-7b61-40fd-a0b0-0cff5ae87a1e%40googlegroups.com.

[Robert Clark]

Sorry, I reported GPU clock wrong. Here is the corrected info:

1. NPS:  74
2. GPU:  AMD Radeon HD 5570
3. Count of GPU used: 1

1. GPU Overclock: [Yes, 650MHz]

1. CPU name: i7-2600K  4.2GHz
2. Number of threads used by Lc0: 2
3. rlc33
4. Lc0.exe type: opencl

[Jupiter]

Thanks, data is added.

As I see it, GPU is not overclocked. It is the CPU that is overclocked because it is 4.2 Ghz and base frequency of i7-2600K is only 3.4 Ghz

https://ark.intel.com/products/52214/Intel-Core-i7-2600K-Processor-8M-Cache-up-to-3_80-GHz

Also GPU at 650 Mhz is at stock frequency.

https://www.amd.com/en-us/products/graphics/desktop/5000/5570

Have a look at the data at site. Check if they were right.

https://lc0bench.netlify.com/

[Laporda]

GTX 1070 Ti

Intel i7-4771 @ 3.50 GHz

2 threads

Laporda

cuda

5931 nps

[Archi]

What's the point of using strictly default settings when on V100 with def config nps is around 15k, but if you set back-end to cuda-fp16, nps will be over 30k

[Dan Kelly]

Do all GPU's support fp16?

On Thu, Sep 6, 2018 at 1:39 AM Archi <verc...@gmail.com> wrote:

What's the point of using strictly default settings when on V100 with def config nps is around 15k, but if you set back-end to cuda-fp16, nps will be over 30k

--

You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/7aeaa81d-47e2-42b4-8ed1-b61b163cd995%40googlegroups.com.

[Jupiter]

The point is to determine the nps output of Lc0 v0.17.0 at a given network id using only the default options. This is not about tweaking the Lc0 options to get the maximum nps, this is more like establishing a baseline where that baseline uses the default options.

Once the benchmark of default options is established, we can for example create a new benchmark where we allow Lc0 options to be modified. This way we will be able to see what will be the impact to nps when we change certain options.

More data is better so we can get a meaningful nps average later.

But this is only for startpos, part of the plan is to test it also on different middle game positions and take the average nps. I will be sending later a program that will do it automatically. 

Then next plan is to test it on test suites (start at tactical) and try to get the results for solved positions.

[Margus Riimaa]

Processor Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, 3201 Mhz, 6 Core(s), 12 Logical Processor(s)

NPS 9015

1 GPU Aorus Nvidia 1080Ti 1683 Hz -Gaming Mode (not overclocked)

Default (2) threads

margus_est

Lc0 type - Cuda

[Jupiter]

This is added. Thanks.

https://lc0bench.netlify.com/

[Jupiter]

This is added. Thanks.

https://lc0bench.netlify.com/

[Vivek Moroni]

Reporting three entries:

Nvidia GeForce GTX 1050 Ti  4GB;  Intel i7-970 6-core @ 3.4GHz

• CUDA
• nps = 2208
  

Nvidia GeForce GTX 1060 6GB;  Xeon E5-2690 v.4 14-core @ 2.65GHz

• CUDA
• nps = 4157

ASUS ROG STRIX GeForce GTX 1080 Ti 11GB;  i9-7900X @ 4.00GHz

• CUDA
• nps = 9326

[Galaga]

The nps value calculated from startpos is not a constant:

I'd made a few repetitions:  9383, 8053, 8422, 8733, 8966, 8957
Same with parameter --futile-search-aversion=0 as suggested by Naphthalin
8659, 9128, 8267, 8170

Mean value: 8700

Processor Intel(R) Core(TM) 3930K i7-3930K CPU @ 4.0GHz, 3200 Mhz, 6 Core(s), 12 Logical Processor(s)
NPS 8700
1 GPU MSI Nvidia 1080Ti 1683 Hz -Gaming Mode (not overclocked)
Default (2) threads
Galaga
Lc0 type - Cuda

I suspect the multitasking OS and probably "Number of worker threads" as cause for the scatter on nps.
At least on my computer the deviations are quite large - can anyone confirm this?

[Francesco Tommaso]

Isn't it the case that since the Monte Carlo shows random possible moves, which activates certain parts of the NN, each one having its mathematical transformations, is normal to have different nodes counting?

[Jupiter]

Thanks, later I will add this.

[Jupiter]

Thanks. The mean for 9383, 8053, 8422, 8733, 8966, 8957 is 8752 so I will take this mean value.

With that data.

SD (Sample Standard Deviation) = 466

M (Mean) = 8752

For futile-search-aversion (fsa)=0

SD = 436

M = 8556

The fsa=0 SD is still close to 466 so this has nothing to do with the fsa set to 0.

It is indeed interesting to see the data from other user, at the moment we cannot say if your SD is high or low unless we can see data from other user having a similar hardware or close to your hardware specs. or close to the average nps that you have. It is possible that at a given nps of say [4000-5000] the SD may vary a lot from your recorded SD. Perhaps you can create a separate thread for this.

[Jupiter]

Table updated, added gpu brand, sub-brand and average nps for each unique gpu name. Open the child row by pressing the green + button at left of row, see image 1.

Image 1:

Site:

https://lc0bench.netlify.com/

[Cordella]

Sorry, I am a newbie. I am confused because it detected 2 OpenCL Platforms. Here is my result:

...

uciok

isready

Found pb network file: E:\Downloads\Softwares\arena_3.5.1\Engines\Leela-017Final/weights_11198.txt.gz

Creating backend [opencl]...

OpenCL, maximum batch size set to 16.

Initializing OpenCL.

Detected 2 OpenCL platforms.

Platform version: OpenCL 2.0 AMD-APP (2482.5)

Platform profile: FULL_PROFILE

Platform name:    AMD Accelerated Parallel Processing

Platform vendor:  Advanced Micro Devices, Inc.

Device ID:      0

Device name:    Hainan

Device type:    GPU

Device vendor:  Advanced Micro Devices, Inc.

Device driver:  2482.5

Device speed:   1030 MHZ

Device cores:   5 CU

Device score:   1120

Device ID:      1

Device name:    Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz

Device type:    CPU

Device vendor:  GenuineIntel

Device driver:  2482.5 (sse2,avx)

Device speed:   1800 MHZ

Device cores:   8 CU

Device score:   520

Platform version: OpenCL 2.1

Platform profile: FULL_PROFILE

Platform name:    Intel(R) OpenCL

Platform vendor:  Intel(R) Corporation

Device ID:      2

Device name:    Intel(R) UHD Graphics 620

Device type:    GPU

Device vendor:  Intel(R) Corporation

Device driver:  23.20.16.4905

Device speed:   1100 MHZ

Device cores:   24 CU

Device score:   621

Device ID:      3

Device name:    Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz

Device type:    CPU

Device vendor:  Intel(R) Corporation

Device driver:  7.5.0.2

Device speed:   1600 MHZ

Device cores:   8 CU

Device score:   521

Selected platform: AMD Accelerated Parallel Processing

Selected device: Hainan

with OpenCL 2.0 capability.

Loaded existing SGEMM tuning for batch_size 256.

Wavefront/Warp size: 64

Max workgroup size: 256

Max workgroup dimensions: 256 256 256

readyok

ucinewgame

position startpos

go movetime 120000

info depth 1 seldepth 2 time 462 nodes 7 score cp 17 hashfull 0 nps 15 tbhits 0 pv d2d4 g8f6

...

info depth 7 seldepth 23 time 115342 nodes 13456 score cp 24 hashfull 52 nps 116 tbhits 0 pv d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 c4d5 e6d5 b1c3 f8e7 d1c2 c7c5 c1f4 c5c4 e2e4 b8c6

bestmove d2d4 ponder g8f6

Regards

Cordella

[Jupiter]

I will use the following:

nps 116

opencl type

Selected platform: AMD Accelerated Parallel Processing

Selected device: Hainan

Intel(R) Core(TM) i5-8250U

BTW AMD GPU Hainan is I think Radeon HD 8800M series could you give the exact series number? If this is difficult for you to find, I will just use series number 8800M

Thanks.

[Cordella]

My laptop's graphic card is AMD Radeon 520.

RAM is 8GB.

Using Arena, Leela Zero has never won to all uci chess engines, even the weakest. Mostly forfeited on time.

Do you know how to optimize leela strength using this poor hardware?

Thanks

[Francesco Tommaso]

Try to put her to play with long time increments (15 seconds or more) or, as I prefer, with fixed time per move, higher than 30 seconds.

[Jupiter]

Lc0 works best with nvidia gpu using cuda exe. Try to download the blas exe type, run same bench and compare the result with opencl.

I am not sure if Lc0 can be optimized using only opencl and blas exe.

Time forfeit can be avoided, by modyfing its options, this also depends on the time control you use.

Try to test blas exe too in arena perhaps this would not forfeit on time.

[Jupiter]

Site updated.

https://lc0bench.netlify.com/

On Sunday, September 9, 2018 at 9:20:30 AM UTC+8, Cordella wrote:

[Cordella]

Yes. I tried 40 second per move and Leela won some games. No forfeit time anymore.

Thanks.

[Cordella]

With blas exe, it seems Leela run only with CPU. Still forfeit on time.

Here is the benchmark test result:

uciok

isready

Found pb network file: E:\Downloads\Softwares\arena_3.5.1\Engines\Leela-017Blas/weights_11198.txt.gz

Creating backend [blas]...

BLAS, maximum batch size set to 256.

BLAS vendor: OpenBlas.

OpenBlas [DYNAMIC_ARCH NO_AFFINITY Prescott].

OpenBlas found 8 Prescott core(s).

OpenBLAS using 1 core(s) for this backend.

BLAS max batch size is 256.

readyok

ucinewgame

position startpos

go movetime 120000

info depth 1 seldepth 2 time 1519 nodes 7 score cp 17 hashfull 0 nps 4 tbhits 0 pv d2d4 g8f6

...

info depth 6 seldepth 18 time 120901 nodes 3176 score cp 23 hashfull 13 nps 26 tbhits 0 pv g1f3 d7d5 d2d4 g8f6 c1f4 c8f5 c2c4 e7e6 b1c3 f8b4 e2e3 e8g8 f1e2 f6e4 a1b1

bestmove g1f3 ponder d7d5

Regards

[JJJ]

Seems there is a problem with me :

network 11198, lc0 cuda

GTX 1060 6go , intel i5-3570 : 971 nps.

But if I run the program on arena instead for 2 min in starting position as well and same network, I have something around 3K nps. So I don't understand.

[Armin J.]

lc0 on my MacBook

a. GPU name

Intel(R) Iris(TM) Graphics 6100

b. GPU count or number of gpu you used

1

c. CPU name

Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz

d. Threads used (this is 2 by default)

2

e. Username that you like to use

arminhhj

f. lc0 exe type (cuda, opencl or blas)

opencl

g. Other info that you think is important

MacBook Pro (Retina 13 Zoll, Anfang 2015)

= >   info depth 6 seldepth 21 time 113960 nodes 8471 score cp 26 hashfull 34 nps 74 

Am Montag, 3. September 2018 11:52:19 UTC+2 schrieb Jupiter:

Trying to compile yet another benchmark. Generally interested to get the nps of v0.17.0 using network id 11198, using start position with go movetime 120000 command.

I am aware of the other benchmark done in google sheet, but I like to have a cleaner table.

Procedure:

1. Setup lc0 v0.17.0 to use default settings

You may use/revise lc0.config and define the network id.

Just use 2 threads first, you may run other settings if you still have time and post results here.

Example entry of your lc0.config file

weights=C:\chess\engines\lczero\v0.17.0\weights_11198.txt.gz

2. Use network id 11198

Lets use this id first to establish a baseline. Other id's maybe added if you are still interested to post your data.

3. Run or double click the lc0.exe

4. In console type 

uci

isready

(wait for the engine to be ready)

ucinewgame

position startpos

go movetime 120000

[Jupiter]

Be sure to send

go movetime 120000

you could have missed this.

that is 120,000

Could you post your console output?

[Jupiter]

Thanks, data added.

https://lc0bench.netlify.com/

[Jupiter]

Beta version of LB.exe is released.

It will bench Lc0.exe on 4 mid positions, and get the average nps. It will also save OS, CPU and GPU (windows only)

This will be used later in the next bench that I plan to record. Number of positions can be increased later.

LB.exe will run Lc0 with the go infinite command and then send stop command when movetime is reached.

Guide:

1. Be able to setup your Lc0.exe first, that includes setting the network id to use and test that it will run.

2. Copy LB.exe and gen-bench.bat in the Lc0 dir  (for simplicity).

3. When using batch file, modify gen-bench.bat depending on movetime and threads you want to use.

4. You can then run the gen-bench.bat

Usage:

a) In console to run the bench at 2 minutes/position 

lb.exe --movetime 120000 --threads 2 --engine Lc0.exe

It will output result-bench.txt with info on OS, mean nps and others. This is in append mode, which means the next time you run another, the old data will not be deleted.

b) To enable logging to see the search info

lb.exe --movetime 120000 --threads 2 --engine Lc0.exe --log

It will output log-bench.txt. This is overwrite mode. It will delete the old logs from old runs.

c) To run at 10 seconds and 4 threads

lb.exe --movetime 10000 --threads 4 --engine Lc0.exe

d) Use short option name

lb.exe -m 10000 -t 4 -e Lc0.exe

e)When Lc0 is in a different dir, you can use path and exe, enclosed it with double quotes

lb.exe --movetime 120000 --threads 2 --engine "c:\chess\NN\Lc0.exe"

d) To see the help

lb.exe -h

usage: lb.exe [-h] [-e ENGINE] [-t THREADS] [-m MOVETIME] [-l] [-v]

optional arguments:

  -h, --help            show this help message and exit

  -e ENGINE, --engine ENGINE

                        Indicate engine filename, default is Lc0.exe but

                        lb.exe should be in the same dir with Lc0.

  -t THREADS, --threads THREADS

                        Indicate number of threads, default is 2.

  -m MOVETIME, --movetime MOVETIME

                        Indicate time in ms, default is 10000 ms.

  -l, --log             Enable logging

  -v, --version         show program's version number and exit

So far this is only tested on windows. 

Notes:

a. If you set lco.config to use 2 threads, you can override it using lb.exe with

--threads 4

that is if you want 4 threads.

There are 3 files in LB.zip

lb.exe

help.txt

gen-bench.bat

and it will output 2 files

result-bench.txt

log-bench.txt

The info in result-bench.txt is what I need.

Download file at

https://drive.google.com/file/d/1qTF0HVjDh4Qa6dNqECTFaW_hYcc7YoRd/view?usp=sharing

Unzip it and copy it to the dir of Lc0.exe or any other dir.

You may report if you encounter an issue especially for windows OS.

Maybe it will work in Linux, but not sure if the exe will run on macOS.

[fadhel majoul]

 

NPS  1748

a. GPU name

ASUS VA327 on NVIDIA GeForce GTX 1050 4095 MB

b. GPU count or number of gpu you used

1

c. CPU name

Intel Core i7 7700HQ @ 2.80GHz

d. Threads used (this is 2 by default)

1. GPU Overclock [Yes, No] NO

e. Username that you like to use

fadhel

f. lc0 exe type (cuda, opencl or blas)

cuda

g. Other info that you think is important

16 GB RAM

[fadhel majoul]

Bench date   : 2018-09-15T08:50:47+04:00

OS           : Windows

CPU          : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz

GPU          : ['NVIDIA GeForce GTX 1050', 'Intel(R) HD Graphics 630']

num threads  : 1

num pos      : 4

time(s)/pos  : 5

nps          : [1313, 1384, 1397, 1286]

mean nps     : 1345

Bench date   : 2018-09-15T08:51:58+04:00

OS           : Windows

CPU          : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz

GPU          : ['NVIDIA GeForce GTX 1050', 'Intel(R) HD Graphics 630']

num threads  : 2

num pos      : 4

time(s)/pos  : 10

nps          : [1276, 1485, 1461, 1249]

mean nps     : 1368

[Hans Albers]

NPS  251

a. GPU name

 Nvidia Quadro K2100M

b. GPU count or number of gpu you used

1

c. CPU name

Intel Core i7 4800MQ  2,70MHz

d. Threads used (this is 2 by default)

1. GPU Overclock [Yes, No] NO

e. Username that you like to use

PeterSal

f. lc0 exe type (cuda, opencl or blas)

cuda

g. Other info that you think is important

16 GB RAM

Am Montag, 3. September 2018 11:52:19 UTC+2 schrieb Jupiter:

Trying to compile yet another benchmark. Generally interested to get the nps of v0.17.0 using network id 11198, using start position with go movetime 120000 command.

I am aware of the other benchmark done in google sheet, but I like to have a cleaner table.

Procedure:

1. Setup lc0 v0.17.0 to use default settings

You may use/revise lc0.config and define the network id.

Just use 2 threads first, you may run other settings if you still have time and post results here.

Example entry of your lc0.config file

weights=C:\chess\engines\lczero\v0.17.0\weights_11198.txt.gz

2. Use network id 11198

Lets use this id first to establish a baseline. Other id's maybe added if you are still interested to post your data.

3. Run or double click the lc0.exe

4. In console type 

uci

isready

(wait for the engine to be ready)

ucinewgame

position startpos

go movetime 120000

5. Wait for around 2 minutes and remember/record the nps after the search.

6. I am interested on the following info and kindly post here.

a. GPU name

b. GPU count or number of gpu you used

c. CPU name

d. Threads used (this is 2 by default)

e. Username that you like to use

f. lc0 exe type (cuda, opencl or blas)

g. Other info that you think is important

[Jupiter]

Thanks data added.

https://lc0bench.netlify.com/

[Jupiter]

Thanks for the info. I will start collecting NPS for middle game positions by November., I need to add more middle game positions, this beta has only 4 positions. Also I will input all moves that leads to the position. Currently the beta only sends the fen position to Lc0.

[Jupiter]

Thanks data is added.

https://lc0bench.netlify.com/

[Bob H.]

Hi,

Brand new to this.  I've been following the chess.com tournament and I thought it would be fun to help make Leela better.

NPS: 3029

a. GPU name: GTX 970

b. GPU count or number of gpu you used: 1

c. CPU name: Intel i5-4690S

d. Threads used (this is 2 by default)

e. Username that you like to use

f. lc0 exe type (cuda, opencl or blas): cuda (w/cudnn)

[Bob H.]

A few more things...

Username: BobH

lc0 version: 0.17.0

Network: 11198

When I downloaded the network, it appeared to give the filename as the SHA rather than weights_11198.txt.gz but lc0 found it okay.

The filename I have is: e7132994bc5705c5a07770e476f30692e845210fae12e1c727432eafb3316050

I used all the defaults and have no config file so I assume 2 CPU threads were used.

[d flo]

a. GPU:                                                         name:NVIDIA GeForce GTX 1060 6GB

b. GPU count or number of gpu you used:   1

c. CPU name                                                AMD A10-7870K

d. Threads used (this is 2 by default)           2

e. Username that you like to use                  dj556u

f. lc0 exe type (cuda, opencl or blas)           cuda

On Monday, September 3, 2018 at 4:52:19 AM UTC-5, Jupiter wrote:

Trying to compile yet another benchmark. Generally interested to get the nps of v0.17.0 using network id 11198, using start position with go movetime 120000 command.

I am aware of the other benchmark done in google sheet, but I like to have a cleaner table.

Procedure:

1. Setup lc0 v0.17.0 to use default settings

You may use/revise lc0.config and define the network id.

Just use 2 threads first, you may run other settings if you still have time and post results here.

Example entry of your lc0.config file

weights=C:\chess\engines\lczero\v0.17.0\weights_11198.txt.gz

2. Use network id 11198

Lets use this id first to establish a baseline. Other id's maybe added if you are still interested to post your data.

3. Run or double click the lc0.exe

4. In console type 

uci

isready

(wait for the engine to be ready)

ucinewgame

position startpos

go movetime 120000

5. Wait for around 2 minutes and remember/record the nps after the search.

6. I am interested on the following info and kindly post here.

a. GPU name

b. GPU count or number of gpu you used

c. CPU name

d. Threads used (this is 2 by default)

e. Username that you like to use

f. lc0 exe type (cuda, opencl or blas)

g. Other info that you think is important

[d flo]

info depth 7 seldepth 23 time 10588 nodes 33818 score cp 25 hashfull 126 nps 3193 tbhits 0 pv d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 c4d5 e6d5 b1c3 f8e7 d1c2 c7c5 c1f4 c5c4 e2e4 b8c6 c3d5

bestmove d2d4 ponder g8f6

[d flo]

this time I turned off other programs using my cpu, and ran 12000 instead of 12000

info depth 10 seldepth 30 time 80418 nodes 287543 score cp 21 hashfull 976 nps 3575

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/3705ccfa-f493-4ba6-9323-6b4e5151b54d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Bigler]

Looking forward to see results with 2080Ti which should be available tomorrow.

rgds

[Jupiter]

Thanks data added, yes  default is 2 threads or cpu.

New url for the startpos table.

https://lc0bench.netlify.com/startpos-benchmark.html

[Jupiter]

Thanks, I'll take this nps.

New url for the startpos table.

https://lc0bench.netlify.com/startpos-benchmark.html

[Jupiter]

Looks like this is delayed by a week.

https://www.theinquirer.net/inquirer/news/3062823/nvidia-rtx-2080-ti-delay

[David Bigler]

Well, the 2080 should be released as I write and should already provide a good idea of what to expect from the 2080 Ti coming next week.

[David Bigler]

The first bench I found here
https://www.lesnumeriques.com/carte-graphique/nvidia-geforce-rtx-2080-ti-p45557/test.html

Unfortunately nothing yet with LC0

[zz4032]

1) 3326 nps

2) GTX 1060 3GB

3) 1x GPU

4) Not overclocked

5) Intel i7-6800K

6) 2 Threads

7) Username: zz4032

8) cuda

9) OS: Linux Ubuntu 16.04

[Jupiter]

Thanks, data is added.

https://lc0bench.netlify.com/startpos-benchmark.html

Also released LB v1.0.beta2, see post #3

https://lc0bench.netlify.com/index.html

[tun nay]

It would be interesting, if we can add version 18 engine speed. I noticed a big speed jump in my card. ( ? about 20%)

[Jupiter]

The next benchmark that I plan to compile is for Lc0 v0.18.0, that engine is not released yet.

Bench will be different, will be using couple of middle game positions with move history. The current one is using starting position.

[zz4032]

1) 12010 nps

2) RTX 2080 Ti 11GB

3) 1x GPU

4) Not overclocked

5) Intel i7-6800K

6) 3 Threads

7) Username: zz4032

8) cuda

9) OS: Linux Ubuntu 16.04

Backend cudnn-fp16 is faster than cudnn, but doesn't run for me with v0.17.0 while it does with v0.19-dev:

error Your GPU doesn't support FP16

If I switch to current v.0.19-dev, there is a general boost due to improvements in LC0:

13051 nps (+8.7% to v0.17.0)

Using cudnn-fp16 I get:

16186 nps (+34.8% to v0.17.0)

[Jupiter]

Is the v0.19.0-dev close to be released?

I plan of recording another benchmark if there are contributors, but prefer the current version which is v0.18.1 with network id 20922. But this time will be run based on 6 middle game positions, then just take the average. 2 or more than 2 threads will be accepted along with fp 16 if there is.

Will release the benchmark program tomorrow.

[zz4032]

Not really. v.0.18 was just released.

It looks like there is a but in the Linux Nvidia driver, which prevents higher nps numbers with cudnn-fp16, so disregard my result with 16186 nps above. In Windows I get about 32000nps.

[Jupiter]

Lc0 bench is released, this tool is used to bench Lc0 on 5 different middle game positions.

Download the pack at https://lc0bench.netlify.com/ see Downloads menu.

You can post your result and I will compile them.

Readme:

Lc0 bench

A command line tool to generate bench results in nps by

running Lc0 on 5 middle game positions. Move sequences

are sent to Lc0, with commands go infinite and stop after

2 minutes. The average nps on 5 positions are then saved to a file.

The 5 positions have different material configurations.

Pos1: All pieces (non-pawn) are still present but some were already developed

Pos2: 1 minor for white and 1 minor piece for black are exchanged

POs3: 2 minors for white and 2 minors pieces for black are exchanged

Pos4: 3 minors for white and 3 minors pieces for black are exchanged

Pos5: 1 queen and 1 minor for white and 1 queen and 1 minor pieces

for black are exchanged

The pack includes:

BT.exe

bench.fen (the final positions from move sequences)

bench.pgn (the 5 games where the positions are taken)

help.txt

readme.txt

gen-bench.bat

Guide:

1. Be able to setup your Lc0 v0.18.1 first, that includes setting the network

id to use and test it if it will run in console. Get Lc0 at

https://github.com/LeelaChessZero/lc0/releases

2. Use network id 20922, get it at http://lczero.org/networks/

3. Unzip the pack and copy files into the Lc0 directory

4. gen-bench.bat file is set to run at 2 minutes and 2 threads. For this

bench that I am gathering, we shall use 2 minutes search time.

5. You can modify the threads if you want. If you run 2 threads and

another 4 threads, you can submit those 2 benchmarks.

6. For other engine options, you can use the Lc0.config file.

7. Open the result-bench.txt, fill up other info and post it in

Lc0 google forum. This is the data that I need to compile.

Compiled benchmarks will be posted at https://lc0bench.netlify.com/

You can also see here the benchmarks from start positions.

Usage:

a) In console to run the bench at 2 minutes/position 

lb.exe --movetime 120000 --threads 2 --engine Lc0.exe

It will output result-bench.txt with info on OS, mean nps and others. 

This is in append mode, which means the next time you run another, the old

data will not be deleted.

b) To enable logging to see the engine search info

lb.exe --movetime 120000 --threads 2 --engine Lc0.exe --log

It will output log-bench.txt. This is overwrite mode. It will delete

the old logs from old bench runs.

c) To run at 10 seconds and 4 threads

lb.exe --movetime 10000 --threads 4 --engine Lc0.exe

d) Use short option name

lb.exe -m 10000 -t 4 -e Lc0.exe

e) When Lc0 is in a different dir, you can use path and exe, enclosed it

with double quotes

lb.exe --movetime 120000 --threads 2 --engine "c:\chess\NN\Lc0.exe"

d) To see the help

lb.exe -h

usage: lb.exe [-h] [-e ENGINE] [-t THREADS] [-m MOVETIME] [-l] [-v]

optional arguments:

  -h, --help            show this help message and exit

  -e ENGINE, --engine ENGINE

                        Indicate engine filename, default is Lc0.exe but

                        lb.exe should be in the same dir with Lc0.

  -t THREADS, --threads THREADS

                        Indicate number of threads, default is 2.

  -m MOVETIME, --movetime MOVETIME

                        Indicate time in ms, default is 120000 ms.

  -l, --log             Enable logging

  -v, --version         show program's version number and exit

[Jon Mike]

That's great, I will try this out.  Can I ask why only 5 positions?  Seems like a small sample.  :)

[Jupiter]

I don't like running a bench that takes too much time. Total bench time is 2min x 5 = 10 minutes. Also the 5 positions have covered a resonable material distributions already, that even if I increase the number of bench positions the average nps won't change that much.

[Andreas Strangmüller]

a. NVidia GeForce GTX 1070
b. 1 
c. Intel Xeon E5-2680v2
d. 2
e. fastgm
f. Lc0 v0.17.0 cuda

5057 nps

Andreas

[apacho321]

a. NVidia GeForce GTX 650
b. 1 
c. Intel Core i5-3570
d. 2
e. cs1g
f. Lc0 v0.19.0 cuda

321 nps

[elliptic]

a. AMD Radeon RX 480

b. 1

c. AMD Ryzen 7 1700X

d. default

e. elliptic

f. opencl

g. 608 nps

[Cor Hur]

a. GTX 1080 & 1070

b. 2

c. I7 8700

d. 2

e. ch

f. cuda

g. nps 13,150

[Jupiter]

Will update this later thanks. 

[Jupiter]

Currently we are using Lc0 v0.17.0 for startpos benchmark. If you could run Lc0 v0.17.0, and use network id 11198, I will include your data in the startpos benchmark.

Thanks.

[Jupiter]

Thanks will include your data in the starpos bench later.

[Jupiter]

Thanks will include your data later.

[Jupiter]

To all enthusiasts who want to submit the Lc0 benchmark of their unit for middle game positions, hold it. I will be introducing a different benchmark for lc0 v0.19.0

For startpos benchmark you can still submit, but use default settings using Lc0 v0.17.0 and net id 11198. See guide here.

https://lc0bench.netlify.com/bench-guide.html

[vaslo...@gmail.com]

a. 1080ti

b. 1

c. AMD Ryzen 5 1500x

d. 2

e. Vasloh

f. backend=cudnn, lc0 version 0.21.0

nps         : [7161, 10245, 11966, 9792, 10135]

average nps : 9860