How to use Leela as benchmark ?

[Alexey Eromenko]

Leela engine seems to have benchmark option, but it's results are non-reproducible, every time giving me different results, which makes it useless as benchmark.

What do you think  ?

[LuckyDay]

really? i always get results usually within +/- 50 nps when i run it.

bear in mind that a very big factor is what weight file you run with it; by default it will select the newest weight file unless specified otherwise, so if you have been contributing games and it has been auto-downloading networks it might just use the latest one downloaded. Network weight file nps can vary by a few hundred nps depending on the weights; for example 32194 gets 2600 nps while 32377 gets around 2300 nps on my benchmark (and those nps are reproducible for me). test10 nets would get a different nps, and test35 would prob get 4x that nps number since it is a 10b net.

so yes, try specifying a specific weight file for the benchmark to use to improve reproducibility

[Alexey Eromenko]

also Leela benchmark runs forever, while I expect it to end in few seconds (up to 1 min max.)

[Alexey Eromenko]

Another problem: lack of Linux builds.

I would like to see some Linux builds. (like Stockfish does)

[Alexey Eromenko]

You say it's reliable benchmark? REALLY ???

Here is my result:

C:\Dropbox\Install-experimental\lc0-v0.20.0-rc2-windows-blas>lc0.exe  benchmark

       _

|   _ | |

|_ |_ |_| v0.20.0-rc2 built Dec 23 2018

Found pb network file: ./256x20-2018_1224_0740_29_942.pb

Creating backend [blas]...

BLAS, maximum batch size set to 256

BLAS vendor: OpenBlas.

OpenBlas [NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=128].

OpenBlas found 4 Haswell core(s).

OpenBLAS using 1 core(s) for this backend.

BLAS max batch size is 256.

Benchmark time 621ms, 5 nodes, 8 nps, move d2d4

Benchmark time 1584ms, 7 nodes, 4 nps, move c2c4

Benchmark time 2349ms, 16 nodes, 6 nps, move c2c4

Benchmark time 2742ms, 23 nodes, 8 nps, move c2c4

Benchmark time 3480ms, 26 nodes, 7 nps, move c2c4

Benchmark time 4483ms, 47 nodes, 10 nps, move c2c4

Benchmark time 4877ms, 67 nodes, 13 nps, move c2c4

Benchmark time 5906ms, 91 nodes, 15 nps, move c2c4

Benchmark time 6891ms, 137 nodes, 19 nps, move c2c4

Benchmark time 8202ms, 216 nodes, 26 nps, move d2d4

Benchmark time 8261ms, 230 nodes, 27 nps, move c2c4

Benchmark time 8349ms, 255 nodes, 30 nps, move d2d4

Benchmark time 9222ms, 297 nodes, 32 nps, move d2d4

Benchmark time 10201ms, 388 nodes, 38 nps, move d2d4

Benchmark time 11235ms, 468 nodes, 41 nps, move d2d4

Benchmark time 12245ms, 555 nodes, 45 nps, move d2d4

Benchmark time 12281ms, 565 nodes, 46 nps, move d2d4

Benchmark time 13130ms, 637 nodes, 48 nps, move d2d4

Benchmark time 14511ms, 740 nodes, 50 nps, move d2d4

Benchmark time 15697ms, 823 nodes, 52 nps, move d2d4

Benchmark time 16677ms, 946 nodes, 56 nps, move d2d4

^C

C:\Dropbox\Install-experimental\lc0-v0.20.0-rc2-windows-blas>./client_linux

....

let alone the fact that there is no final score and no auto-end.

[Vassilis]

Hi Alexey

It is reliable. As you can see, the test converges to a value (nps) that has small fluctuations.

Vas

[Alexander Lyashuk]

It runs benchmark for 30000 nodes. For so low nps, it will indeed 20 minutes to complete it.

You can pass --nodes=1000 to make it stop earlier. Possibly it makes sense to develop something to detect slow backends and finish benchmark earlier automatically.

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/b7879ee9-7554-4158-9644-f798125d3113%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vassilis]

Alexander!
Personally I don't mind at all. Ctrl+C is always there!

Cheers
Vas

[Vassilis]

Alexey!

You ask for Linux builds, but I see you use Windows!!!
He he :) Just kidding... Use whatever feels right for you!

[Alexey Eromenko]

I have lots of Linux and Windows PCs and Servers alike; I would like to bench all of them.

I was able to build Linux build for myself. 

[Alexey Eromenko]

 Engine : Leela 0.20.0 + Neural Network ID 32400 (20/256).

Linux Ubuntu 18.04 x64

~/leela-0.20/lc0 benchmark -t 12

core-i7-8700: Intel(R) Core(TM) i7-8700 CPU  @ 3.20GHz, Ubuntu 18.04 (Coffee Lake; 6 core; 12-thread)

thread- 1: Nodes/second :  69.767

thread- 4: Nodes/second :  222

thread- 6: Nodes/second :  276

thread-12: Nodes/second :  255  (slowdown due to Hyper-Threading ?)

Are you getting similar results ? Also slow down due to hyper-threading ?

[Alexey Eromenko]

Ideally the Leela benchmark should cover anything between Single-core Intel-Atom and up to Dual GeForce RTX 2080 with the same settings, so that benchmark gets reproducible between different people.

Perhaps use the same network ID file everywhere.

[Alexey Eromenko]

Another problem : AMD CPUs seem very slow.

I'm getting only 26 nps for AMD EPYC 7401P vs 69 nps for Intel Core i7 8700. That's a 3x times difference !

(test was run with 1 thread)

core-i7-8700: Intel(R) Core(TM) i7-8700 CPU  @ 3.20GHz, Ubuntu 18.04 (Coffee Lake; 6 core; 12-thread)

~/leela-0.20/lc0 benchmark -t 1

thread- 1: Nodes/second :  69.767

thread- 4: Nodes/second :  222

thread- 6: Nodes/second :  276

thread-12: Nodes/second :  255  (slowdown due to HT!)

epyc-01: AMD EPYC 7401P 24-Core Processor, Ubuntu 18.04  (24 core; 48-thread)

~/leela-0.20/lc0 benchmark -t 1

thread- 1: Nodes/second :  26?

thread- 4: Nodes/second :  100 nps (99.x)

thread-24: Nodes/second :  257.618

thread-48: Nodes/second :  167.922 (slowdown due to HT!)