CCRL

[Sam Jukes]

Leela Zero is now on CCRL:

http://www.computerchess.org.uk/ccrl/404/cgi/compare_engines.cgi?family=Leela%20Chess&print=Rating+list&print=Results+table&print=LOS+table&print=Ponder+hit+table&print=Eval+difference+table&print=Comopp+gamenum+table&print=Overlap+table&print=Score+with+common+opponents

However even at ID 187 she is only about 2200s where most agree at this time she was about 2700. Is this because CCRL are only letting her use CPU? I mean technically that would be fair as all other engines are using CPU and not GPU on their website but LZ is not very good with CPU.

Do you think they should make an exception with LZ as she is a neural network engine and allow her to use GPU? After all pretty much every game you see her play with strong engines is her using GPU so that's what we're interested in. The 2200s rating on CCRL is pretty meaningless otherwise.

[PawnBecomeQueen]

For now, i think Graham won't do exception for Leela by accepting GPU use. But if Leela reaches a high level like in the first 10 in CCRL rankings then she will be unavoidable.

And reasonably a chesslover as Graham who likes organizing matches between engines won't be able to ignore her for very long.

[Tadeusz R]

If they run modern Intels then Lc0 will still use IGPs, so there is some hope...
(-:

[Albert Silver]

He's not ignoring Leela. The problem is having no GPU worth mentioning. I spoke with him.

[Hartmut Hering]

I don't think there will ever be tested a GPU-Version of Leela at CCRL. They have different testers and very strict test conditions.

Time control
Our time control is equivalent to 40 moves in 40 minutes on AMD X2 4600+ at 2.4GHz.
We use Crafty 19.17 BH as a benchmark to determine the equivalent time control for particular machine.

How to test: Reboot a machine (not required but preferrable), extract Crafty executable into a separate folder, make sure there are no other files in that folder, run crafty, type 'bench' <enter>, wait a while (don't use computer in that time), when the benchmark ends type 'quit' <enter> (to quit the Crafty). You will then see a new file "log.001" in the Crafty folder. Open that file and find a line "Total elapsed time: 96" near the end. (Your time may be different of course).

Time control for your machine is then computed based on Crafty 19.17 BH benchmark result as following:

CCRL 40/40: T minutes / 40 moves repeated, where T = 40 * <your elapsed seconds> / 48 = <your elapsed seconds> / 1.2
CCRL 40/12: T minutes / 40 moves repeated, where T = 40 * <your elapsed seconds> / 160 = <your elapsed seconds> / 4
CCRL 40/4: T minutes / 40 moves repeated, where T = 40 * <your elapsed seconds> / 480 = <your elapsed seconds> / 12

Example: Your machine runs Crafty 19.17 BH benchmark in 55 seconds and you want to run long time control games (CCRL 40/40). Compute T as 40 * 55 / 48 = 45.833333, rounding to the nearest integer we get T = 46. So, your time control for CCRL 40/40 is 40 moves in 46 minutes.

They would never have a chance to convert such a test to the GPU-Version. Therefore I have doubts that the GPU-Version will ever be tested at CCRL. It would not be compareable.

Only solution. We have to train Leela until it is able to beat Stockfish even with the CPU :-)

[Gian-Carlo Pascutto]

On 03-05-18 17:49, Hartmut Hering wrote:
> How to test: Reboot a machine (not required but preferrable), extract
> Crafty executable into a separate folder, make sure there are no other

...

> Only solution. We have to train Leela until it is able to beat Stockfish
> even with the CPU :-)

Another problem is that their benchmark is completely worthless for
Leela Zero performance. It would give very similar results between (for
example) an Ivy Bridge and a Haswell GPU.

Yet for Leela Zero, the latter should be significantly faster (twice the
floating point throughput).

Anyway, many people do have a GPU, even if not a brilliant one. So CPU
only rating lists that do not adapt will become relics because they
won't have the information people look for.

--
GCP

[Chessie K]

Hi Gian-Carlo,

at first I want to thank you very much for the most interesting project in computerchess I have seen. LCZero ist the first chess engine running in a GPU. Maybe more will come out. Would be very interesting to see how Deep Sjeng performs on a GPU :-)))

We have the same problems in the CEGT group how to compare such engines with the already tested ones and which standard for GPU we can use. I am sure we have to discuss an adaption - but it will not be easy. My only GPU is an Intel HD Graphics 730 which is only twice as fast as 1CPU and i know this is not enough.

Werner

[123]

Sam Jukes:

I have done exactly the same thing like CCRL.

And to be as fair as CCRL, I decided to let all engines play on gpu:)

After 3000 games I'm happy to tell you LC0 won all games.

Now feel free to answer the question: How much elo LC0 have now(hint...too much) and how much elo the opponents have(hint...<0 elo).

[Gian-Carlo Pascutto]

On 03-05-18 18:37, Chessie K wrote:
> Hi Gian-Carlo,
> at first I want to thank you very much for the most interesting project
> in computerchess I have seen. LCZero ist the first chess engine running
> in a GPU. Maybe more will come out. Would be very interesting to see how
> Deep Sjeng performs on a GPU :-)))
> We have the same problems in the CEGT group how to compare such engines
> with the already tested ones and which standard for GPU we can use. I am
> sure we have to discuss an adaption - but it will not be easy. My only
> GPU is an Intel HD Graphics 730 which is only twice as fast as 1CPU and
> i know this is not enough.
> Werner

Hi Werner,

if the NN engines include a benchmark, comparing between GPUs or CPUs
for them should be feasible. Although the people who don't have a GPU
and an older CPU will probably not want to run matches with NN engines,
as the time compensation could get very big.

As for aligning between CPU and GPU engines, I would put the comparison
on an equally costly CPU and GPU. This seems fair.

I know we implemented a benchmark in Leela Zero, but I'm not sure Leela
Zero Chess has one. If not, it should be easy to add.

--
GCP