Leela has no chance on anything below RTX cards series

[Ivan Ivec]

I made an experiment:

I put GTX 1060 Nvidia card in my old computer, CPU is Core 2 Quad Q9550 (from 2008).

I started a match between CorChess (Stockfish clone) and Leela (test 30).

CorChess was 80-100 Elo better although Leela was running on pretty new GPU.

I let the reader to make conclusions...

[Vassilis]

Ivan, I don't object, but actually I don't mind.
An RTX20xx is an affordable card to buy, so you can have the best of two world's (AB engine and NN) inside the same computer!
Imagine if you had to buy Deep Blue to have the same chess power ?

[Ingo Weidner]

Besides oother things this also depends on the time control. Since Lc0 net 32700 and now with both 32865 and 32890 i am able to gets wins of Lc0 against Stockfish 10 at a mobile GTX 1050 TI and a mobile i7-7700HQ quad-core CPU using a time control of 15min+3s. Also had some wins with a 5min TC already.

Currenty at 15min+5s time control Lc0 ID 32890 had some wins against latest Stockfish development version.

[Vassilis]

True

[Ingo Weidner]

Of course it is also important which Lc0 network you use. So far the best here are 32890, 32865, 32700 and AntiFish Mark 228.

[Dietrich Kappe]

If you restrict stockfish to 3, 2 or 1 cores, leela does quite nicely. You could also give leela more time. No ones watching. The engine police isn’t going to come and arrest you.

[Ivan Ivec]

I was first using 32864 with 3min+2s, and then I also tried 32930 with 15min+10s.

I also opened this topic because people do not understand that Leela ratio is not well defined.

Namely 20 Mnps on 1 thread would be much more effective than 20Mnps on let say 16 threads.

[Ingo Weidner]

At my system (mobile GTX 1050 TI + i7-7700HQ) using 2 CPU cores gives a good balance between the GPU and CPU while the Leela ratio is still below 1.0. In previous tests i mostly used 3 cores which gives a Leela ratio around 0.6. A time control of 5min is still usable that way but with less i would have to reduce the CPU to 1 core. Overall the best combination is 2 CPU cores and a TC around 15mins where Lc0 could even score wins against Stockfish development version..

[Ingo Weidner]

Update of my previous post:

With 2 cores of the mobile i7-7700HQ CPU i get those average search speeds when using Stockfish 10 and Lc0 CUDA version:

- GPU speed:  average speed = 2k - 3k nodes/s

- CPU speed (2 of 4 cores used):  average speed = 3400k nodes/s 

[Vassilis]

[...Namely 20 Mnps on 1 thread would be much more effective than 20Mnps on let say 16 threads]

Yes, correct! Multi-threading adds some overhead. It depends of course on how you define the word "effective" here.

In terms of chess strength in both cases it's the same [same Mnps = same number of positions searched, although I'm not so certain about this:) ], but in the later one you reserve more resources.

The truth is, that performance does not increase linearly with the number of cores.

What actually saves us, is that if we achieve 20 Mnps on 1 core, we could expect much more on 16 cores. (Note I use the word core and not thread)

If this worth all the extra programming effort, is another matter.

[Alexey Eromenko]

 1. Yes, it sucks because old hardware became obsolete with RTX series. (esp. laptops, where it's not upgrade-able)

2. RTX 2060 GPU is very affordable. (and not much weaker than 2070 or 2080 for Leela chess)

3. It is sad that there is no competition from AMD or other vendors (Google TPU?) for Leela hardware. NVIDIA becomes a monopoly.

[graci...@gmail.com]

RTX 2060 is still around 400 dollars tough and still 40% slower than the recommended 2080ti, so you'll probably need 2 of those to be competitive. While used CPUs are cheap, you can get a pair of 6 core cpus for 40 bucks so you have a 12 core 24 thread system for less than 300, or you can get a 24 core /48 thread for around 800 dollars. And stockfish elo isn't stagnant either, it is still being developed and they still manage to add around 5 elos a month. So cpus and traditional engines are still king budget wise, maybe leela from a year from now or the next gen GPUs this will all change but who knows...

[Pawel.NewYork]

You dont have to be cheap comparing 400usd 2060 rtx to 100usd processor, because it is not about a few hundred dollars , most of the ppl can afford paying more. It is about the quality of chess, it is about a progress, it is about new technology, this is what matters! 

[graci...@gmail.com]

Well not all people are rich and have 400 or 800 dollars to spare, also it can become a high tech paper weight if a new cheaper gpu comes along and it is significantly fast like those guys who bought 1080ti for leela only for the rtx series to be released in a few months. If someone is still looking for consistent elo per dollar, stockfish with used cpu is still the best. And more elo = more quality chess. You rich guys go buy your expensive gpu and tune lc0 and push it to the limit, I'll buy my own gpu when it matches sf's elo per dollar.

[Pawel.NewYork]

Of course that in the future newer cards will come out, that's the way things are. This does not mean that you have to wait infinitely for the current generation's price to fall. This is a personal approach, but taking into account the statistical prices of new cards at the turn of the years, $ 400 does not seem like a high price. We have to take into account the fact that the old card can always be re-sold. In addition, those who bought 1080ti can still sell them and for the same money buy 2080rtx which has even higher performance in games, and definitely higher performance of NN so those who bought them arent in the lost position. Also having only CPU  you will never truly join community since all the progress is based on GPU so you will always be standing aside waiting for price to drop that wont happen very soon. 

[brian.p.r...@gmail.com]

Please include addition information:

What was the Leela Ratio?

Specify the book and depth used.

Were tablebases used?

Was the tournament manager doing adjudications?

How many games were played and what were the Ordo (or Bayeslo) results, with error bars?

Thanks.

PS The result is far from unexpected, but really, within 100 Elo of the beast is pretty good for an engine that is just over a year old.

[Dietrich Kappe]

I guess this post could have been titled: "I'm sad I didn't get an RTX card."

[Sebastian Reinke]

This is actually untrue based on my own tests, and your sample size is likely very small.
I use a 1070 Ti against a 6700k, and Leela does reliably get wins against SF, despite not being significantly better than SF10, for example. Have you considered that you may have misconfigured LC0?

[Ivan Ivec]

Leela Ratio = 0.9

2moves_strong (208 positions)

5men tablebases

Arena adjudication (default)

around 100 games at 3min+2sec

NNCacheSize = 2M, all other at default

[brian.p.r...@gmail.com]

Thanks for the details.

Maybe T40 nets will do better.

Still, on the 1060, Leela is probably a 3,000+ Elo engine.

I have not done any tests, but with Leela's endgame weakness, I wonder if 6 men tablebases would make any difference.

[Ivan Ivec]

Leela has more problems with evaluation interpretation, then with endgames themselves.

I would prefer that +2.00 means victory in at least 95% of cases.

[brian.p.r...@gmail.com]

Not sure if this might be helpful or not, but Leela's scores are derived from the Q value.

You can specify what output format you wan; it is one of the UCI options:

option name ScoreType type combo default centipawn var centipawn var win_percentage var Q

Here is a code snippet:

	if (score_type == "centipawn") {
	uci_info.score = 290.680623072 * tan(1.548090806 * edge.GetQ(0));
	} else if (score_type == "win_percentage") {
	uci_info.score = edge.GetQ(0) * 5000 + 5000;
	} else if (score_type == "Q") {
	uci_info.score = edge.GetQ(0) * 10000;
	}

[Vassilis]

Well, this code snippet only affects how Leela's evaluation is presented to the user.
If we want the default to be centipawns, how easy would be to alter the scaling function to be more realistic?
This would not harm, her decisions.

[brian.p.r...@gmail.com]

IIRC it has been tweaked a few times already.

If you would like to come up with a better formula, it might be worth a try.

[Dietrich Kappe]

A formula adapted for converting centipawn to Q is (python):

(2/(1+math.exp(-0.004 * cp)) - 1)

It's a little less optimistic at the low end, and more optimistic at the high end.

[Vassilis]

There it is!
Just take the reverse of this formula (solve for cp) and see what values it gives.

[Jeff Wads]

How about giving us your average nps for Lc0?  That looks like a decent gpu.

[svoi s]

GTX 1060 Nvidia is very bad GPU, I have only about 2knps (3Gb)

четверг, 7 февраля 2019 г., 17:27:52 UTC+3 пользователь Jeff Wads написал:

[Ivan Ivec]

I have around 3knps on GTX 1060. The point is that my processor is from 2008 and costs around 26$ (+tax) at the moment.