What's the strongest NN overall for now?

[Antony]

Hi all,

Please keep this post active and continue posting the best and strongest NN overall.

Thanks!

[Mathz]

J13B.2-188 playing at CCCC
or
T2 Playing at TCEC
?

[Jon Mike]

3-160 and 3-180

[Mahmoud Nabil Mohammad]

thanks a lot for sharing 

[Mathz]

IMO, for chess pros for chess analysis...

Opening Study use Latest T60 Net
End Game Study use Latest SF Dev
Middle Game Study use Both

Try and enjoy!

[Anil Dharan]

Sergei's 384x30 NNs of course. lco and jhorthos NNs don't even come close. Of course, you'll need strong GPUs, nothing less than a Geforce 2070, to run it effectively. 1060, 1080s plzz stay away and play with lc0s kiddy nets like 128x10.

[Mathz]

Thanks a lot for sharing. Where shall we get Sergei's 384x30 NN weights file?

[Capri Corn]

I believe you can find the NN files here: https://www.comp.nus.edu.sg/~sergio-v/t40/384x30/

[Francesco Tommaso]

Never heard of it. Is it the "Terminator" net? I will test it against the last T40 network. I have a RTX 2080ti. 

[Anil Dharan]

Even a single 2080 Ti may not be enough to fully utilize its potential. I run two ASUS 2080 Tis and it runs really HOT ! GPU-Z shows loads going over 90 %. But if I use this NN on the PlayChess Server, I'm almost impossible to beat, even by 44 Core Xeon users !

[Mathz]

Thanks, Capri Corn, I also found Sergio's T60 files along with T40 files.

[Jon Mike]

Anil, 

While Sergio's net is largest, it is far from strongest or best.  No tests have shown it stronger than any top networks (as seen below where it places last against less than best nets)

Although the crosstable below is at 1kn per move, 384x30 hasn't had a winning performance by any tester (If it had, we would have sent it to CCC or TCEC)

# PLAYER : RATING ERROR POINTS PLAYED (%) CFS(%) W D L 1 lc0.net.41800 : 0 ---- 2585.0 4500 57.4 96 1427 2316 757 2 lc0.net.60927 : -18 21 237.0 500 47.4 83 105 264 131 + 3 lc0.net.60940 : -33 21 227.0 500 45.4 50 91 272 137 4 lc0.net.60910 : -33 21 227.0 500 45.4 66 92 270 138 5 lc0.net.60900 : -39 22 222.5 500 44.5 86 90 265 145 6 lc0.net.60853 : -56 22 211.0 500 42.2 56 82 258 160 7 lc0.net.60876 : -58 21 209.5 500 41.9 56 80 259 161 8 lc0.net.60888 : -60 21 208.0 500 41.6 52 88 240 172 9 lc0.net.60865 : -61 20 207.5 500 41.5 100 79 257 164 10 lc0.net.384x30-1024 : -124 22 165.5 500 33.1 --- 50 231 219

This is all well known, so if you are looking for a strong network, just go with one featured at TCEC or CCC, those nets are the best. ;)

[pjw]

On  a RTX 2060 I get:

 

Ultrabullet 10+0.5

Rank Name                          Elo     +/-   Games   Score   Draws

   1 lc0 0.22 T40B.4-160           127      85      20   67.5%   65.0%

   2 lc0 0.22 J13B.3-120            53      75      20   57.5%   75.0%

   3 lc0 0.22 33000                -35      84      20   45.0%   70.0%

   4 lc0 0.22 36030                -53      75      20   42.5%   75.0%

   5 lc0 0.22 57399                -89      71      20   37.5%   75.0%

Bullet 60+1

   # PLAYER            :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)

   1 Lc0 T40B.4-160    :     107     65    19.0      28  67.9      99

   2 Lc0 J13B.3-120    :     -14     55    14.0      29  48.3      56

   3 Lc0 36030         :     -21     60    13.0      28  46.4      50

   4 Lc0 57399         :     -22     58    13.0      28  46.4      73

   5 Lc0 33000         :     -50     57    12.0      29  41.4     ---

Blitz 180+2

   # PLAYER            :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)

   1 Lc0 T40B.4-160    :     101     61    21.5      32  67.2      97

   2 Lc0 57399         :       8     56    16.5      32  51.6      65

   3 Lc0 J13B.3-120    :      -9     58    15.5      32  48.4      72

   4 Lc0 36030         :     -36     57    14.0      32  43.8      72

   5 Lc0 33000         :     -64     59    12.5      32  39.1     ---

 

[Francesco Tommaso]

Seems like this net is only good for heating your home. Good to know.

[Mathz]

Your satire touched me, these nets are really heating my room, but not entire house. I am scared of next summer. Lol...

[Stefan Pohl]

I am testing NN nonstop. Look here:
https://www.sp-cc.de/lc0-testing.htm

[Anil Dharan]

Jon,

      Thank you for the information, most enlightening.

However, there is a crucial difference between Testing against different NNs and actually playing against the mortal enemy, Stockfish.

I'm a Gamer (not a Tester) on Playchess and I can tell you with ABSOLUTE confidence, that Sergio's NNs peform much better against Stockfish than ANY of the NNs you mentioned.

Perhaps this difference lies in another crucial Factor you may have ignored, the GPU.

I have not one, but TWO ASUS 2080 Tis running in my System. The other NNs you mention no doubt give much better Kn/s than Sergio's 384x30 NN, BUT this does NOT translate into better performance against Stockfish.

Somehow, the 2 2080 Tis running together utilize the large 384x30 NNs more efficiently and enable me to get Draws REGULARLY against even 22 Core and 32 Core players using the latest Stockfish/Brainfish. Using the other NNs, I was being regularly beaten even by 4-6 Core Players.

Your Theoretical Results seem flawless, but unless your Theory can explain my Practical Results, it remains incomplete and inconclusive.

Regards

[M MUSTERMANN]

Anil Dharan:

That's correct.

You need 2x RTX 2080 Ti GPUs for 30x384 NNs. 

[doan ferdiansyah]

Yes correct musyerman

[123]

Jon Mike:

This Test is useless because:

1. No information if 2x RTX 2080 Ti were used.

2. No information about time control.

3. Let's say 2 minute per move and ~100 moves per game = 200 minutes x 2 engines = 400 minutes per game x 500 games = 200000 minutes = it will take 3333 hours.

4. 400 minutes x 4500 games = 1800000 minutes = it will take 30000 hours.

I don't think they have tested it this way, but feel free to correct me if I'm wrong.

[123]

Every position which can't be solved, because net is to small, could be solved by a larger net. Which also has a limit, but this limit is much higher, than by a smaller net.

Or in other words: if the limit of hardware + best move is 1 minute with a small net. The limit of hardware + best move could be something like 10 minutes with a larger net. So you will find nothing new (after 1 minute) with a small net but you can find something new best move (after 1 minute up to 10 minutes) with a big net.

Best nets are 30x384 if we are talking about maximum elo, when best moves are important and time is not important https://www.comp.nus.edu.sg/~sergio-v/t40/384x30/

[Anil Dharan]

Yes, I think you are right. What you say seems to agree with my practical experiences on the Playchess Server, while using the 380x34 NNs with my twin RTX 2080 Ti System.

[Thomas Spark]

Who is Sergio?

How are his networks derived?

Where can we read more about the concept?

How can we contribute?

[Jon Mike]

@123 and @Anil - Are you saying that 384x30 plays better on Playchess Server compared to T40B.4-160 if 2 x 2080 TI's are used?

@Thomas - Just ping him on discord :)

[Legolas]

So many ridiculous claims here of Sergio's net being the strongest. Such ridiculous claims should be backed up by data with error small enough that we can definitely conclude that it is the strongest. Otherwise, such claims should not be taken seriously.

[Anil Dharan]

Hi Thomas

                You were the inspiration for me to go for 2x2080 Tis. No regrets,especially 'cos I use the superb 384x30 Sergio NNs, the links for which are given somewhere above.

I strongly recommend the Sergio NNs especially for Gamers like you.

Just ignore the nitpickers who are demanding Proof.

Try it out for real on Playchess and see the Practical results.

[Thomas Spark]

Hi Anil,

I am indeed satisfied with 2 x 2080TI OC and I usually play on ChessBase in the Engine Room under the name Spark9.

For the past 12 hours I am playing 60+15 games there with the latest Sergio's net and the results are already visible (see below).

I am gonna test the net for the next couple of days to see how it performs and report back.

[Stephen Timothy McHenry]

I don't get it, your screenshot is dated from back in February??

[Jon Mike]

@stephen,

Don't worry, this thread is a huge trollish effort giving the masses false info.  Sergio's nets are not best.  If they were we would have sent it to CCC with 3 2080 ti GPUs..  Here's data proving my statement (less than 24 hours old).

Match: 384x30-t40-1097 vs J13B.2-136) (net from , trained with T40 games) LC0 version: 0.22 LC0 options: cudnn-fp16, 1 thread Time control: Fixed nodes 1kn Hardware: RTX 2070 Book: Hert 250 book, in sequence, reversed color Tablebases: 6-man TB Adjudication: 6-man TB, -draw movenumber=50 movecount=5 score=8 -resign movecount=5 score=1000 Software:cutechess-cli Speed: 384x30-t40-1097: 5.6 Kn/s, J13B.2-136: 9.3 Kn/s (lc0 benchmark ... --smart-pruning-factor=0 --nodes=200000) Comments: This net 384x30-t40-1097 is trained with T40 games, and not with T60 games as 384x30-1024 that I tested here . So it is stronger, and competed with Terminator 2 (version J13B.2-136 here): +87 -117 =296 (-21 elo) : weaker in fixed nodes, so it should be weaker than T2 in time control in whatever kind of hardware because it is also slower.

threads=1, fixed 1k nodes per move vs J13B.2-136, hert 250 book:

   # PLAYER                     :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L
   1 lc0.net.J13B.2-136         :       0     10   265.0     500  53.0      98  117  296   87
+  2 lc0.net.384x30-t40-1097    :     -21     10   235.0     500  47.0     ---   87  296  117

White advantage = 39.57 +/- 9.71
Draw rate (equal opponents) = 60.42 % +/- 2.12

[Francesco Tommaso]

Yes. Either they are trolling or are completely delusional.

[Thomas Spark]

Stephen,

the screenshot is from November 2nd.

Jon,

Just to clarify, I am not claiming anything, not even defending anything, except the power of HW (2 x RTX 2080Ti).

My experience is that T40B.4-160 seems to be the strongest net yet.

I am opened to suggestions and Segio's net is a suggestion which I am trying to play with at the moment.

Do you think J13B.2-136 might be stronger than T40B.4-160?

[M MUSTERMANN]

Jon Mike:

Really Jon:

1.You don't know what you are talking about.

2.And by the way you proved the opposite of what you are telling us all the time. xD xD xD

3.Take a look at the data you have taken to prove your statement:

Time control: Fixed nodes 1kn Hardware: RTX 2070

The time control is in this case the biggest joke I have ever seen. Example: I have more or less 64 kn/s!!! That means when I would finish 1 game, your data would finish 64 games in the same time. Or in other words: super lightning bullet.

And Hardware is a joke too. Comparing 1 RTX 2070 to 2 RTX 2080 Ti

Your result is 53% to 47% = 6%. Or in other words: If Sergio's big nets get such a great result under these terrible conditions for a big net, then Sergio's nets are the absolute overkill with stronger Hardware and longer time control.

[Jon Mike]

@ M MUSTERMANN,

The main point is many testers with good HW (and much greater resources than 2x2080 TI say otherwise.  We chose to send 42850 and JH nets ahead of SV nets.  Why?  Because not one time control ever resulted in SV being stronger than top JH nets or even close to 42850.  Fast forward to today and there is the new SV net, hundreds of points stronger (trained off T40 instead of T60)  Still, it has only proven to be weaker.  So we have ALL evidence pointing contrary to your statements.  Do you have any tournaments that actually show SV to beat our top TCEC and CCC nets?  No.  

So either you are trolling or delusional, until you show your opinion is based on facts which can be shared on this post.  There are countless tests on #test-results in discord supporting my opinion as shared by all those in the "know".

I actually do know what I am talking about, that's why I thought I'd keep you from mis-guiding those who don't know any better yet.

(If you post evidence in .pgn format of a tournament of any time controls showing SV is better, everyone will reconsider)  but don't hold your breath people, because it just isn't gonna happen.  Wait a couple of months for SV nets to get stronger and sure, then it is possible.  For now T40B.4-160 is the most solid choice for most GPU users and J13 nets are best for 2x2080 TI users.

Hoped this help clear up the confusion.

(Go to discord->test-results for facts and data about which networks are best and don't feed the trolls)

:)

[Francesco Tommaso]

M MUSTERMAN,

No, you are wrong. That tradeoff of a bigger, denser network, is a better evaluation in order to compensate lower speed. I would agree that if nodes weren't fixed, all of your criticism would be valid. 

I still think you are trolling, since this is such a simple point that I am having difficulties trying to take it seriously. If that is the case, please stop.  

[Mathz]

https://github.com/LeelaChessZero/lc0/wiki/Best-Nets-for-Lc0