256x20 testserver

[pradip ingale]

Test server 256x20 can mush stronger than main server in future?

[Hasan ÇALIŞIR]

For 256x20 they started from scratch on test server.

It seems main server's 192x15 will never bootstrapped to 256x20.I didn't understand the reason.

I think if 256x20 makes good progress they will replace it with current main server's 192x15.

Short answer we hope it will be stronger.

Maybe they consider to bootstrap main server to 256x20.Because main server is now meaningless.

5 Tem 2018 Per 17:45 tarihinde pradip ingale <pradip...@gmail.com> şunu yazdı:

 Test server 256x20 can mush stronger than main server in future?

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/ea45fdbf-d974-432f-80dd-d6f879e9d07d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Curious]

I might be outdated on the theory here, but I've never heard about resizing a neural net and somehow use the old weights for something. New size = start from scratch.

The blog talks about moving main server to a larger net. It will happen, but some things must be ironed out first.

Whatever comes out of the current main server will go to TCEC 13, so the training is not useless. (As far as I have understood. Communication from devs are few and far between). Good TCEC score gives good PR.

[Hasan ÇALIŞIR]

It can be my fault.It is very hard to update yourself here.There is no regular information flow.I lose my mind usually in this group.

Anyway it can be my misunderstanding.

6 Tem 2018 Cum 00:55 tarihinde Curious <spam...@gmail.com> şunu yazdı:

I might be outdated on the theory here, but I've never heard about resizing a neural net and somehow use the old weights for something. New size = start from scratch.

The blog talks about moving main server to a larger net. It will happen, but some things must be ironed out first.

Whatever comes out of the current main server will go to TCEC 13, so the training is not useless. (As far as I have understood. Communication from devs are few and far between). Good TCEC score gives good PR.

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/1ae936f6-998d-48db-b3e8-e3ff4bce6f2b%40googlegroups.com.

[lamber...@gmail.com]

It is possible though. If you take the old network and simply 'add' a 64 neurons in each layer to make it 256 instead of 192, also initiate (with zero or, maybe better, epsilon-values) the cross-dependencies of the new neurons into the next and previous layer. This way, you are preserving the old scores from the smaller network while expanding it. Then, when you play more games, the new weights will also be adjusted. 

[ashinpan]

There is a technique called net2net. You can check this link: https://arxiv.org/abs/1511.05641

[Hasan ÇALIŞIR]

"During real-world workflows, one often trains very many different neural networks during the experimentation and design process. This is a wasteful process in which each new model is trained from scratch. Our Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network."

Because of that i said main server is meaningless if they stay at 192x15.They have already started 256x20 on test server from scratch.They lose time.

I understand they want to see how far she is able to go from 6000 on main server but in test server *9 series show that if hard fluctuation is started there is no another option to increase NN size.

After NN 452 there is no progress.

6 Tem 2018 Cum 15:53 tarihinde ashinpan <ashi...@gmail.com> şunu yazdı:

--

You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/748877d7-690e-4865-b99e-efbd446fb3b6%40googlegroups.com.

[Curious]

Looks like I am outdated.

[Scott Turner]

On Friday, July 6, 2018 at 8:53:02 AM UTC-4, ashinpan wrote:

There is a technique called net2net. You can check this link: https://arxiv.org/abs/1511.05641

The Leela developers are very conservative, so I doubt they'll consider it, but that's really very smart.

-- Scott 

[Robert Filter]

I also believe that increasing the net size is overdue. We are just burning money atm.

[Hasan ÇALIŞIR]

Sometimes I think 100 years old senile university professor owned the project,lives in slow motion.

6 Tem 2018 Cum 20:56 tarihinde Robert Filter <r.fi...@gmx.de> şunu yazdı:

I also believe that increasing the net size is overdue. We are just burning money atm.

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/27aa5e43-0900-43d4-b61e-fdc2a66ad7f1%40googlegroups.com.

[Curious]

The blog is quite clear that increasing the net size decreases the speed of training games generation so much that lc0 must be used. Lc0 is currently not ready for main server usage. They are working on it.

[Jesse Jordache]

You're missing an important clause here: "this is a wasteful process in which each new model is trained from scratch".  In other words, there's nothing wrong with messing around with multiple network sizes so long as you don't restart the whole process every time you update.  However, that's exactly what the devs do on the main server, since they have a beast of a machine that can take all the training games and input them until a new network size and be done in a day or so.  Then they drop in the new net and voila! we have an upgrade.  They're aware of net2net, but I think they're frightened off by the fact that if they get it wrong, the results might be real but too subtle to detect.

This is the beta.  At a certain point, when the devs are done experimenting over on the test server, we're going to start over from zero, so that there are no periods where the policy net goes haywire and every subsequent game is to a certain extent an over-reaction to it.  I think they're going to start at 256x20, since periods where Leela is limited by net size introduces oddities in her play that mess up the evaluation in the subsequent games.  Zero is like, a sine qua non of this project.  The games have got to be nothing but Leela vs Leela with nothing extraneous introduced, whether it's intentional or simply an oddity in code that produces a behavior that isn't a result of Leela's policy network.

[hsn...@gmail.com]

"They're aware of net2net, but I think they're frightened off by the fact that if they get it wrong, the results might be real but too subtle to detect."

If you find this logical topic closed for me Jesse.

Thanks god Linus Torvalds didn't frightened off by linux kernel.

8 Temmuz 2018 Pazar 16:25:45 UTC+3 tarihinde Jesse Jordache yazdı: