On costs of Leela Chess training & Deep Learning economics

[Alexey Eromenko]

Hello,

Just want to share an important idea about Deep Learning economics.
I'm participating in Leela Chess AI model training as a sponsor. Here are quick numbers:
A single RTX 2080 Ti can generate about 14 games/min or 20,000 games/day (for Leela model 40). This equal 600k games/month. 

For $150/month at Vast.ai current-going prices.
A strong Leela takes about 60 million games of training (or more). (Let's say to match SF8 strength )

About the 60 million games number required to train:
That is empirical evidence based on Leela model 30. Maybe a better AI model can train faster than 60 million games.

This means that a farm of 100x RTX 2080 Ti can train a Leela in 1 month time for $15,000 dollars.
A hypothetical 1000x RTX 2080 Ti farm can complete the job sooner, in 3 days, but will cost the same $15,000.
A hypothetical 5000x RTX 2080 Ti farm can complete the job sooner, in 15 hours, but will cost the same $15,000.

Google, with it's 5000 TPUs were able to train Alpha Zero in only 9
hours to beat Chess.

Basically time can be reduced: the next mission is to reduce costs of
training. That's up to AI scientists, I guess.
--
I hope it is informative for you.
-- 
-Alexey Eromenko "Technologov". 2019-03-03

[Amberlain]

Thanks for the numbers and again for your contribution.

I hope that in the years to come, the prices can be reduced enough that community efforts such as Lc0 get to try and tackle more complex challenges than perfect information games. Seeing Leela progress is very entertaining already and I can only imagine what will be achievable ten or twenty years form now by a few ressourceful people.

[David Bigler]

Agreed!

[MasterMark]

This is my first post here... I have an RTX 2080 that I plan to dedicate to generate training games for LCZero for the next 3 months 24/7. In the first 24h, it generated about 13.000 games. Does that sound right? Or should the output be higher?

Thanks

Mark

On Sunday, March 3, 2019 at 5:49:36 AM UTC+1, Alexey Eromenko wrote:

[Alexey Eromenko]

RTX 2070 does about 12000 games/day.
RTX 2080 TI does about 20000 games/day.
RTX 2080 should do around 14500 games/day

[MasterMark]

Thanks for you answer. Is there anything I can do to optimise the performance?

[Alexey Eromenko]

most important thing is to have fast bus (like PCI-E x16 pipe... 8x is okay, but 4x or 1x will slow things to a crawl).

ensure that GPU clocks and power is okay. (MSI afterburner tool on Windows or Linux command nvidia-smi)

[MasterMark]

Thanks! Bus is PCIe 3 x16. Used MSI afterburner to optimize as you suggested and should now get to 14500 games/day.

Processor is a i3-4160. Would an upgrade to a i7-47xx lead to higher performance? Or is the processor not limiting the performance?

[Alexey Eromenko]

Processor is very much limiting with test50 (small 10x128 Neural Networks are CPU bound, large 20x256 networks are GPU bound), but I haven't tried on an i3, so I dunno.

IMO, For 1 GPU, even RTX 2080, core i3 should be enough with test40. (Assuming no other programs running, especially multi tab chrome eats a lot of CPU).

[Dave Champagne]

Very close estimate - I have an OC'd 2080, and I'm getting around 16500-17000 games a day and the only other thing I use that PC for is Netflix on occasion. 

[Dave Champagne]

Oh, and meant to mention I'm running an i7 4790k @ 4.9ghz with 16gb of ram. 

[MasterMark]

the RTX2080 has now produced about 300k games...

since I have the syzygy tablebases from chessbase (all 5-piece and 27 6-piece endgames) I wondered if my PC can do anything more for the Lc0 project

I do not use that PC and the RTX2080 for anything else, so I am open to suggestions...