How to use Google Cloud or Google Colab for running tournaments

[Jon Mike]

It is my hope that the large blank area below will contain the answer to the title of this post.  

The goal is to utilize a remote gpu/tpu for engine comparison tournaments.  If you know how to do this, would you please consider sharing some examples or code? 

[LuckyDay]

A few people have asked about this, but nobody has really been able to give a satisfactory answer yet. I myself haven't tried it as all my compute time has been directed towards game generation.

There is a guide about testing new vs old nets using google colab on the github wiki; this might give you some pointers.  

https://github.com/LeelaChessZero/lc0/wiki/Script-for-testing-new-nets-versus-old-nets-on-Google-Colab

[Jon Mike]

 LuckyDay, I speak for many here: You are very appreciated and I couldn't thank you enough for what you have done so far for this project!

On Thursday, October 11, 2018 at 12:26:04 AM UTC-5, LuckyDay wrote:

A few people have asked about this...

[Jupiter]

I tried that script but was not successfull.

The error was:

Using 2 thread(s). Weights file is the wrong version. lczero: /content/leela-chess/src/Network.cpp:370: static void Network::initialize(): Assertion `m_format_version > 0' failed. Using 2 thread(s). Weights file is the wrong version. lczero: /content/leela-chess/src/Network.cpp:370: static void Network::initialize(): Assertion `m_format_version > 0' failed.

I changed to other network id's, and still there is error.

That script too is not using Lc0 but lczero.

Other steps tried on google colab

1. Download and compile the current Lc0 at https://github.com/LeelaChessZero/lc0. Then compiled Lc0 only supports blas and opencl. Tried to run a match using cutechess-cli but google colab exited.

2. Use the script in google cloud to compile Lc0 on google colab hoping that cuda will be available. Compilation was successful. Compile cutechess-cli too. But the test match would not continue because there is an error in cutechess-cli.

The error was:

./cutechess-cli: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory

Checking the cutechess dir, that file libQt5Core.so.5 is there, but did not see it. Copy that file to usr/bin dir or so and still cutechess-cli would not run.

I will try again item (1) when colab gpu is available.

[LuckyDay]

yes that is the problem with colab; it's very hard to test anything since actually getting access to a gpu is nigh impossible atm, maybe 1/2-1 hour daily at best if you're lucky.

Jon - no worries, only happy to try to provide assistance where I can (which is limited since I have very little technical/programming knowledge and atm am too busy to learn more)

[Jupiter]

I posted this in discord, noone answered. So we are on our own ATM.

The sample match script for google colab is a jupyter notebook. I am looking into the possibility of installing jupyter notebook in google cloud, then run the modified sample match script there.

At some point I will ask this in stack overflow when all my attempts will fail.

[brian]

I've tested this script on the 18.04 LTS images on google cloud (15gb). This should compile lc0, and cutechess-cli. 

lc0 will be in the lc0/build/release/lc0, 

and cutechess is in cutechess/project/cli/... 

You will need to download the weights for lc0 though. Hope this helps.

-Brian

[Jupiter]

Thank you, will try this out.

[Jupiter]

I added the following code to your script.

cd ~

echo "Download startpgn"

wget https://cdn.discordapp.com/attachments/433622350626422795/435942405913575435/2moves_LT_1000.pgn

echo "Download network files, 11198 and 21432"

wget http://lczero.org/get_network?sha=e7132994bc5705c5a07770e476f30692e845210fae12e1c727432eafb3316050

wget http://lczero.org/get_network?sha=bb0cebf71a54ff6453be0a01c240a20a8b0bca1e6b67231458b322644349a123

echo "Running Leela Chess Zero matches"

rm -f cutechess/output_pgn.pgn

cd cutechess/project/cli && QT_LOGGING_RULES="*.debug=true" ./cutechess-cli -engine name="11198" cmd=/content/lc0/build/release/lc0 arg="--weights=/content/e7132994bc5705c5a07770e476f30692e845210fae12e1c727432eafb3316050" nodes=800 stderr=lc01.err -engine name="21432" cmd=/content/lc0/build/release/lc0 arg="--weights=/content/bb0cebf71a54ff6453be0a01c240a20a8b0bca1e6b67231458b322644349a123"  nodes=800 stderr=lc02.err -each tc=inf proto=uci -recover -games 2 -rounds 2 -repeat -pgnout output_pgn.pgn -openings file="2moves_LT_1000.pgn" order=random

All went well except that it did not find cutechess

INFO startup-script: Running Leela Chess Zero matches

/startup-7qtxkd46/tmp6ovwtwq3: line 48: cd: cutechess/project/cli: No such file or directory

In google colab I can send commands and get results, like ls, cd, and others but in google cloud, I don't know how to query the contents of my cloud  instance, I need research on this.

On Thursday, October 11, 2018 at 5:30:39 PM UTC+8, brian wrote:

[brian]

Ooops my mistake the directory should be "cutechess/projects/cli" its "projects" not "project". It was really early in the morning when I wrote the post  =P. 

Also make sure the image is ubuntu 18.04 and that cutechess was probably installed. I should also mention that the script was taken from the contrib page on github with some modification for 18.04.

[Jupiter]

Even with projects it still could not see it.

Later I found out that cutechess is inside Lc0 because of this

...

...

cd lc0

CC=clang CXX=clang++ ./build.sh

sudo apt-get install -y qt5-default libqt5svg5*

sudo apt-get install 

git clone https://github.com/cutechess/cutechess.git

cd cute*

qmake && make -j 2

There was a cd to Lc0.

To solve this I revised the script to

echo "Installing lc0"

git clone -b release --recurse-submodules https://github.com/LeelaChessZero/lc0.git

cd lc0

CC=clang CXX=clang++ ./build.sh

echo "Install and compile cutechess"

cd ~

sudo apt-get install -y qt5-default libqt5svg5*

sudo apt-get install 

git clone https://github.com/cutechess/cutechess.git

cd cute*

qmake && make -j 2

With that,

cd cutechess/projects/cli

would now work.

But there is a problem now on running the engine

lc0/build/release/lc0

In cutechess command line I use this

cmd=lc0/build/release/lc0

Cutechess cannot find that path/file

This is now my cutechess command line

cd cutechess/projects/cli && QT_LOGGING_RULES="*.debug=true" ./cutechess-cli -engine name="11198" cmd=lc0/build/release/lc0 ...

This is the last part of Lc0 compilation.

...

...

Oct 11 17:35:21 instance-1 startup-script: INFO startup-script: 8 warnings generated.

Oct 11 17:35:23 instance-1 startup-script: INFO startup-script: [251/268] Compiling C++ object 'encoder_test@exe/src_neural_network_check.cc.o'.

Oct 11 17:35:24 instance-1 startup-script: INFO startup-script: [252/268] Compiling C++ object 'encoder_test@exe/src_mcts_search.cc.o'.

Oct 11 17:35:24 instance-1 startup-script: INFO startup-script: [253/268] Compiling C++ object 'encoder_test@exe/src_neural_network_random.cc.o'.

Oct 11 17:35:24 instance-1 startup-script: INFO startup-script: [254/268] Compiling C++ object 'encoder_test@exe/src_neural_network_mux.cc.o'.

Oct 11 17:35:25 instance-1 startup-script: INFO startup-script: [255/268] Compiling C++ object 'encoder_test@exe/src_neural_network_st_batch.cc.o'.

Oct 11 17:35:26 instance-1 startup-script: INFO startup-script: [256/268] Compiling C++ object 'encoder_test@exe/src_neural_writer.cc.o'.

Oct 11 17:35:28 instance-1 startup-script: INFO startup-script: [257/268] Compiling C++ object 'encoder_test@exe/src_selfplay_game.cc.o'.

Oct 11 17:35:29 instance-1 startup-script: INFO startup-script: [258/268] Compiling C++ object 'encoder_test@exe/src_selfplay_loop.cc.o'.

Oct 11 17:35:30 instance-1 startup-script: INFO startup-script: [259/268] Compiling C++ object 'encoder_test@exe/src_utils_histogram.cc.o'.

Oct 11 17:35:30 instance-1 startup-script: INFO startup-script: [260/268] Compiling C++ object 'encoder_test@exe/src_utils_configfile.cc.o'.

Oct 11 17:35:32 instance-1 startup-script: INFO startup-script: [261/268] Compiling C++ object 'encoder_test@exe/src_selfplay_tournament.cc.o'.

Oct 11 17:35:32 instance-1 startup-script: INFO startup-script: [262/268] Compiling C++ object 'encoder_test@exe/src_utils_optionsdict.cc.o'.

Oct 11 17:35:33 instance-1 startup-script: INFO startup-script: [263/268] Compiling C++ object 'encoder_test@exe/src_utils_random.cc.o'.

Oct 11 17:35:33 instance-1 startup-script: INFO startup-script: [264/268] Compiling C++ object 'encoder_test@exe/src_utils_optionsparser.cc.o'.

Oct 11 17:35:34 instance-1 startup-script: INFO startup-script: [265/268] Compiling C++ object 'encoder_test@exe/src_utils_transpose.cc.o'.

Oct 11 17:35:34 instance-1 startup-script: INFO startup-script: [266/268] Compiling C++ object 'encoder_test@exe/src_utils_string.cc.o'.

Oct 11 17:35:34 instance-1 startup-script: INFO startup-script: [267/268] Compiling C++ object 'encoder_test@exe/src_utils_filesystem.posix.cc.o'.

Oct 11 17:35:34 instance-1 startup-script: INFO startup-script: [268/268] Linking target encoder_test.

Does it really compile Lc0?

Yes I notice the script has similarities to the startup script used to generate training games on google cloud.

Do you know how to send a bash command to query the Instance to see where the files where located.

I tried the google cloud shell send ls command and nothing is there.

Tried the ssh window send ls command and there is nothing there also.

[brian]

echo "Install and compile cutechess"
cd ~
sudo apt-get install -y qt5-default libqt5svg5*
sudo apt-get install 
git clone https://github.com/cutechess/cutechess.git
cd cute*
qmake && make -j 2

 

Yes that should should the problem, sorry!

 

cd cutechess/projects/cli && QT_LOGGING_RULES="*.debug=true" ./cutechess-cli -engine name="11198" cmd=lc0/build/release/lc0 ...

 

for cmd=..  I think you need to specify the full path lc0 is not in the same directory. So I think it should be

cmd=/home/<yourusername>/lc0/build/release/lc0 ...

Oct 11 17:35:34 instance-1 startup-script: INFO startup-script: [268/268] Linking target encoder_test.

 

Yup, this looks like it finished compiling. The last few lines of output can be a bit confusing I guess.

Do you know how to send a bash command to query the Instance to see where the files where located.

 

Try this: find <directory_name> -type f -name "<filename_in_quotes>"

I tried the google cloud shell send ls command and nothing is there.
Tried the ssh window send ls command and there is nothing there also.

 

I'm using ssh, when you first connect the ls command returns nothing because the home folder is empty. After running the script, running ls should return some new items. 

 

I'm going to create an instance with both our edits to make sure there are no other issues. I'll follow up in a bit with more detailed steps (I realized my initial post had a lot of missing details).

[Jupiter]

Thanks will try you suggestion tomorrow.

I have some minor changes to the script, trying to put startpgn and network files on dirs. Not tested, don't know if the script is right.

echo "Install and compile cutechess"

cd ~

sudo apt-get install -y qt5-default libqt5svg5*

sudo apt-get install 

git clone https://github.com/cutechess/cutechess.git

cd cute*

qmake && make -j 2

cd ~

echo "Create folder for book"

mkdir -p book_file

echo "Download startpgn into book_file folder"

cd book_file

wget https://cdn.discordapp.com/attachments/433622350626422795/435942405913575435/2moves_LT_1000.pgn

cd ~

echo "Create folder for network id"

mkdir -p network_file

echo "Download network files, 11198 and 21432 into the network_file folder"

cd network_file

wget http://lczero.org/get_network?sha=e7132994bc5705c5a07770e476f30692e845210fae12e1c727432eafb3316050

wget http://lczero.org/get_network?sha=bb0cebf71a54ff6453be0a01c240a20a8b0bca1e6b67231458b322644349a123

cd ~

echo "Remove existing pgn putput file"

rm -f cutechess/output_pgn.pgn

Later I will try your suggestion on running the cutechess-cli with correct Lc0 path naming.

[Jon Mike]

Thanks so much for your work thus far, hopefully in a few days we can come upon a working solution to share with everyone in an easy to follow format.  Keep it up friends! 

[brian]

Okay, I updated the original script with you changes and others to get everything working (fingers crossed..). 

These are the steps I took:

1) Create instance (4cpu, 15gb disk image, 1 gpu) OS: 18.04 LTS

2) ssh into instance and copy contents of attached script to another file in the home directory

    for example I used the following:

       *  nano run.sh

       *  copy contents of attached file to run.sh

       *  ctrl-x y enter (close and save)

3) chmod +x run.sh

4) ./run.sh 

At this point everything should be installed and a small match be completed.

This is what I get for the last few lines of output for run.sh

Running Leela Chess Zero matches

Indexing opening suite...
Started game 1 of 4 (11198 vs 21432)
Finished game 1 (11198 vs 21432): 1-0 {White mates}
Score of 11198 vs 21432: 1 - 0 - 0  [1.000] 1
Started game 2 of 4 (21432 vs 11198)
Finished game 2 (21432 vs 11198): 0-1 {Black mates}
Score of 11198 vs 21432: 2 - 0 - 0  [1.000] 2
Started game 3 of 4 (11198 vs 21432)
Finished game 3 (11198 vs 21432): 1-0 {White mates}
Score of 11198 vs 21432: 3 - 0 - 0  [1.000] 3
Started game 4 of 4 (21432 vs 11198)
Finished game 4 (21432 vs 11198): 0-1 {Black mates}
Score of 11198 vs 21432: 4 - 0 - 0  [1.000] 4
Elo difference: inf +/- nan, LOS: 97.72 %
Finished match

In /home/$USER/bin/ there is a script called lc0_startup. This script basically calls lc0 from the build/release directory with additional args. 

You can modify these args to what you would like. This makes the cutechess-cli commands look cleaner imho. 

Hopefully no issues this time :)

On Friday, October 12, 2018 at 12:11:42 PM UTC-5, Jupiter wrote:

[Jupiter]

Wow it worked.

Got the games but I use nano and copy and paste to save the games. I tried using scp but

gcloud compute scp [myinstance]:~/cutechess/projects/cli/output_pgn.pgn ~/document

WARNING: The public SSH key file for gcloud does not exist.

WARNING: The private SSH key file for gcloud does not exist.

WARNING: You do not have an SSH key for gcloud.

WARNING: SSH keygen will be executed to generate a key.

Generating public/private rsa key pair.

Enter passphrase (empty for no passphrase): 

Enter same passphrase again: 

Your identification has been saved in /home/[username]/.ssh/google_compute_engine.

Your public key has been saved in /home/[username]/.ssh/google_compute_engine.pub.

The key fingerprint is: [kkk]

And then

ERROR: (gcloud.compute.scp) Could not fetch resource:

 - Insufficient Permission

Also tried this,

scp -i /home/[username]/.ssh/google_compute_engine.pub [username]@ipaddress:home/[username]/cutechess/projects/cli/output_pgn_1.pgn cutechess_file/output_pgn_1.pgn

And got this.

ERROR: (gcloud.compute.scp) Could not fetch resource:

 - Insufficient Permission

Been trying this for 4 hours without success.

So as a temporary solution, I used nano.

Still researching the net on how to download the played games.

Do you have any idea how to copy or download the played games.

Or maybe install a text processor to open the file for copy paste. nano and vim is difficult to copy and paste. I tried to install gedit but not successful.

Thanks.

BTW I use 2 vCPU and a V100 GPU , based from the game move comments the test on fixed 2000 nodes/move is only around 0.10 sec in the opening, so this is about

nps = 2000/0.10 = 20,000

Setting at TC 12s+0.1s, a game is finished after around 40s.

[brian]

It looks like you need to set up your ssh keys. Then you can use scp.

https://cloud.google.com/compute/docs/instances/transfer-files

Would not recommend a text editor for transferring pgn. They can get quite large.

[Jon Mike]

This is very very good news.  We are so close.  We just need some insight to easily store and download the games.  Can you help?

Once we get this, we shouldn't have such discrepancies with test results, since we can share the code and access the same hardware remotely.  Great job, Brian and Jupiter!  Almost there!

On Saturday, October 13, 2018 at 10:38:30 AM UTC-5, Jupiter wrote:

Wow it worked....

[LuckyDay]

great work! Once you guys get it all up and working, consider making an article in the github leela wiki so that the instructions can be saved for future reference. https://github.com/LeelaChessZero/lc0/wiki

[brian]

Another idea: set up a Google drive account which has a shared directory, then you can use something like rclone to move the pgn/other files from the instance to the drive directory.

[Jupiter]

Finally able to transfer file. I use the gcloud compute scp command. Using ssh requires ssh key management. Using gcloud compute, ssh keys are automatically managed by gcloud.

This is how I understand now the google cloud structure for file transfer.

user_local_computer <---> user remote google cloud <---> user google compute instance

user remote google cloud (don't know the exact term for that), but user google compute instance has to be created to have access to user remote google cloud

1. We compile Lc0 and cutechess at user google compute instance

2. We run the match games using cutechess-cli at user google compute instance

3. The generated match games is located at user google compute instance

specifically at

~/cutechess/projects/cli/output_pgn.pgn

because that is what we have specified in our match script

4. To get the file output_pgn.pgn and download it to user_local_computer

a. Open gcloud shell and send command in b.

b. gcloud compute scp --zone [my selected zone] --recurse [instance name]:~/cutechess/projects/cli/output_pgn.pgn output_pgn.pgn

In my case, my selected zone = us-west1-b.

The last part of that command which is output_pgn.pgn will be saved in user remote google cloud

Note the command in b. will only function if user google compute instance is running.

But once the file is already transfered to user remote google cloud, you may stop the instance and still you can download the file from user remote google cloud to

user_local_computer.

5. You can access the file output_pgn.pgn in  user remote google cloud from the gcloud shell. Once you open the gcloud shell (4.a) there is an icon for editor at top right of the box, open it. The file output_pgn.pgn is there and you can select and right-click then export and it will download the file to the user_local_computer,

my user_local_computer is run on windows. That user google compute instance is a linux.

Other note:

1. If you don't delete the instance, the compiled Lc0 and cutechess are still there when you run again the instance. But to run a match, you have to run the match script again thru ssh.

2. If you test another network id, modify the script that would download that id and run that part of the script only.

[Jupiter]

I think brian can do the writings, I can but my understanding is limited.

It is similar to using goole cloud to generate training games, but brian's idea does not use the startup script, instead, he used the ssh window and run script from there. The instance needs to be run first before one can access the ssh window.

If you don't delete the instance and you exited and come back to run it again, there is no need to run the script to compile the Lc0 and cutechess-cli, because those and other files are still there. What you are going to do is to run the tournament script.

If you want to use other network id, then just run the part of the script that downloads the network id. there are other things that needs to be revised in the tournament script, because your are now using the new network file.

Somehow the code can be optimized to simplify the complication, but I am not expert in bash scripting.

Basically you need to write a bash script save it in instance and run it. If you change other tournament conditions you need to modify again or create a new script for it.

One idea is to make an interactive scripts like

What first network id you want to use? 11198

What second network id you want to use? 11248

Then the script will check the instance if those id's are there, if not then the script will automatically download it.

After the download the script will ask

What time control? 12s+0.1s

What is the the filename of pgn output? test1.pgn

And other questions, once done, the script will run on his own.

I believe this kind of thing is only for the people with a little bit of knowledge on scripting, but also for people who are really interested to learn.

[brian]

Ah, I did not realize you could provide a startup script for the instances. Using the startup script is definitely better than copying the script after the instance starts. 

Do you know if there is a way to download a specific weight from the command line? 

[Jupiter]

The disadvantage of startup script is that it will always download/compile lc0 and cutechess. This takes too much time every time you start the session. In my case I  don't use a startup script and don't delete the instance when I stop the session. When I comeback the compiled Lc0 and cutechesss are still there saving me a lot of time. All I have to do is just run the tournament script.

Do you know if there is a way to download a specific weight from the command line? 

I don't know, but to get the weight or network id, I just right-click the link and copy the link address and get the part after the sha=

http://lczero.org/get_network?sha=620d9d9032ae23afb4dad07b02312c6e6186dbe3fb15ac822a488a24d1e7bafe

This 620d9d9032ae23afb4dad07b02312c6e6186dbe3fb15ac822a488a24d1e7bafe is the one that the user will be updating in the script.

If you can automate this, by promting the user of what network id is, one method is to make a list of [network id, sha value] pair

[[11198, sha_value1],[11250, sha_value2] ...]

Then you can lookup the sha value when user supplies the number. Then use that value in the script to download the file when file is not yet in the instance.

To create such list I can use scrapy spider to crawl the network page then save the id, and sha. But will do this from my desktop and upload it to my instance. The issue in this setup is that I need to run the spider regularly to get the latest id's and sha, but it should work for older nets already. But perhaps a spider can be run from that instance, but have not tried it. My background is more on windows not in linux.

[brian]

Alright, I'll take a look when I can. I'm a bit busy this week.

[Jon Mike]

I just wanted to send encouragement for this side project.  It would enable everyone to contribute to testing even if they didn't have the gpus, also it should help immensely for the standardized testing needed for finding the best networks.  Yet, its advantage is far beyond the reasons mentioned.  If you are a programmer/developer and know how to help Brian or Jupiter develop this script, please consider giving a bit of your time to move this forward for the benefit of all.  :)

[Jupiter]

Can you have a look too on using ender62 to be used by Lc0 v0.18.1 in glcoud.

I tried to use this file but the engine would not run when using Ender62. Maybe it has to do with lc0_startup.

I already tried not using the lc0_startup but same problem, engine would not run.

On my pc on win7 after the download, ender62 has a filename ender62.txt.gz

In gcloud if I download ender62 using

wget <url> I only see Ender62 after ls -la command.

Thanks.

You can get ender62 at https://github.com/dkappe/leela-chess-weights/releases/tag/Ender62

[brian]

Hmm, I think you may need to extract it first. 

After downloading the weights try running:

"gunzip ender62.txt.gz"

This will extract the weights in the current directory.

After extracting, it runs with no issue for me.

[Jupiter]

Thanks, got it working now.

My mistake was

wget https://github.com/dkappe/leela-chess-weights/releases/tag/Ender62

It should have been

wget https://github.com/dkappe/leela-chess-weights/files/2507425/ender62.txt.gz

[Jupiter]

Sorry for bothering, but I would like to know if you have a script to install Lc0 v0.19.1-rc2 cuda in google colab? There are interesting params in this release.

I tried to install v0.19.0 before, but was not successfull.

Thanks.

On Saturday, October 13, 2018 at 3:45:52 AM UTC+8, brian wrote:

[brian]

Wait, google colab or google cloud. I think I have the one for google cloud.

[Jupiter]

Sorry, should be for google cloud.

[brian]

If you have a cloud instance, you could go to the lc0 directory and try

git pull origin master 

CC=clang CXX=clang++ ./build.sh

I think this should build the latest dev version. If that did not work you could remove the lc0 directory and rebuild via (I had to do this a while ago...);

rm -rf lc0

git clone --recurse-submodules https://github.com/LeelaChessZero/lc0.git    #(previously this had -b release)

cd lc0 && CC=clang CXX=clang++ ./build.sh

Also, I don't check this forum as often at the moment; but feel free to shoot me an email. 

[Jupiter]

All right I will try that.

[Jupiter]

git pull origin master 
CC=clang CXX=clang++ ./build.sh

Does not work on my existing instance 

4 warnings generated.

[18/120] Compiling C++ object 'lc0@exe/src_mcts_search.cc.o'.

ninja: build stopped: subcommand failed.

I think this should build the latest dev version. If that did not work you could remove the lc0 directory and rebuild via (I had to do this a while ago...);
rm -rf lc0
git clone --recurse-submodules https://github.com/LeelaChessZero/lc0.git #(previously this had -b release)
cd lc0 && CC=clang CXX=clang++ ./build.sh

 

I am not removing Lc0 at the moment, it still has Lc0 v0.18.1, that I can use.  

Instead I created a new instance using the previous script and that change above, but still did not work too.

22 warnings generated.

ninja: build stopped: subcommand failed.

On Sunday, December 9, 2018 at 1:23:14 PM UTC+8, brian wrote:

[brian]

Oh... I'm actually not sure whats causing that particular build error. Usually rebuilding fixes it for me. 

The people on discord would probably know about that error better than I do. Sorry!