make CNN data & use in oakfoam

[cheolmin ki]

hello,

i want to make hdf5 and leveldb data, but i don't know how to sgf file convert.

so i don't test

in oakfoam, i want to use trained data.

regard,

cheolmin

[Detlef Schmicker]

Hello, nice to see you interested,

have a look at the scripts in

/scripts/CNN

they are modifications of
/scripts/features, which are better documented.



mainly you create file with all game file names

than

cat allgamenames.txt| ./CNN-data-collection.sh initial.gamma 13

allPositions.txt

they should already be randomized (the initial.gamma and the 13 are
not really necessary for CNN, it is just because I tried to minimalize
differences to the scripts in /scripts/features)

next step to produce the leveldb database

cat allPositions.txt | python ./generate_sample_data_leveldb.py

produces the leveldb database




I am not sure, if the produced database is consitant with the net code
in the actual oakfoam version, it is very much a work in progress at
the moment, and pushes to the repository are more or less "backups".


pretrained nets

http://physik.de/net.tgz
http://physik.de/net2.tgz

even here, I am not sure if it is read by the actual version of
oakfoam, but should be simple to figure out 


Have fun,


Detlef

[cheolmin ki]

thank you for your reply

i don't know this sentence

cat allgamenames.txt| ./CNN-data-collection.sh initial.gamma 13

allPositions.txt

allPositions.txt file is CNN-data-collection.sh's output?  

2016년 4월 6일 수요일 오후 11시 29분 25초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

sorry, my copying formatted it wrong it

output piped into allPositions.txt, just in case again the next line has wrong format again :) 

cat allgamenames.txt| ./CNN-data-collection.sh initial.gamma 13 > allPositions.txt

Detlef

--
You received this message because you are subscribed to the Google Groups "Oakfoam" group.
To unsubscribe from this group and stop receiving emails from it, send an email to oakfoam+u...@googlegroups.com.
To post to this group, send email to oak...@googlegroups.com.
Visit this group at https://groups.google.com/group/oakfoam.
For more options, visit https://groups.google.com/d/optout.

[cheolmin ki]

thank you, finally i make the leveldb file.

i want to know there are few question.

1. i make the allPosition.txt, but in generate_sample_data_leveldb.py's split location is difference. i don't know what made the file correctly.

2. in lenet_train_test.prototxt, i don't know why composed of two input (train set composed of leveldb, test set composed of hdf5 format)

    how the two may be constructed in the same format?

3. using my data(549 of sgf file)  from caffe, result is iteration 5000 to 9%.(leveldb and hdf5 data is same file allPosition.txt).

   i don't know it is correctly result,

i attached my file, i want to you test this file.

https://drive.google.com/open?id=0B6wWJs0EGrqsc3ZHeWRTTXlEUzQ

thank you for reading

cheolmin

2016년 4월 7일 목요일 오전 1시 3분 16초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

Detlef

Am 08.04.2016 um 10:52 schrieb cheolmin ki <icom...@chol.com>:

thank you, finally i make the leveldb file.

i want to know there are few question.

1. i make the allPosition.txt, but in generate_sample_data_leveldb.py's split location is difference. i don't know what made the file correctly.

probably the file is not consistent:( There are usually minor changes, please have a look.

I can send you my actual files in the afternoon, but you have to use the oakfoam version from dsmic branch than, but you will have other problems, as this will produce a different format. I think the easiest is to have a look, what is different in the file and correct in the generate.   This will help you to understand as well:(

2. in lenet_train_test.prototxt, i don't know why composed of two input (train set composed of leveldb, test set composed of hdf5 format)

    how the two may be constructed in the same format?

you usually take part of the games to make a testing database, most for the training database

3. using my data(549 of sgf file)  from caffe, result is iteration 5000 to 9%.(leveldb and hdf5 data is same file allPosition.txt).

   i don't know it is correctly result,

you will have to ust several 10000 sgf's

[cheolmin ki]

i will be reinstall dsmic version.

thank you for your kindness,

if sgf file consist of little set, caffe test accuracy may be incorrectly?

cheolmin

2016년 4월 8일 금요일 오후 6시 25분 59초 UTC+9, Detlef Schmicker 님의 말:

[cheolmin ki]

hello, i trained sgf file.

so i use classify.py for test accuracy
but i don't know how to use this file.

classify's input data npy or jpg?

and i collected sgf file from kgs go server, but data collection was so little.
could you send me your sgf files?

cheolmin



2016년 4월 8일 금요일 오후 11시 14분 25초 UTC+9, cheolmin ki 님의 말:

[Steve Kroon]

Hi.

Hopefully Detlef can answer your technical questions.  But I can help with the last bit: You should be able to get plenty of (almost 60000) pro go games from http://badukmovies.com/pro_games (see the right sidebar).

Steve

[Detlef Schmicker]

sorry, had a full mailbox some days ago :(

classify.py I used in the very first tests, it is probably consistent with my first nets, but did not get updated to later versions, as now I use oakfoam to do this tests.

It would take the same file as the positions file used for generate I think... but this file has changed since then, sorry :( I think we should remove the file from the repository!

on the other hand it shows the principle of classification quite nicely, so I dont know, mor doc would be good I think. 

The accuracy now is displayed during training usually!

I used http://www.u-go.net/gamerecords/

about 130000 games

which is enough if you dont try value network. take care, there is a parameter in the generation to take only 8% of the positions at the moment, than you need more games 4d+ to get enough! I dont remember the archive at the moment, aya author gave the link some weeks ago in computer go mailing list...

about 1 million games, but I did not use once, which are lost on time or used handicap

Detlef

[cheolmin ki]

thank you. i have made leveldb and training now.

i have some question,

i want to use pretrained file in oakfoam and test single play, but i don't know how to apply to oakfoam.
what should i do?

and in lenet_train_test_value.prototxt, i had to all blobs_lr=1 and it is prediction net + value net?
i wonder if i training prediction net and value net, do i need seperately training?

cheolmin


2016년 4월 14일 목요일 오후 8시 10분 57초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

loadcnnp /home/detlef/oakfoam-hg/oakfoam/scripts/CNN/go_value4d_19_b.prototxt /home/detlef/oakfoam-hg/oakfoam/scripts/CNN/go_value4d_19_b.trained 5

loads the prediction value net (the 5 indicates the type of net, you can see in the loadcnnp function)

param test_p100 1.0

param test_p101 0.0

param test_p102 0.0

puts only cnn mode on

I can send my conf file directly to you, as I do not have it on my IPad:)

Yes, I trained it in two phases, first only prediction network, than value. It also worked training both together, but I had some overfitting, and had to use less filters in value network.

Also it might  be a good idea to train prediction with 6d+ games (130000) and use 4d+ games for value net, as 130000 games are not enough!

Detlef

[cheolmin ki]

i try to oakfoam, but i met this problem

caffe not availible, compile with-caffe

why this exception came up?

and *.trained file is caffemodel file? i don't know this file.

cheolmin

2016년 4월 19일 화요일 오후 5시 13분 18초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

you have to configure with this option, and give the caffe directory. caffe is not standard installed on linux systems, so Configure can not guess where it is!

.trained is the parameter file you got from training (snapshot directory .caffemodel file)

Detlef

[cheolmin ki]

i try to linking caffe, but caffe lib is not available,

i put on the command

./configure --with-caffe=/home/kk/caffe

it makes error, and open the config.log

configure:4164: g++ -E  -I/home/kk/caffe/include -I/usr/local/cuda/include conftest.cpp
configure:4164: $? = 0
configure:4178: g++ -E  -I/home/kk/caffe/include -I/usr/local/cuda/include conftest.cpp
conftest.cpp:12:28: fatal error: ac_nonexistent.h: No such file or directory
 #include <ac_nonexistent.h>
                                           ^
compilation terminated.

cheolmin

2016년 4월 19일 화요일 오후 11시 24분 51초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

I use 

./configure --with-caffe=/home/kk/caffe/distribute

Detlef

[cheolmin ki]

i'm already your method..

but showed message

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of g++... gcc3
checking whether g++ understands -c and -o together... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking how to run the C++ preprocessor... g++ -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... no
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking cuda.h usability... yes
checking cuda.h presence... yes
checking for cuda.h... yes
checking whether caffe lib is available... no
configure: error: in `/home/kk/oakfoam':
configure: error: Caffe couldn't be found!
See `config.log' for more details

ansi c header file and caffe lib's status no.

cheolmin

2016년 4월 20일 수요일 오전 1시 53분 1초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

did you produce the distribute directory with make distribute. after building caffe?

Detlef

[cheolmin ki]

yes i was build caffe after make distribute.

i try to various method but i don't complete yet

cheolmin

2016년 4월 21일 목요일 오전 1시 39분 48초 UTC+9, Detlef Schmicker 님의 말:

[cheolmin ki]

thank you very much!

finally i finished configure & make.

and i start ./oakfoam -c nicego-clop.gtp.

after start oakfoam, i met the error

segmentation error (core dumped) because my prototxt file is different your go_value4d_19_b.prototxt

so receive me your prototxt file? thanks detlef

cheolmin

2016년 4월 21일 목요일 오전 2시 15분 11초 UTC+9, cheolmin ki 님의 말:

[Detlef Schmicker]

Nice to hear this!!!!

Would be great, if you could write some notes on how you managed to
compile!!!

I will add them to the INSTALL file!!!!!

(or just modify the INSTALL file and I will push it :)

This would help the next who tries to compile!!

Detlef

[cheolmin ki]

ok if i complete this, i will submit installing guide :)

i finished ./configure and make so i start oakfoam but

./oakfoam -c nicego-clop.gtp

after created net, i met warning.

ok, created net
I0422 00:58:58.826647 22771 net.cpp:752] Ignoring source layer mnist
I0422 00:58:58.826683 22771 net.cpp:752] Ignoring source layer label_mnist_1_split
I0422 00:58:58.826701 22771 net.cpp:752] Ignoring source layer slice
I0422 00:58:58.826706 22771 net.cpp:752] Ignoring source layer data_slice_0_split
I0422 00:58:58.826724 22771 net.cpp:752] Ignoring source layer slice
I0422 00:58:58.826727 22771 net.cpp:752] Ignoring source layer slice
I0422 00:58:58.827903 22771 net.cpp:752] Ignoring source layer ip_zw_flat_0_split
I0422 00:58:58.828186 22771 net.cpp:752] Ignoring source layer pool1
I0422 00:58:58.828191 22771 net.cpp:752] Ignoring source layer flat_pool
I0422 00:58:58.828197 22771 net.cpp:752] Ignoring source layer power
I0422 00:58:58.828200 22771 net.cpp:752] Ignoring source layer conv19b_3x3_1
I0422 00:58:58.828205 22771 net.cpp:752] Ignoring source layer conv19c_3x3_1
I0422 00:58:58.828209 22771 net.cpp:752] Ignoring source layer accuracy
I0422 00:58:58.828213 22771 net.cpp:752] Ignoring source layer loss1
I0422 00:58:58.828217 22771 net.cpp:752] Ignoring source layer loss2
I0422 00:58:58.828222 22771 net.cpp:752] Ignoring source layer loss3
I0422 00:58:58.828227 22771 net.cpp:752] Ignoring source layer loss4
Attention, fixed coded as caffe support broken?!   --->>> !!!!!!!!!!!!!!!!!!!!!!!! 1 shape 20

i don't know why cause this warning.
it was incorrectly entered in the file on loadcnnp?

loadfeaturegammas /home/kk/oakfoam/trained_playout_and_circ_for_cnn.gamma
loadcnnp /home/kk/oakfoam/go_value4d_19_b.prototxt /home/kk/oakfoam/go_value4d_19_b.trained 5

cheolmin

2016년 4월 21일 목요일 오후 11시 21분 15초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

this is perfectly fine, the layers are in the training net, but not in the net used by oakfoam (they are below and above the 

#this part should be the same in ....

part


everything should be fine now!

Detlef

[cheolmin ki]

hi detlef

i'm sorry to late write readme file. but I'm not that good at english. so I don't know whether I wrote for the sentence. :)

https://drive.google.com/file/d/0B6wWJs0EGrqsN05xeGpodGxsVmc/view?usp=sharing

It encourages, you to modify. :)

and i have some question,

do you upload bot in kgs go server? what is bot's name?

your method's feature plain and facebook computer go feature plain are same?

cheolmin

2016년 4월 22일 금요일 오전 1시 15분 54초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

Detlef

Am 26.04.2016 um 15:42 schrieb cheolmin ki <icom...@chol.com>:

hi detlef

i'm sorry to late write readme file. but I'm not that good at english. so I don't know whether I wrote for the sentence. :)

https://drive.google.com/file/d/0B6wWJs0EGrqsN05xeGpodGxsVmc/view?usp=sharing

I did it, thanks a lot!

It encourages, you to modify. :)

and i have some question,

do you upload bot in kgs go server? what is bot's name?

nicego19n usually, but mostly on tournaments

at the moment I am running on cgos under the name NG-

NG-ct_only_cnn is the net I send you and shuffling between 3 best moves of the nets move prediction

param cnn_random_for_only_cnn 3

param move_policy cnn

when you look at the stderr (or in autosave sgf) you can also see the estimation of the value network, which seems quite fine....

your method's feature plain and facebook computer go feature plain are same?

yes, they should be similar, but not exactly the same.

I do have different loss function, as I do not think caffe supports softmax_loss layers with other planes, only with labels.

And it does not support more than 1 label, therefor I used eucledian loss for future prediction layers....

[cheolmin ki]

hi detlef

how to consist of your value network's input plane?
in allPositions.txt, RE is your value network's input data?
i don't understand how to consist of value network...

cheolmin

2016년 4월 26일 화요일 오후 11시 51분 53초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

yes, re is taken from the sgf game records (result) and is taken as the input.

it fills a whole plane if I remember correctl! because caffe does not support different input sizes....

the value network gives a winrate which it learned from the game results.

Detlef

[cheolmin ki]

thank you, i have question change paramerter.

param playouts_per_move 8000
param playouts_per_move_max 8000
param playouts_per_move_min 4000

above this parameter, MCTS search count? if i increase this parameter, win rate is increase?

and i don't know uct parameter...

i'm sorry many question

cheolmin

2016년 5월 3일 화요일 오전 12시 31분 44초 UTC+9, Detlef Schmicker 님의 말:

[Detlef Schmicker]

see inserts below

Detlef

Am 04.05.2016 um 03:47 schrieb cheolmin ki <icom...@chol.com>:

thank you, i have question change paramerter.

param playouts_per_move 8000
param playouts_per_move_max 8000
param playouts_per_move_min 4000

above this parameter, MCTS search count? if i increase this parameter, win rate is increase?

yes

and i don't know uct parameter...

they are named as in most papers, e.g. remi's papers

http://www.remi-coulom.fr

also there are references in the deepming paper

https://gogameguru.com/i/2016/03/deepmind-mastering-go.pdf

[Long Pham]

I have a problem with "cat allgamenames.txt | ./CNN-data-collection.sh initial.gamma 13 > allPositions.txt". When I run it, it says:

cat: patterns_circ_2016-07-27_11:58:44222757214.tmp: No such file or directory

Can anyone help me?

[Detlef Schmicker]

Hmm,

it is in CNN-data.sh and something is wrong with

gogui-adapter "$OAKFOAM" 

My guesses: gogui is very old, or 
OAKFOAM="$WD/../../oakfoam --nobook --log $TEMPOUTPUT" 

../../oakfoam is not the executable and gogui is not producing an error message?!

[Long Pham]

What do you mean? And how can I fix this?

P.S: My gogui is new

[Detlef Schmicker]

my guess is: cygwin names the executable oakfoam.exe ??

so you habe to change this in CNN-data.sh.

if you really want to use all scripts it might be a good idea, to change to Linux! you will probably get even more problems, if you try to compile with cnn support under cygwin!

The standard compilation does not have cnn support (it is not needed for the script, so this is not the problem.) but if you try to run with a cnn oakfoam has to be configured --with-caffe= cafffe disteibution directory, and I really dont know what happens to caffe under cygwin!

[Long Pham]

You mean gogui-adapter "$OAKFOAM"  is not oakfoam.exe?

[Detlef Schmicker]

line 10

OAKFOAM="$WD/../../oakfoam --nobook --log $

TEMPOUTPUT"

defines that ../../oakfoam is executed

might be

OAKFOAM="$WD/../../oakfoam.exe --nobook --log $TEMPOUTPUT"

in your case

than 


gogui-adapter "$OAKFOAM"  

uses oakfoam.exe


But really, change to Ubnuntu 14.04lts :)

[Long Pham]

I still don't understand

1. OAKFOAM="$WD/../../oakfoam --nobook --log $TEMPOUTPUT" - What does this code mean?

2. gogui-adapter "$OAKFOAM"  

   uses oakfoam.exe

=> gogui-adapter "$OAKFOAM.exe" right?

P.S: sorry for my stupidity :(

[Detlef Schmicker]

On Wednesday, July 27, 2016 at 1:12:35 PM UTC+2, Long Pham wrote:

I still don't understand

1. OAKFOAM="$WD/../../oakfoam --nobook --log $TEMPOUTPUT" - What does this code mean?

this line means the:

in line 5 the variable WD is set to the current directory:

WD="$(dirname "$0")"

in line 10 the variable OAKFOAM is set to "../../oakfoam --nobook --log $TEMPOUTPUT"   from the current directory

this means: ../../oakfoam should execute oakfoam from the current directory: but it will not, therefore I suggested to change this to oakfoam.exe !!!

oakfoam is called with the parameters --nobook (do not use opening book) and --log $TEMPOUT (put the log file into TEMPOUT, which is

patterns_circ_........)
this log file will contain info used for producing the database files !!


gogui-adapter "$OAKFOAM" is than translated to

gogui-adapter "../../oakfoam --nobook --log patterns_circ_........"

but it should in your case be translated to
gogui-adapter "../../oakfoam.exe --nobook --log patterns_circ_........"


therefore my suggestion: probably you have to change line 10 to 

OAKFOAM="$WD/../../oakfoam.exe --nobook --log $TEMPOUTPUT"

[Long Pham]

Thanks Detlef, but I have new problem: allPositions.txt is empty. Is this a problem?

[Detlef Schmicker]

yes:(
do you have full pathes in allgames?

[Long Pham]

Sure

I have patterns_circ and collection_circ but allPositions is empty

[Detlef Schmicker]

and

collection_circ_ ???


uncomment the two last lines in CNN-data-collection.sh to keep the intermediate files for debugging

cat ${TEMPOUTPUT} | sort -R >> ${TEMPCOMBINED}
cat $TEMPCOMBINED

#rm -f $TEMPOUTPUT
#rm -f $TEMPCOMBINED

[Long Pham]

Thanks Detlef

I have a question: What is different between generate_sample_data and generate_sample_data_leveldb?

I see generate_sample_data_leveldb has caffe and leveldb. I know about caffe but I don't know about leveldb

Can you explain it for me?

[Detlef Schmicker]

I started with hdf5 files for training, but quickly used leveldb databases (hdf5 is keept in ram memory, and my actual training database is 200GByte compressed).

I strongly recomand using leveldb databases for training with caffe, so you will have to use generate_sample_data_leveldb!

both is supported by caffe, and documented in there :)

[Long Pham]

New problem is appeared. I don't have leveldb, so I build leveldb for windows. Ok, no module name leveldb.

What's file in leveldb and how can I fix this?

[Detlef Schmicker]

https://pypi.python.org/pypi/leveldb

[Long Pham]

How can I compile this? It says "don't know how to compile leveldb for cygwin"

[Detlef Schmicker]

sorry, no idea :(

use ubuntu 14.04 :)

[Long Pham]

After solve the import leveldb, I have a new problem: no module name _caffe

I installed the pycaffe, so I don't know why this happen :(

[Long Pham]

I try to use generate_sample_data.py and have sample_data.h5. What should I do to train by caffe?

[Detlef Schmicker]

please have a look at the caffe examples for both questions.

there are classify.py samples, which use caffe module and the net definitions use the hdf5 files for training.

just do one of the samples, have a look at the prototxt files ...

the lenet_train_test.prototxt demonstrates to use both types in the beginning.

you will need a training and a test dataset.....

[Long Pham]

What should I get after run train_lenet.sh?

[Detlef Schmicker]

you get a .caffemodel file in the snapshot directory, which you load together with the prototxt file of the net.

I usualy rename the files to .trained, as you see in my sample config file.

you will need to set parameter test_p100 to 1 to use the file in tree search...

attention, the sample file uses some parameters, which are not supportet in francois brench, you might want to use dsmic branch .

and you will probably load your net file with obmitting the 5 at the end of the loadccnp  command, as this indicates it is a prediction and value network!

but before all this you will have to configure --with-caffe !!

if you use dsmic you might set move policy to cnn in parameters(general), which would only use your net for making the move....

[Long Pham]

Hi Detlef,

I have a new problem. When I run train_lenet.sh, it says "db_leveldb.cpp:18 Opened leveldb train_leveldb/" and stops caffe

Can you help me fix this?

[Detlef Schmicker]

no, database is not correct?

this is how it looks like if caffe really can open the database

I0731 08:27:59.678020 22207 net.cpp:399] mnist -> data_1
I0731 08:27:59.678027 22207 net.cpp:399] mnist -> label
I0731 08:28:00.103323 22214 db_leveldb.cpp:18] Opened leveldb test_4d_valueb/
I0731 08:28:00.164263 22207 data_layer.cpp:41] output data size: 64,23,19,19
I0731 08:28:00.168335 22207 net.cpp:141] Setting up mnist
I0731 08:28:00.168395 22207 net.cpp:148] Top shape: 64 23 19 19 (531392)
I0731 08:28:00.168413 22207 net.cpp:148] Top shape: 64 (64)
I0731 08:28:00.168422 22207 net.cpp:156] Memory required for data: 2125824
I0731 08:28:00.168433 22207 layer_factory.hpp:77] Creating layer slice
I0731 08:28:00.168457 22207 net.cpp:91] Creating Layer slice
I0731 08:28:00.168469 22207 net.cpp:425] slice <- data_1

[Long Pham]

Maybe my database is incorrect. What is inside your test_4d_valueb folder?

[Detlef Schmicker]

you seperate all games into two groups, around 1000 games you use for testing and create the test_ database from it (you rename the produced to test_) and from the rest you create a train_ database....

[Long Pham]

My caffe.exe still crash when open leveldb. I think my train_leveldb is incorrect The .log txt file in it is empty. Is this a problem?

[Detlef Schmicker]

I really dont know. Did generate throw error messages?

I think it schauld output every 1000 something....

you might turn on some output in generate...

And I dont know, but can you thrust leveldb on cygwin, I read it is not officially supportes?

[Long Pham]

Generate don't have error messages, it runs normally

I cannot install leveldb for cygwin, so I use cmd :)

[Long Pham]

Hi Detlef,

I have a question. When I make position file, my computer crash, so it is incomplete. I have a collection_circ tmp file and lots of patterns_circ tmp files, but allPositions.txt is empty.  What can I do to sort and combine patterns to allPositions.txt?

[Detlef Schmicker]

use line 33 of the cnn-data-colect script,

cat colect_circ | sort -R > allPositions.txt

(the script weites the number of lines into the first line, but I think this is not ncessary anymore)

sort -R can take quite long and is using a lot of temp space (in the /tmp directory). it randomizes all lines of collect_circ, in big databases it can take some days (200GB)

if for some reason your /tmp directory is in ram (I dont know cygwin defaults and am to lazy too google them) you might have to choose another directory!

export TMPDIR=/home/where_ever_you_have_lotsof_GB

Detlef

Am 10.08.2016 um 06:44 schrieb Long Pham <superm...@gmail.com>:

Hi Detlef,

I have a question. When I make position file, my computer crash, so it is incomplete. I have a collectione_circ tmp file and lots of patterns_circ tmp files, but allPositions.txt is empty.  What can I do to sort and combine patterns to allPositions.txt?

--
You received this message because you are subscribed to the Google Groups "Oakfoam" group.
To unsubscribe from this group and stop receiving emails from it, send an email to oakfoam+u...@googlegroups.com.
To post to this group, send email to oak...@googlegroups.com.
Visit this group at https://groups.google.com/group/oakfoam.
For more options, visit https://groups.google.com/d/optout.

[Steve Kroon]

Note that if you do not have 200GB available on /tmp, sort also has a -T option for specifying the temp directory.

Steve

On Wed, Aug 10, 2016 at 7:25 AM, Detlef Schmicker <dschm...@physik.de> wrote:

use line 33 of the cnn-data-colect script,

cat colect_circ | sort -R > allPositions.txt

(the script weites the number of lines into the first line, but I think this is not ncessary anymore)

sort -R can take quite long and is using a lot of temp space (in the /tmp directory). it randomizes all lines of collect_circ, in big databases it can take some days (200GB)

if for some reason your /tmp directory is in ram (I dont know cygwin defaults and am to lazy too google them) you might have to choose another directory!

export TMPDIR=/home/where_ever_you_have_lotsof_GB

Detlef

Am 10.08.2016 um 06:44 schrieb Long Pham <superm...@gmail.com>:

Hi Detlef,

I have a question. When I make position file, my computer crash, so it is incomplete. I have a collectione_circ tmp file and lots of patterns_circ tmp files, but allPositions.txt is empty.  What can I do to sort and combine patterns to allPositions.txt?

--
You received this message because you are subscribed to the Google Groups "Oakfoam" group.

To unsubscribe from this group and stop receiving emails from it, send an email to oakfoam+unsubscribe@googlegroups.com.

To post to this group, send email to oak...@googlegroups.com.
Visit this group at https://groups.google.com/group/oakfoam.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Oakfoam" group.

To unsubscribe from this group and stop receiving emails from it, send an email to oakfoam+unsubscribe@googlegroups.com.

[Detlef Schmicker]

yes, and it reads the TMPDIR variable, at least on most linux distibutions :)

Detlef

To unsubscribe from this group and stop receiving emails from it, send an email to oakfoam+u...@googlegroups.com.

[Long Pham]

When I make leveldb database, can I combine something like leveldb database, allPositions.txt, collection_circ, patterns_circ,...?

I don't want to start again :(

[Detlef Schmicker]

I dont fully understand, bur 

from collection_circ   you can make  allPositions.txt as in the last mails,

from Positions.txt you make leveldb database.

The scripts dont handle break inbetween, sorry...

for this to handle, one would need to keep track of the last game (position) finished....

but ...

Detlef

Am 11.08.2016 um 02:39 schrieb Long Pham <superm...@gmail.com>:

When I make leveldb database, can I combine something like leveldb database, allPositions.txt, collection_circ, patterns_circ,...?

I don't want to start again :(

--

[Long Pham]

So we cannot upgrade leveldb database or collection_circ, we must start from beginning when upgrade it, right?

[Detlef Schmicker]

yes, but it should not be magic to write a script which can concat two databases....

I made a python script to count the number of entries:

import os

import sys

import leveldb

db_file_name=sys.argv[1]

print "counting database: ",db_file_name

db = leveldb.LevelDB(db_file_name)

count=0

for key in db.RangeIter(include_value=False):

  count +=1

print "counted: ",count

Detlef

Am 11.08.2016 um 11:37 schrieb Long Pham <superm...@gmail.com>:

So we cannot upgrade leveldb database or collection_circ, we must start from beginning when upgrade it, right?

--

[Long Pham]

Hi Detlef,

I have an error when run train_lenet:

I0905 17:25:44.874114  8382 db_leveldb.cpp:18] Opened leveldb test_leveldb/

*** Aborted at 1473071144 (unix time) try "date -d @1473071144" if you are using GNU date ***

PC: @     0x7fa3b4347a38 (unknown)

*** SIGSEGV (@0x0) received by PID 8374 (TID 0x7fa396777700) from PID 0; stack trace: ***

    @     0x7fa3bb2eacb0 (unknown)

    @     0x7fa3b4347a38 (unknown)

    @     0x7fa3b434455e (unknown)

    @     0x7fa3bca073d2 caffe::db::LevelDBCursor::value()

    @     0x7fa3bca2230b caffe::DataReader::Body::read_one()

    @     0x7fa3bca226d4 caffe::DataReader::Body::InternalThreadEntry()

    @     0x7fa3bc9be660 caffe::InternalThread::entry()

    @     0x7fa3b33f8a4a (unknown)

    @     0x7fa3b16c8184 start_thread

    @     0x7fa3bb3ae37d (unknown)

    @                0x0 (unknown)

Segmentation fault (core dumped)

Do you have any idea?

[Detlef Schmicker]

sorry, no

Detlef

[Long Pham]

Hi Detlef,

I don't know why I cannot use leveldb, so I want to use lmdb, but I don't know how to write it. Can u help me?

[Detlef Schmicker]

sorry, no 

I never used it :(

Detlef

Am 10.09.2016 um 14:14 schrieb Long Pham <superm...@gmail.com>:

Hi Detlef,

I don't know why I cannot use leveldb, so I want to use lmdb, but I don't know how to write it. Can u help me?

--

[Long Pham]

Ok :( Can u send me your generate leveldb?

[Detlef Schmicker]

This is what I use at the moment, but it produces additional layers (e.g. results and therefore uses other input format)

I think this three is all you need, but I am not sure :(

[Long Pham]

Thank you very much! I finally solve my problem

[Long Pham]

Hi Detlef,

I have new error: data_transformer.cpp:465 Check failed: datum_channels > 0 (0 vs. 0)

[Detlef Schmicker]

sorry, no

Detlef

--

[Long Pham]

I have a question:

Why color=elements[19*19] and color=elements[19*19*3] are so different?

When I use color=elements[19*19], my data is empty, and I cannot train it.

But when I use color=elements[19*19*3], my data has some sst file (or ldb file on Ubuntu), and I can train it.

Can u help me?

[Detlef Schmicker]

There is more info in my txt file now. I do not remember exactly what, but obciously I offer 3 boards, one with the stones, one with simple ko positions and probably one with calculated territorial, if enabled in the script?

But I am not sure...

Detlef

--

[Long Pham]

Hi Detlef,

I train my data, but it's accuracy always = 0

Do u know why?

[Detlef Schmicker]

it might he 0 quite a while, but you must see the loss going down.

accurency is the ratio of correct predictions.....

how many iterations and what batch size are you talking about?

Detlef

--

[Long Pham]

The loss doesn't go down, it's very random...

I use batch size 64 and iterations is 10k

[Detlef Schmicker]

10k, really

this is verry kittle

if loss is jumping around probably learningrate is too big

Detlef

Am 23.09.2016 um 01:36 schrieb Long Pham <superm...@gmail.com>:

The loss doesn't go down, it's very random...

I use batch size 64 and iterations is 10k

--

[Long Pham]

I see in features has many scripts to harvest patterns and train.

What is "train gammas"? Can I use caffe instead of it?

[Detlef Schmicker]

yes, it is fot pattern instead of cnn. only used for cs style playouts now

Detlef

Am 23.09.2016 um 05:24 schrieb Long Pham <superm...@gmail.com>:

I see in features has many scripts to harvest patterns and train.

What is "train gammas"? Can I use caffe instead of it?

--

[Long Pham]

Hi Detlef,

I have some troubles with my caffemodel. When I try to use it, it says: blob.cpp 420 trying to copy blobs of different size

I don't know how to fix it. Do you have any idea?

[Detlef Schmicker]

you must use the correct number after the loadcnnp comman, depending of the entries in your input layer. have a look in the source.

my most actuell files use 5, but this is for combined policy and value net....

Detlef

--

[Long Pham]

Hi detlef,

How many iterations are you think enough?

My loss decrease at the beginning and then it's very random, even though I decrease learning rate

[Detlef Schmicker]

about 100000 itterations should give reasonable accurency and loss. 

you are talking about testin loss and accurency? Training loss is very random, depending on batch size!

Detlef

--

[Long Pham]

So I must wait to 100000 iterations to see everything is correct or not. Is there any faster way?

[Detlef Schmicker]

Not really, I do testing every 5000 iterations. for testing I use test_iter 656 with batch size 128.

this number is from the number of positions in my test database (128*656)

you might check every 1000 iterations, and this should decrease more smoothly (but even they are jumping around, if learning rate is big.

In the beginning I use learning rate 0.005 at the moment.

Detlef

Am 29.09.2016 um 10:09 schrieb Long Pham <superm...@gmail.com>:

So I must wait to 100000 iterations to see everything is correct or not. Is there any faster way?

--

[Long Pham]

Hi Detlef,

I have some problems with my caffemodel, but it's difficult to describe it...

So here is my caffemodel, can you check it and give me some advices?

Thank you in advance!

https://drive.google.com/open?id=0Bw58qGzeN0qBdE04YS1obElxZE0

[Detlef Schmicker]

I am not sure, but I do not support only 2 input layers for quite a while anymore.

You might have a look in the history and build an old version of oakfoam but the minimum now is probaly the 13 layer model?!

you might of cause reintroduce the 2 layer into ::getCNN in Engine.cc ....


but I did not think this makes too much sense, as the 2 layer nets are too weak .....

[Long Pham]

Did you use it in oakfoam?

When I use it, the bot plays very strange. It plays d4, d3, d17, d16, q16, q17, q3, q4 and don't care about my moves...

[Detlef Schmicker]

I could not use it in my version, please send me your gtp file. may be I dont remember how to load 2 layer cnn...

Detlef

Am 26.10.2016 um 03:22 schrieb Long Pham <superm...@gmail.com>:

Did you use it in oakfoam?

When I use it, the bot plays very strange. It plays d4, d3, d17, d16, q16, q17, q3, q4 and don't care about my moves...

--

[Long Pham]

In engine.cc, I change "caffe_test_net_input_dim" line 699 to 13 and build it

[Detlef Schmicker]

hmm, sounds wrong...

in this post I published a 2 layer cnn and the source to use it

http://computer-go.org/pipermail/computer-go/2015-April/007573.html

Detlef

Am 27.10.2016 um 07:30 schrieb Long Pham <superm...@gmail.com>:

In engine.cc, I change "caffe_test_net_input_dim" line 699 to 13 and build it

--