Changing Parameters when Fine Tuning

155 views
Skip to first unread message

j.b...@churadata.okinawa

unread,
Aug 20, 2018, 9:57:55 PM8/20/18
to tesseract-ocr
Is it possible to change the parameters when Fine Tuning?

The documentation says "Fine tuning is the process of training an existing model on new data without changing any part of the network", but does that mean that parameters like momentum, learning rate, etc.  cannot be changed?

Shree Devi Kumar

unread,
Aug 21, 2018, 12:14:29 AM8/21/18
to tesser...@googlegroups.com
If you want to change parameters, please look at the replace layers option. With fine tuning you cannot change them.

On Tue 21 Aug, 2018, 7:27 AM , <j.b...@churadata.okinawa> wrote:
Is it possible to change the parameters when Fine Tuning?

The documentation says "Fine tuning is the process of training an existing model on new data without changing any part of the network", but does that mean that parameters like momentum, learning rate, etc.  cannot be changed?

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-oc...@googlegroups.com.
To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/c671d6ad-7437-4d5c-8072-3c5a77c88268%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jacob Biros

unread,
Aug 21, 2018, 2:30:21 AM8/21/18
to tesser...@googlegroups.com
Thank you for the prompt response!

On Tue, Aug 21, 2018 at 1:14 PM, Shree Devi Kumar <shree...@gmail.com> wrote:
If you want to change parameters, please look at the replace layers option. With fine tuning you cannot change them.

On Tue 21 Aug, 2018, 7:27 AM , <j.b...@churadata.okinawa> wrote:
Is it possible to change the parameters when Fine Tuning?

The documentation says "Fine tuning is the process of training an existing model on new data without changing any part of the network", but does that mean that parameters like momentum, learning rate, etc.  cannot be changed?

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-ocr+unsubscribe@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/c671d6ad-7437-4d5c-8072-3c5a77c88268%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "tesseract-ocr" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tesseract-ocr/hjZizqQm2wA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tesseract-ocr+unsubscribe@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/CAG2NduVyb9NMLmA%2BCnY2ZSi%2BSv0LT1cmQ8SgcuP5xLHBoa69fg%40mail.gmail.com.

j.b...@churadata.okinawa

unread,
Aug 21, 2018, 3:46:17 AM8/21/18
to tesseract-ocr
Sorry, one more question.  We set up 4 different machines all running the command below except for minor differences in the momentum and the learning rate.  Changing the momentum and learning rate in this situation, because it is fine tuning, shouldn't affect anything right?  In our case though each machine produced different results.  Do you have any idea what exactly is causing this?  I can provide more information as necessary.  Thanks.

training/lstmtraining --traineddata ~/tesstutorial/jpntrain/jpn/jpn.traineddata \
  --continue_from ~/fine_tuning/models/jpn.lstm \
  --net_spec '[1,36,0,1 Ct3,3,16 Mp3,3 Lfys48 Lfx96 Lrx96 Lfx256 O1c111]' \
  --model_output ~/bouch_fine_tuning/0819_fine20fonts_mom05_lr1e-4_010/base --learning_rate 1e-4 \
  --momentum 0.5 \
  --train_listfile ~/bouch_train/jpntrain/jpn.training_files.txt \
  --eval_listfile ~/bouch_train/jpneval/jpn.training_files.txt \
  --max_iterations 100000 &>~/bouch_fine_tuning/0819_fine20fonts_mom05_lr1e-4_010/basetrain.log \
  --old_traineddata /usr/local/share/tessdata/__official_jpn.traineddata \
  --debug_interval -1

Shree Devi Kumar

unread,
Aug 21, 2018, 5:09:20 AM8/21/18
to tesser...@googlegroups.com
>lstmtraining --model_output ./layer_from_deva/layer --continue_from ./layer_from_deva/Devanagari.lstm --append_index 5 --net_spec '[Lfx192 O1c1]' --traineddata ./sanplustrain/san/san.traineddata --train_listfile ./sanplustrain/san.training_files.txt --eval_listfile ./sanpluseval/san.training_files.txt --debug_interval 0 --max_image_MB 6000 --max_iterations 500
Loaded file ./layer_from_deva/Devanagari.lstm, unpacking...
Warning: LSTMTrainer deserialized an LSTMRecognizer!
Continuing from ./layer_from_deva/Devanagari.lstm
Appending a new network to an old one!!Warning: given outputs 1 not equal to unicharset of 149.
Num outputs,weights in Series:
  Lfx192:192, 197376
  Fc149:149, 28757
Total weights = 226133
Built network:[1,48,0,1[C3,3Ft16]Mp3,3Lfys64Lfx64Lrx64Lfx192Fc149] from request [Lfx192 O1c1]
Training parameters:
  Debug interval = 0, weights = 0.1, learning rate = 0.001, momentum=0.5


I  have not tried changing the parameters even with replace layers. Do provide feedback on your experience,


On Tue, Aug 21, 2018 at 12:00 PM Jacob Biros <j.b...@churadata.okinawa> wrote:
Thank you for the prompt response!
On Tue, Aug 21, 2018 at 1:14 PM, Shree Devi Kumar <shree...@gmail.com> wrote:
If you want to change parameters, please look at the replace layers option. With fine tuning you cannot change them.

On Tue 21 Aug, 2018, 7:27 AM , <j.b...@churadata.okinawa> wrote:
Is it possible to change the parameters when Fine Tuning?

The documentation says "Fine tuning is the process of training an existing model on new data without changing any part of the network", but does that mean that parameters like momentum, learning rate, etc.  cannot be changed?

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-oc...@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/c671d6ad-7437-4d5c-8072-3c5a77c88268%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "tesseract-ocr" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tesseract-ocr/hjZizqQm2wA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tesseract-oc...@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/CAG2NduVyb9NMLmA%2BCnY2ZSi%2BSv0LT1cmQ8SgcuP5xLHBoa69fg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-oc...@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/CAGO_jRmrNJ%2BYG0E_xuKhhs5DFdvamKWL3NdZLNkLcjxzxCt4pQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.


--

____________________________________________________________
भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

Jacob Biros

unread,
Aug 21, 2018, 5:17:16 AM8/21/18
to tesser...@googlegroups.com
Thank you for your response.  The command that I posted previously was for fine tuning, so I was hoping to figure out what was causing the changes in accuracy when changes to the parameters should have no effect at all.j

I'm not sure if we will move forward with replacing the layers, but I will post about it if we do.

Thanks again!

On Tue, Aug 21, 2018 at 6:08 PM, Shree Devi Kumar <shree...@gmail.com> wrote:
>lstmtraining --model_output ./layer_from_deva/layer --continue_from ./layer_from_deva/Devanagari.lstm --append_index 5 --net_spec '[Lfx192 O1c1]' --traineddata ./sanplustrain/san/san.traineddata --train_listfile ./sanplustrain/san.training_files.txt --eval_listfile ./sanpluseval/san.training_files.txt --debug_interval 0 --max_image_MB 6000 --max_iterations 500
Loaded file ./layer_from_deva/Devanagari.lstm, unpacking...
Warning: LSTMTrainer deserialized an LSTMRecognizer!
Continuing from ./layer_from_deva/Devanagari.lstm
Appending a new network to an old one!!Warning: given outputs 1 not equal to unicharset of 149.
Num outputs,weights in Series:
  Lfx192:192, 197376
  Fc149:149, 28757
Total weights = 226133
Built network:[1,48,0,1[C3,3Ft16]Mp3,3Lfys64Lfx64Lrx64Lfx192Fc149] from request [Lfx192 O1c1]
Training parameters:
  Debug interval = 0, weights = 0.1, learning rate = 0.001, momentum=0.5


I  have not tried changing the parameters even with replace layers. Do provide feedback on your experience,


On Tue, Aug 21, 2018 at 12:00 PM Jacob Biros <j.b...@churadata.okinawa> wrote:
Thank you for the prompt response!
On Tue, Aug 21, 2018 at 1:14 PM, Shree Devi Kumar <shree...@gmail.com> wrote:
If you want to change parameters, please look at the replace layers option. With fine tuning you cannot change them.

On Tue 21 Aug, 2018, 7:27 AM , <j.b...@churadata.okinawa> wrote:
Is it possible to change the parameters when Fine Tuning?

The documentation says "Fine tuning is the process of training an existing model on new data without changing any part of the network", but does that mean that parameters like momentum, learning rate, etc.  cannot be changed?

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-ocr+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "tesseract-ocr" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tesseract-ocr/hjZizqQm2wA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tesseract-ocr+unsubscribe@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-ocr+unsubscribe@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/CAGO_jRmrNJ%2BYG0E_xuKhhs5DFdvamKWL3NdZLNkLcjxzxCt4pQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--

____________________________________________________________
भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

--
You received this message because you are subscribed to a topic in the Google Groups "tesseract-ocr" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tesseract-ocr/hjZizqQm2wA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tesseract-ocr+unsubscribe@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/CAG2NduUyeO%2BsoKOBLje9j3%2B7cCDmvmTeLf%2BmXVEQx_hyhRMf0w%40mail.gmail.com.

Shree Devi Kumar

unread,
Aug 21, 2018, 5:44:20 AM8/21/18
to tesser...@googlegroups.com
When you specify the complete network spec as in 

--net_spec '[1,36,0,1 Ct3,3,16 Mp3,3 Lfys48 Lfx96 Lrx96 Lfx256 O1c111]' \  

It probably treats it as a training from scratch and ignores the continue_from.

I haven't looked at how the training command is parsed. I just followed Ray's examples for the different kinds of training.

example: from scratch

lstmtraining \
  --debug_interval -1 \
  --traineddata ~/tesstutorial/engtrain/eng/eng.traineddata \
  --net_spec '[1,36,0,1 Ct3,3,16 Mp3,3 Lfys48 Lfx96 Lrx96 Lfx192 O1c111]' \
  --model_output ~/tesstutorial/engoutput/base --learning_rate 20e-4 \
  --train_listfile ~/tesstutorial/engtrain/eng.training_files.txt \
  --eval_listfile ~/tesstutorial/engeval/eng.training_files.txt \
  --max_iterations 5000

example: finetune for impact

lstmtraining \
  --model_output ~/tesstutorial/impact_from_full/impact \
  --continue_from ~/tesstutorial/impact_from_full/eng.lstm \
  --traineddata ../tessdata_best/eng.traineddata \
  --train_listfile ~/tesstutorial/engeval/eng.training_files.txt \
  --max_iterations 400

example: finetune plusminus

lstmtraining \
  --model_output ./plus_from_deva/sanplustrain \
  --continue_from ./plus_from_deva/Devanagari.lstm \
  --old_traineddata ../tessdata_best/script/Devanagari.traineddata \
  --traineddata ./sanplustrain/san/san.traineddata \
  --train_listfile ./sanplustrain/san.training_files.txt \
  --debug_interval 0 \
  --max_image_MB 6000 \
  --max_iterations $num_iterations

example: replace top layer

lstmtraining \
  --model_output ./layer_from_deva/layer \
  --continue_from ./layer_from_deva/Devanagari.lstm \
  --append_index 5 --net_spec '[Lfx192 O1c1]' \
  --traineddata ./sanplustrain/san/san.traineddata \
  --train_listfile ./sanplustrain/san.training_files.txt \
  --eval_listfile ./sanpluseval/san.training_files.txt \
  --debug_interval 0 \
  --max_image_MB 6000 \
  --max_iterations $num_iterations


Jacob Biros

unread,
Aug 21, 2018, 5:57:15 AM8/21/18
to tesser...@googlegroups.com
Oh, that thought hadn't even occurred to me.  I will look into it!

Thank you!

--
You received this message because you are subscribed to a topic in the Google Groups "tesseract-ocr" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tesseract-ocr/hjZizqQm2wA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tesseract-ocr+unsubscribe@googlegroups.com.
To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/CAG2NduWn9Q_h97cp2q7HRdN-a6u%3DFiGGeH2oAmykywfbEGPUew%40mail.gmail.com.

Shree Devi Kumar

unread,
Aug 21, 2018, 5:59:06 AM8/21/18
to tesser...@googlegroups.com
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-oc...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "tesseract-ocr" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tesseract-ocr/hjZizqQm2wA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tesseract-oc...@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-oc...@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.


--

____________________________________________________________
भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

--
You received this message because you are subscribed to a topic in the Google Groups "tesseract-ocr" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tesseract-ocr/hjZizqQm2wA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tesseract-oc...@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-oc...@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.

For more options, visit https://groups.google.com/d/optout.

Shree Devi Kumar

unread,
Aug 21, 2018, 6:01:49 AM8/21/18
to tesser...@googlegroups.com
On Tue, Aug 21, 2018 at 1:16 PM <j.b...@churadata.okinawa> wrote:
Sorry, one more question.  We set up 4 different machines all running the command below except for minor differences in the momentum and the learning rate.  Changing the momentum and learning rate in this situation, because it is fine tuning, shouldn't affect anything right?  In our case though each machine produced different results.  Do you have any idea what exactly is causing this?  I can provide more information as necessary.  Thanks.

training/lstmtraining --traineddata ~/tesstutorial/jpntrain/jpn/jpn.traineddata \
  --continue_from ~/fine_tuning/models/jpn.lstm \
  --net_spec '[1,36,0,1 Ct3,3,16 Mp3,3 Lfys48 Lfx96 Lrx96 Lfx256 O1c111]' \
  --model_output ~/bouch_fine_tuning/0819_fine20fonts_mom05_lr1e-4_010/base --learning_rate 1e-4 \
  --momentum 0.5 \
  --train_listfile ~/bouch_train/jpntrain/jpn.training_files.txt \
  --eval_listfile ~/bouch_train/jpneval/jpn.training_files.txt \
  --max_iterations 100000 &>~/bouch_fine_tuning/0819_fine20fonts_mom05_lr1e-4_010/basetrain.log \
  --old_traineddata /usr/local/share/tessdata/__official_jpn.traineddata \
  --debug_interval -1

The above is NOT a finetuning command, since you are providing the complete network spec.

With finetuning for impact (new font), recommended iterations is 400.

With finetuning for plusminus (adding a new character) recommnded iterations is 3000-3600.

However, these required iterations numbers as well as Ray's tutorial is for English.

I have found that these do not directly apply to other
 
languages which require recoding of the unicharset.

You will get quicker results if you replace top layer (compared to your earlier version which might have started from scratch).

Yo can try the different commands with --debug_interval -1 that will show you the debug output on console itself, giving you an idea of the training. eg.

File /tmp/tmp.o98cvEGUNe/akk/akk.CuneiformOB.exp-1.lstmf page 1 (Perfect):
Mean rms=0.167%, delta=0.772%, train=2.703%(4.359%), skip ratio=0.2%
Iteration 600506: ALIGNED TRUTH : 𒀀𒈾 𒉺𒉌𒅀
Iteration 600506: BEST OCR TEXT : 𒀀𒈾 𒉺𒉌𒅀

With finetuning, iteration 1 should start with a very low error rate.

For training from scratch it may be even 400% error rate.

For replacing a layer it may start around 150% error rate and come down to 100% after about 600 iterations,


Reply all
Reply to author
Forward
0 new messages