Traineddata distorted and provides bad read, last trained sample is as usual

[Mitya]

I've trained 8000 samples with set of commands below:

echo "~/source/source.lstmf" > /home/j/trainingCurrentEng/data/list.eval
echo "~/source/source.lstmf" > /home/j/trainingCurrentEng/data/list.train


lstmtraining   --continue_from /home/j/trainingCurrentEng/data/checkpoints/eng_trained_checkpoint --traineddata /home/j/trainingCurrentEng/data/eng.traineddata --traineddata /home/j/trainingCurrentEng/data/eng.traineddata --train_listfile /home/j/trainingCurrentEng/data/list.train --eval_listfile /home/j/trainingCurrentEng/data/list.eval --model_output /home/j/trainingCurrentEng/data/checkpoints/eng_trained --learning_rate 0.001 --debug_interval 10 --max_iterations 8000000





lstmtraining --stop_training   --continue_from /home/j/trainingCurrentEng/data/checkpoints/eng_trained_checkpoint   --traineddata /home/j/trainingCurrentEng/data/eng.traineddata   --model_output /home/j/trainingCurrentEng/data/eng_trained.traineddata

IMPORTANT/RESULT:
 tesseract source.tiff output_text -l eng --tessdata-dir /home/j/trainingCurrentEng/data --psm 7
 cat output_text.txt

abcdef

 tesseract source.tiff output_text_1 -l eng_trained --tessdata-dir /home/j/trainingCurrentEng/data --psm 7
 cat output_text_1.txt

laldlfk

Question:
Syntax one looks better, but after 8000 results I got Tesseract eng_trained model distorted, so it reads completely wrong
But If you read THE LAST sample trained/updated eng_trained, it reads this exact data flawlessly

What am I doing wrong? How to fix current syntax?

IMPORTANT: I use images in same color pallete: black background white(close to gray) font, without any masks applied.



j@j-Aspire-A515-58M:~/source$ ls
source.box  source.lstmf  source.tiff  source.txt  unicharset


Source Box:

a 251 52 355 178 0
b 356 51 444 176 0
c 446 22 530 175 0
d 534 22 622 173 0
e 626 60 766 174 0
f 768 59 870 173 0

source.txt
abcdef

unicharset
9
NULL 0 Common 0
Joined 7 0,255,0,255,0,0,0,0,0,0 Latin 1 0 1 Joined # Joined [4a 6f 69 6e 65 64 ]a
|Broken|0|1 15 0,255,0,255,0,0,0,0,0,0 Common 2 10 2 |Broken|0|1    # Broken
a 3 0,255,0,255,0,0,0,0,0,0 Latin 3 0 3 a   # a [6f ]a
b 3 0,255,0,255,0,0,0,0,0,0 Latin 4 0 4 b   # b [65 ]a
c 3 0,255,0,255,0,0,0,0,0,0 Latin 5 0 5 c   # c [64 ]a
d 3 0,255,0,255,0,0,0,0,0,0 Latin 6 0 6 d   # d [6b ]a
e 3 0,255,0,255,0,0,0,0,0,0 Latin 7 0 7 e   # e [6d ]a
f 3 0,255,0,255,0,0,0,0,0,0 Latin 8 0 8 f   # f [63 ]a

[Mitya]

Summary:
I decided to train one source image  (without any filters), but still getting major issue, assumable with set of commands to train model or (Highly Likely) in area where we update eng.trainedadata or interfere with checkpoints!

Could you please take a look?

Best Regards,
Mitya

[Ger Hobbelt]

Haven't checked your info further, but note your remark:

IMPORTANT: I use images in same color pallete: black background white(close to gray) font, without any masks applied.

Please, do NOT train with inverted imagery like that (white on black), particularly when you are working with existing models, as those have been trained to deal with book sources (black printed text on white paper) and feeding such a model with inverted images and forcing it to learn those too can only lead to total model confusion and consequently headaches and "weird, inexplicable results".

Yes, when you dig deep ("rtfc") you'll find tesseract carries a bit of code to detect white-on-black inverted image inputs and invert those for you before feeding them to the core engine, but forget about that bit as it is only triggered under a set of very particular circumstances and (AFAIR) never in a training scenario.

TL;DR: any and all training is best done based on black/dark text on white/light background as both training images and ocr-ing (processing) images' code flow *implicitly* assumes this type of input.

 (When you use tesseract for a longer while, you will discover that feeding it white-on-black works just well enough to give you the idea that this might fly, but "weird shit" keeps happening in your decoded outputs and the hassle never goes away whatever you try, until you adjust your preprocess to always pump out black-on-white, guaranteed, and that "sometimes it's plain weird!" stuff ... just goes away. There's technical explanations for this, surely, but way too many ifs and buts there for easy comprehension and a simple story.)

If, for instance, you plan to train and use tesseract for screen reader / subtitle action, where often light text occurs on black backgrounds, the above statement implies that your customized process MUST *invert* all source images, both in the training and the using/decoding paths, as the tesseract core is meant to receive black text on white BG, always, for optimal results.

In your case, may I suggest re-running all what you did, but with inverted source, i.e. all your training images turned into black text on top of white background? I expect this will deliver fewer "weird results" versus what you currently experience.

Take care,

Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   g...@hobbelt.com
mobile: +31-6-11 120 978
--------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-oc...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/tesseract-ocr/260866d4-8131-4b62-86a3-e9bb88d18187n%40googlegroups.com.