Re: [tesseract-ocr] Documentation related to lang data

[Shree Devi Kumar]

desired_characters

This is used by Google internally when creating the training text.

Should I enter all those compound character combinations to this file?

No, since this is not used by tesstrain.sh - at least in the open source version in Github.

okfonts.txt

This lists the Unicode fonts used for the LSTM training.

Can I include non Unicode fonts into this file?

NO. Because the rendered text will be incorrect.

sin.numbers

This file include all the number characters used in Sinhala.

Unless something is changed in Google's internal training method, this should NOT have the number characters. Rather it should have patterns of how numbers maybe formatted when used in this language. It may help to look at the eng.numbers file for reference.

sin.punc

In lang data this contains punctuation combinations.

Similar to the numbers file, this should have patterns of punctuation characters used in the language. Again, refer to eng.punc for reference.

sin.singles_text

Similar file to wordlist. Contains unique words followed by a new line

In Devanagari it also has unique/rare syllables (compound character combinations). Without having the scripts used by Ray (Google) for training, it is difficult to say how this is used. I am guessing that these are used in a addition to training_text to build the unicharset.

sin.training_text

The training_text in langdata_lstm seems to be random words, numbers and phrases (based on English and Devanagari). So this maybe based on word frequencies in language. While Ray's notes on training say to use text that is representative of the language or text to be recognized, the training_text does not seem to be full sentences. It's possible that this kind of training_text gives better results with LSTM for recognizing text/words not seen before. I do not really know.

sin.unicharset

This file will be created when creating training data

Yes, please check the sin.lstm-unicharset in the sin.traineddata files to check that all required characters are there.

sin.wordlist

Contains unique words followed by a new line

This dictionary as well as punc and numbers are used to create dawg files which are stored in traineddata files and provide some improvement in recognition.

-------------------------

What you could do is create a file with all valid characters and syllables ( compound character combinations) for Sinhala. Then use this file as input and grep the sin.training_text in langdata_lstm to mke sure that all combos are included in your training text for fine tuning.

On Sat, Sep 15, 2018 at 7:43 PM, Shandigutt <strid...@gmail.com> wrote:

Hi,

I downloaded latest lstm langdata from tesseract repository. I found it consists of a lot of false data for Sinhala. I'm trying to train tesseract for Sinhala. According to tesseract wiki guidelines, we need to create lang data before creating training data using tesstrain.sh script. I'm referring to the below wiki guidelines,

https://github.com/tesseract-ocr/tesseract/wiki/TrainingTesseract-4.00

I couldn't find proper wiki guidelines on creating lang data. When I inspected the 'sin' folder in langdata-lstm I found the below files,

• desired_characters
  
• okfonts.txt
  
• sin.numbers
  
• sin.punc
  
• sin.singles_text
  
• sin.training_text
  
• sin.unicharset
  
• sin.wordlist
  

Please let me know if there's a proper documentation that I can follow if I create these files on my own from the scratch. According to my observations I have the following idea of these files. If there's no any proper documentation of them please correct me if I mention anything wrong here,

desired_characters

This file contains all the unique characters found in the language. Each character followed by new line. My question is Sinhala language has many vowel characters that create compound characters with Sinhala consonants. Unlike English once a vowel character is attached to a consonant it creates a single compound character most of the time which I can erase from a single keyboard backspace. Please refer to the below example,

Example 1:

Consonant : ද

Vowel character : ො

Compound character : දො

Example 2:

Consonent : බ

Vowel character : ්

Compound character : බ්

So each consonant + different vowel characters it makes a lot of compound characters. Should I enter all those compound character combinations to this file?

okfonts.txt

This file includes the fonts I use in my training_text. Format is font name followed by a new line. Can I include non Unicode fonts into this file?

sin.numbers

This file include all the number characters used in Sinhala. Number character followed by a new line. Normally this contains only 10 characters

sin.punc

This character contains all the punctuation characters that can be used in Sinhala text. Format is punctuation character followed by a new line. In lang data this contains punctuation combinations. Please explain why?

sin.singles_text

Similar file to wordlist. Contains unique words followed by a new line

sin.training_text

Training text to be used when creating training data. Should contain around 40000 text lines. Each line can have any amount of characters. It’s better if this document contains text in multiple fonts that we have defined in okfonts.txt. (These fonts can be passed as a command line argument as well)

sin.unicharset

This file will be created when creating training data

sin.wordlist

Contains unique words followed by a new line

Appreciate your response on this.

Thanks

--
You received this message because you are subscribed to the Google Groups "tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-oc...@googlegroups.com.
To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/9e8be0bf-b0d5-4408-98b7-283913ccf642%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

____________________________________________________________
भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com up

[Pubudu Tharaka Viswakula]

Hi Shree,

Thank you very much for improving my awareness. I have one more question,

When we create training files it creates the traineddata file. It's better to include word-dwag, punc-dwag and number-dwag into it. Are they created using the same files we're talking as sin.numbers, sin.punc and sin.wordlist? As you explained me in an earlier questions they gets included into traineddata file when we use tesstrain.sh right?

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/CAG2NduVwUEM4SmsO8nSVwB76wsmdbzynwcK8-30_cDnEawW2Gg%40mail.gmail.com.

[Shree Devi Kumar]

>Are they created using the same files we're talking as sin.numbers, sin.punc and sin.wordlist? 

Yes, the dawg files are created from these and the unicharset. The same unicharset should be used for lstm training. 

To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/CAKOih%3DkGPrkU%2BCm1KFooKyUknQjR0i8zxf6ZrHzZkLg9vwNjfA%40mail.gmail.com.

[Pubudu Tharaka Viswakula]

Thank you very much Shree

To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/CAG2NduU58m%3DYybzA-sEMbEp80Zzke%3DYKK4YfKAG34zpaFy2Xww%40mail.gmail.com.

[Shree Devi Kumar]

cat ~/sin.syllables_text | while IFS="

" read target; do grep -F -m 7 "$target"  ~/langdata_lstm/sin/sin.training_text  ; done > tmp.txt

sort -u  tmp.txt  >  ~/sin.sample7.training_text

The above will create a training text with min 7 samples for each line in syllables_text. Change -m 7 to -m 1 to create file with just one sample of each. Sort unique removes duplicate lines.

This can be used to create a smaller training_text useful for finetuning.

To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-ocr+unsubscribe@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/9e8be0bf-b0d5-4408-98b7-283913ccf642%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

____________________________________________________________
भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com up

[Shandigutt]

Thank you very much for your kind help 

To unsubscribe from this group and stop receiving emails from it, send an email to tesseract-oc...@googlegroups.com.

To post to this group, send email to tesser...@googlegroups.com.
Visit this group at https://groups.google.com/group/tesseract-ocr.
To view this discussion on the web visit https://groups.google.com/d/msgid/tesseract-ocr/9e8be0bf-b0d5-4408-98b7-283913ccf642%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

____________________________________________________________
भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com up