Libthai word breaking
John Tapsell
I have a bug report that the string: ม.ค. shouldn't have word
break in the middle. But libthai places one in the middle like:
ม.-ค.
Where the "-" indicates a word break.
Is this a bug in libthai? Can it be fixed? Can I fix it in our
copy of libthai?
John
Theppitak Karoonboonyanan
Yes, it's a known bug. I've just been too busy with life and other
projects to fix it. Patches are welcome.
Regards,
--
Theppitak Karoonboonyanan
http://linux.thai.net/~thep/
Pattara Kiatisevi (List)
>> break in the middle. But libthai places one in the middle like:
>>
>>
>> Where the "-" indicates a word break.
>>
>> Is this a bug in libthai? Can it be fixed? Can I fix it in our
>> copy of libthai?
> Yes, it's a known bug. I've just been too busy with life and other
> projects to fix it. Patches are welcome.
Adding
๏ฟฝ.๏ฟฝ.
๏ฟฝ.๏ฟฝ.
๏ฟฝ๏ฟฝ.๏ฟฝ.
๏ฟฝ๏ฟฝ.๏ฟฝ.
๏ฟฝ.๏ฟฝ.
๏ฟฝ๏ฟฝ.๏ฟฝ.
๏ฟฝ.๏ฟฝ.
๏ฟฝ.๏ฟฝ.
๏ฟฝ.๏ฟฝ.
๏ฟฝ.๏ฟฝ.
๏ฟฝ.๏ฟฝ.
๏ฟฝ.๏ฟฝ.
into libthai's dictionary should solve the problem?
Ott
>
> Regards,
Pruet Boonma
Pruet
-----Original Message-----
From: "Pattara Kiatisevi (List)" <pattar...@mm.co.th>
Sender: thai-linux...@googlegroups.com
Date: Thu, 26 Jan 2012 11:46:29
To: <thai-linux...@googlegroups.com>
Reply-To: thai-linux...@googlegroups.com
Subject: Re: [thai-linux-foss-devel] Libthai word breaking
On 26/1/2555 7:31, Theppitak Karoonboonyanan wrote:
> 2012/1/25 John Tapsell<john...@gmail.com>:
>
>> break in the middle. But libthai places one in the middle like:
>>
>>
>> Where the "-" indicates a word break.
>>
>> Is this a bug in libthai? Can it be fixed? Can I fix it in our
>> copy of libthai?
> Yes, it's a known bug. I've just been too busy with life and other
> projects to fix it. Patches are welcome.
Adding
ม.ค.
ก.พ.
มี.ค.
เม.ย.
พ.ค.
มิ.ย.
ก.ค.
ส.ค.
ก.ย.
ต.ค.
พ.ย.
ธ.ค.
into libthai's dictionary should solve the problem?
Ott
>
> Regards,
--
คุณได้รับข้อความนี้เนื่องจากคุณสมัครรับข้อมูลจากกลุ่ม "Thai Linux/FOSS developers" ของ Google Groups
หากต้องการโพสต์ถึงกลุ่มนี้ ให้ส่งอีเมลไปที่ thai-linux...@googlegroups.com
หากต้องการยกเลิกการสมัครสำหรับกลุ่มนี้ โปรดส่งอีเมลไปที่ thai-linux-foss-...@googlegroups.com
สำหรับตัวเลือกเพิ่มเติม โปรดเข้าสู่กลุ่มนี้ที่ http://groups.google.com/group/thai-linux-foss-devel?hl=th
Theppitak Karoonboonyanan
> Adding
>
> ม.ค.
> ก.พ.
> มี.ค.
> เม.ย.
> พ.ค.
> มิ.ย.
> ก.ค.
> ส.ค.
> ก.ย.
> ต.ค.
> พ.ย.
> ธ.ค.
>
No. It involves changing the pre-tokenization rules before the
dictionary lookup stage.
John Tapsell
> No. It involves changing the pre-tokenization rules before the
> dictionary lookup stage.
I had a quick look at this last night, and found the function where it
tokenizes the string but couldn't see an obvious fix. I could maybe
add a specific case tokenizing if it's just "one thai character, full
stop, one thai character, full stop". Would that be the right way to
do it?
John
Theppitak Karoonboonyanan
There can be more than one character in Thai abbreviations,
such as "กศ.บ.".
Probably, some action needs to be modified. You can see some
note I took in a blog when designing it (in Thai, sorry):
http://thep.blogspot.com/2007/06/libthai-thbrk-redesign.html
Regards,
--
Theppitak Karoonboonyanan
http://linux.thai.net/~thep/
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d(+) s:- a C++ UL P-@ L+++ E--- W+++ N o? K? w-- !O !M V? PS(+)
PE-- Y(-) PGP++ t? 5? X+ R- tv+ b++ DI D-- G e++ h-- r? y?
------END GEEK CODE BLOCK------
John Tapsell
> 2012/1/25 John Tapsell <john...@gmail.com>:
>
>> I have a bug report that the string: ม.ค. shouldn't have word
>> break in the middle. But libthai places one in the middle like:
>>
>> ม.-ค.
>>
>> Where the "-" indicates a word break.
>>
>> Is this a bug in libthai? Can it be fixed? Can I fix it in our
>> copy of libthai?
>
> Yes, it's a known bug. I've just been too busy with life and other
> projects to fix it. Patches are welcome.
For quick testing, I modified the code to treat a "." as part of the
thai, if the preceeding text is thai.
So below are some strings that are passed to brk_maximal_do and the
"-" indicates where it suggests word breaks:
-ม.ค.
-มธ.-หาม-ใช-พนท-เคลอนไหว
-ครม.เหต-มะ-กน-ขน-แบ-ลก-ลสต
This last one looks strange - any idea why brk_maximal_do would not
place a break after the "." ?
John
John Tapsell
Handle the full stop better.
To handle a word like: "ม.ค." we treat the "." preceeded by thai as
thai when tokenizing, so that we pass the whole word to brk_maximal_do
To handle a word like "ม.112" we keep the effective_class of the "."
untouched, so that "ม." is passed to the brk_maximal_do but ".1" follows
the op rules which says that this cannot be broken. Thus the whole
string is treated as unbreakable.
To handle a word like "ครม.เหตมะกนขนแบลกลสต" where neither "ครม." nor
"เหต" are in the libthai dictionary we force a break after a period when
trying to recover in brk-maximal.c
This is not perfect behaviour - for example พ.ร.ก.กเงนของวฒฯถงศาล gets
broken like: -พ.ร.-ก.-ก-เงน-ของ-วฒฯ-ถง-ศาล but it's hard to see how to
improve this nicely.
John
John Tapsell
Add a list of common acronyms with multiple fullstops to the dictionary,
Also add the fullstop to the dictionary unicode range
We don't add acryonyms with a single fullstop because we already break
at the first fullstop when we don't recognise a word, and because in
testing it caused more problems than it fixed.
John Tapsell
The first patch adds support for "." but feels like a bit of hack on
top, rather than using the classes idea that everything else uses. I
did try various different ways, but the behaviour of the fullstop was
a bit different from everything else. This change was the cleanest
way that I could find.
The second patch adds in common acronyms with multiple fullstops.
This does not increase the dictionary size significantly - it
increases the dictionary file size by 1.3%.
I have a thai tester for these patches and will get back to you about
the full testing results.
I also have patches for the Qt library and harfbuzz to actually pass
the "." to libthai. I am currently getting these merged in.
John
Theppitak Karoonboonyanan
> Just a quick comment about the two patches.
Thanks for your patches.
> The first patch adds support for "." but feels like a bit of hack on
> top, rather than using the classes idea that everything else uses. I
> did try various different ways, but the behaviour of the fullstop was
> a bit different from everything else. This change was the cleanest
> way that I could find.
>
> The second patch adds in common acronyms with multiple fullstops.
> This does not increase the dictionary size significantly - it
> increases the dictionary file size by 1.3%.
Now I'm at a dilemma: the abbreviation list can grow indefinitely
when new entities are created, such as the recent "ศปภ." for
Flood Relieve Operation Center (FROC), and people can also
create arbitrary abbreviations at whim, such as "ส.บ.ม."
(originally slang for "สบายมาก", but now used in a commercial
for "สุขแบ่งมิตร", along with many other new interpretations).
We have two choices: maintain the list or cover it all with pure
matching rules.
I tend to prefer the latter, but I'm also aware of the side effect
on some existing written styles in which English punctuations
(comma, full stop, etc.) are used in Thai sentences.
We've got to be careful when drafting the rules.
> I have a thai tester for these patches and will get back to you about
> the full testing results.
Oh, that's good!
> I also have patches for the Qt library and harfbuzz to actually pass
> the "." to libthai. I am currently getting these merged in.
OK. Thanks for working on this.
John Tapsell
> when new entities are created, such as the recent "ศปภ." for
> Flood Relieve Operation Center (FROC), and people can also
> create arbitrary abbreviations at whim, such as "ส.บ.ม."
> (originally slang for "สบายมาก", but now used in a commercial
> for "สุขแบ่งมิตร", along with many other new interpretations).
I agree, but this is true for normal words too. For example straight
away my thai friend noticed that "คอล" is not in the dictionary
either, being a transliteration. Do we include all possible
transliterations?
As a fix, I'm thinking of treating anything in the format "<1 or 2
thai letters>.<1 or 2 thai letters>.<etc>" as a single word. i.e.
modify that recovery patch that I wrote to find the last "." to
recover on, skipping over a maximum of 2 thai letters at a time. This
should cover most acronyms nicely.
Testing quickly, that pattern (i.e. ^[^.]*[.]\([^.][^.]\?[.]\)*$ )
matches 434 out of the 471 acronyms in that acronyms list in my patch.
This might be good-enough (TM) without getting any false positives
afaics.
On another note, I compiled a new word list against lexitron database.
After sanitizing the database and getting it into the format that we
use, I found that it would at least double, and possible triple, the
size of our word database. I need to iron out some bugs first, but
just something to think about.
John
Theppitak Karoonboonyanan
>> Now I'm at a dilemma: the abbreviation list can grow indefinitely
>> when new entities are created, such as the recent "ศปภ." for
>> Flood Relieve Operation Center (FROC), and people can also
>> create arbitrary abbreviations at whim, such as "ส.บ.ม."
>> (originally slang for "สบายมาก", but now used in a commercial
>> for "สุขแบ่งมิตร", along with many other new interpretations).
>
> I agree, but this is true for normal words too. For example straight
> away my thai friend noticed that "คอล" is not in the dictionary
> either, being a transliteration. Do we include all possible
> transliterations?
I'm more happy to maintain the normal word list than to do the
abbreviation list, as abbreviations are more arbitrary than words.
New words are linguistic phenomena, while abbreviations emerge
at least on every committee created. Arbitrary abbreviations are
also overwhelming.
Regarding the transliterations, I only add words that are common
enough. For lower-frequency entries, we can rely on the unknown
word recovery.
> As a fix, I'm thinking of treating anything in the format "<1 or 2
> thai letters>.<1 or 2 thai letters>.<etc>" as a single word. i.e.
> modify that recovery patch that I wrote to find the last "." to
> recover on, skipping over a maximum of 2 thai letters at a time. This
> should cover most acronyms nicely.
Isn't it more efficient to try to do it in th_brk() layer and to not pass it
to brk_maximal_do() at all? The recovery algorithm is quite costly and
should be avoided if possible.
> Testing quickly, that pattern (i.e. ^[^.]*[.]\([^.][^.]\?[.]\)*$ )
> matches 434 out of the 471 acronyms in that acronyms list in my patch.
> This might be good-enough (TM) without getting any false positives
> afaics.
The pattern may cover up to 3 letters, for some common abbreviations
like "กอ.รมน.". And I think that should not cause false positive, either.
> On another note, I compiled a new word list against lexitron database.
> After sanitizing the database and getting it into the format that we
> use, I found that it would at least double, and possible triple, the
> size of our word database. I need to iron out some bugs first, but
> just something to think about.
Lexitron is a Thai-English dictionary. The problem it tries to solve is
translation, not word breaking. So, some compound words or even
phrases may be included. It may not be optimal for line wrapping.
John Tapsell
> enough. For lower-frequency entries, we can rely on the unknown
> word recovery.
Which is pretty much what I'm suggesting for acronyms, combined with a
slightly improved acronym matcher.
Speaking of word recovery, I was thinking of tweaking it slightly.
For example if you have an unknown three letter, I don't think it's
correct to split this into a one-letter unknown word and a two-letter
known word. For example it currently splits คอล into คอ and ล.
I'll have to think about this some more.
>> As a fix, I'm thinking of treating anything in the format "<1 or 2
>> thai letters>.<1 or 2 thai letters>.<etc>" as a single word. i.e.
>> modify that recovery patch that I wrote to find the last "." to
>> recover on, skipping over a maximum of 2 thai letters at a time. This
>> should cover most acronyms nicely.
>
> Isn't it more efficient to try to do it in th_brk() layer and to not pass it
> to brk_maximal_do() at all? The recovery algorithm is quite costly and
> should be avoided if possible.
I don't really know what you mean - bear in mind I'm new to this code
:-) But please bear in mind that there's no guaranteed pattern to the
abbreviations. They don't need to end with a "."
For example:
กทม.กรงเทพมหานคร
กรป.กลาง
กพ.ทหาร
พ.อ.พเศษ
Let me know if you think that these are incorrect or should be treated
as two words.
John
Theppitak Karoonboonyanan
>> Regarding the transliterations, I only add words that are common
>> enough. For lower-frequency entries, we can rely on the unknown
>> word recovery.
>
> Which is pretty much what I'm suggesting for acronyms, combined with a
> slightly improved acronym matcher.
Right, but acronyms are more arbitrary, while the pattern is
relatively obvious to match without the aids of external
automaton.
If we can catch them beforehand, the cost of the recovery algorithm
can also be saved.
> Speaking of word recovery, I was thinking of tweaking it slightly.
> For example if you have an unknown three letter, I don't think it's
> correct to split this into a one-letter unknown word and a two-letter
> known word. For example it currently splits คอล into คอ and ล.
> I'll have to think about this some more.
There will be scoring scheme, with penalty on unknown word
occurrences, to cope with this when choosing the best answer.
(See best_brk_contest(), for example, along with scoring in
brk_maximal_do_impl() loop.)
>>> As a fix, I'm thinking of treating anything in the format "<1 or 2
>>> thai letters>.<1 or 2 thai letters>.<etc>" as a single word. i.e.
>>> modify that recovery patch that I wrote to find the last "." to
>>> recover on, skipping over a maximum of 2 thai letters at a time. This
>>> should cover most acronyms nicely.
>>
>> Isn't it more efficient to try to do it in th_brk() layer and to not pass it
>> to brk_maximal_do() at all? The recovery algorithm is quite costly and
>> should be avoided if possible.
>
> I don't really know what you mean - bear in mind I'm new to this code
> :-)
I mean, we can change the th_brk() loop to match acronyms and
treat them on the fly, just like English parts, without passing them
to the dictionary matching algorithm, which is the most costly part.
> But please bear in mind that there's no guaranteed pattern to the
> abbreviations. They don't need to end with a "."
>
> For example:
>
> กทม.กรงเทพมหานคร
This one is irrelevant. (กทม. = กรุงเทพมหานคร)
> กรป.กลาง
> กพ.ทหาร
> พ.อ.พเศษ
>
> Let me know if you think that these are incorrect or should be treated
> as two words.
Yes, these three should be treated as two separate words.
The part after the last full stop is the continuation of the
entity name, which can be considered as next words.
John Tapsell
> Right, but acronyms are more arbitrary, while the pattern is
> relatively obvious to match without the aids of external
> automaton.
I'm starting to be convinced that we can indeed use patterns to match
all the acronyms.
> If we can catch them beforehand, the cost of the recovery algorithm
> can also be saved.
Agreed.
>> Speaking of word recovery, I was thinking of tweaking it slightly.
>> For example if you have an unknown three letter, I don't think it's
>> correct to split this into a one-letter unknown word and a two-letter
>> known word. For example it currently splits คอล into คอ and ล.
>> I'll have to think about this some more.
>
> There will be scoring scheme, with penalty on unknown word
> occurrences, to cope with this when choosing the best answer.
> (See best_brk_contest(), for example, along with scoring in
> brk_maximal_do_impl() loop.)
Yes of course - this is what I was thinking. Tweak the scoring system.
>> กรป.กลาง
>> กพ.ทหาร
>> พ.อ.พเศษ
>
> Yes, these three should be treated as two separate words.
> The part after the last full stop is the continuation of the
> entity name, which can be considered as next words.
Very interesting.
Can you please comment on the rest too:
กก.ตชด.
กก.รสช.
กกล.รพน.
กทม.กรงเทพมหานคร
กพ.ทหาร
กรป.กลาง
กอ.รพน.
กอ.รมน.
กอ.รสต.
ขว.ทหาร
จ.๑๘
ทส.รมว.กห.
ธ.ญ
น.ส.๓
บก.ตชด.
บก.ภ.เขต
บก.รอย.ตชด.
บช.ตชด.
ป.กศ.สง
ป.ร.ร.๔
ป.ร.ร.๕
ป.ร.ร.๖
ป.๑
ผบ.ทสส.
ผบช.ตชด.
ผว.กทม.
พ.ต.อ.พเศษ
พ.อ.พเศษ
พล.ปตอ.
พล.ม.๒
ภ.ง.ด.๙
ภ.บ.ท.๕
ม.อ.ปตตาน
รร.จปร.
รร.นรต.
ส.ค.1
สนง.สสอ.
John
Theppitak Karoonboonyanan
> 2012/2/4 Theppitak Karoonboonyanan <th...@linux.thai.net>:
>>> Speaking of word recovery, I was thinking of tweaking it slightly.
>>> For example if you have an unknown three letter, I don't think it's
>>> correct to split this into a one-letter unknown word and a two-letter
>>> known word. For example it currently splits คอล into คอ and ล.
>>> I'll have to think about this some more.
>>
>> There will be scoring scheme, with penalty on unknown word
>> occurrences, to cope with this when choosing the best answer.
>> (See best_brk_contest(), for example, along with scoring in
>> brk_maximal_do_impl() loop.)
>
> Yes of course - this is what I was thinking. Tweak the scoring system.
Anyway, the scoring is quite a compromise between different cases.
Optimize too much on some cases and it works more poorly on others.
Let's be careful and rounded.
>>> กรป.กลาง
>>> กพ.ทหาร
>>> พ.อ.พเศษ
>>
>> Yes, these three should be treated as two separate words.
>> The part after the last full stop is the continuation of the
>> entity name, which can be considered as next words.
>
> Very interesting.
>
> Can you please comment on the rest too:
>
> กก.ตชด.
> กก.รสช.
> กอ.รมน.
> กกล.รพน.
> กอ.รพน.
> กอ.รสต.
> บก.ตชด.
> บช.ตชด.
> ผบ.ทสส.
> ผบช.ตชด.
> รร.จปร.
> รร.นรต.
> สนง.สสอ.
These are valid acronyms.
Line should be wrapped after the last period.
> กทม.กรงเทพมหานคร
I've already said this is irrelevant.
Only "กทม." is acronym, while "กรุงเทพมหานคร"
is what it stands for.
> กพ.ทหาร
> กรป.กลาง
> ขว.ทหาร
> บก.ภ.เขต
> ป.กศ.สูง
> พ.ต.อ.พิเศษ
> พ.อ.พิเศษ
> ม.อ.ปัตตานี
Acronym + continuation words.
Line can be wrapped after the last period (before the
continuation words).
> จ.๑๘
> น.ส.๓
> ป.ร.ร.๔
> ป.ร.ร.๕
> ป.ร.ร.๖
> ป.๑
> ภ.ง.ด.๙
> ภ.บ.ท.๕
> ส.ค.1
Acronym + digits, similar to ป. ๑, ม. ๑/๒.
Line can be wrapped after the last period, but considered
bad style. (In LaTeX, one may input like ป.~๑, ม.~๑/๒)
> ทส.รมว.กห.
Abbreviating 3 separate entities: ทส. and รมว. and then ก.ห.
Not commonly used in combination. (Can be dismissed)
> ธ.ญ
No idea what it is.
> บก.ร้อย.ตชด.
> พล.ปตอ.
> พล.ม.๒
With shortened words (ร้อย, พล). Can be treated like
normal acronyms.
> ผว.กทม.
2 separate entities: ผว. and กทม. (Usually delimited with space
and can be wrapped in between.)
John Tapsell
>...
Thanks. It doesn't seem possible to programmatically distinguish between say:
กอ.รมน.
which can't be word wrapped and:
ผว.กทม.
which can be word wrapped. We could put ผว. and กทม. in the dictionary.
John
Theppitak Karoonboonyanan
I'm not serious for this case. Not wrapping in between is OK.
And it's usually delimited with space (ผว. กทม.).
John Tapsell
last ".", looking four characters ahead (i.e. we can have three thai
letters between a period).
As a minor point - could you please add เป็นจริง to the dictionary.
I've been told that this should be treated as one word.
Theppitak Karoonboonyanan
> Here's another attempt at acronym detection. This time we find the
> last ".", looking four characters ahead (i.e. we can have three thai
> letters between a period).
And here's my first patch for my approach: purely handling in th_brk(),
no acronym in dictionary. (To be further adjusted after testing.)
> As a minor point - could you please add เป็นจริง to the dictionary.
> I've been told that this should be treated as one word.
In term of line wrapping, I'd rather treat this as a phrase, not word.
Otherwise, there would be too many possible entries in the dictionary,
and it would also result in unnecessarily ragged right margin in the
output.
Theppitak Karoonboonyanan
> 2012/2/6 John Tapsell <john...@gmail.com>:
>> Here's another attempt at acronym detection. This time we find the
>> last ".", looking four characters ahead (i.e. we can have three thai
>> letters between a period).
>
> And here's my first patch for my approach: purely handling in th_brk(),
> no acronym in dictionary. (To be further adjusted after testing.)
Patch version 2: jump back to the acronym end and let the usual
loop logic do the rest, so that the appropriate brk op is applied,
instead of forcing the break.
John Tapsell
> 2012/2/7 Theppitak Karoonboonyanan <th...@linux.thai.net>:
>> 2012/2/6 John Tapsell <john...@gmail.com>:
>>> Here's another attempt at acronym detection. This time we find the
>>> last ".", looking four characters ahead (i.e. we can have three thai
>>> letters between a period).
>>
>> And here's my first patch for my approach: purely handling in th_brk(),
>> no acronym in dictionary. (To be further adjusted after testing.)
>
> Patch version 2: jump back to the acronym end and let the usual
> loop logic do the rest, so that the appropriate brk op is applied,
> instead of forcing the break.
Nice - This new patch passes the tests in my test program, giving the
same results as my patch.
I still prefer my way so that we can have common acronyms in the
dictionary, but it's not so important and you're the boss. :-)
John
Theppitak Karoonboonyanan
> 2012/2/7 Theppitak Karoonboonyanan <th...@linux.thai.net>:
>> Patch version 2: jump back to the acronym end and let the usual
>> loop logic do the rest, so that the appropriate brk op is applied,
>> instead of forcing the break.
>
> Nice - This new patch passes the tests in my test program, giving the
> same results as my patch.
Thanks for testing. Committed.
> I still prefer my way so that we can have common acronyms in the
> dictionary, but it's not so important and you're the boss. :-)
Well, ... :-)