Re: Regarding Russian dictionary

[Linas Vepstas]

Hi,

I'mm cc'ing the mailing list, as perhaps there are other windows experts who have experience with wide characters?

On 1 April 2013 05:34, Alexey Andreev <lex.a...@gmail.com> wrote:

Hi Linas,

I am writing for Alex's request regarding the Russian dictionary. As Alex mentioned in previous emails, we got stuck with a bug in the Russian dictionary under the Windows platform. I revealed that the key problem was the file_get_content method. A wchar under Windows is represented by 16 bits, and the same value we get for Cyrillics symbols in the UTF-8 encoding. Unfortunately, for some reasons every single Russian letter in Linkgrammar was represented by two wchar. For instance,
the letter 'Р': wchar1 - 1056 (which is "Р" itself) and another one 160 (which was treated as a whitespace)
the same result I got for the letter "А":  1056 again (which is "Р" itself) and another wchar.  

That's .. crazy ... when you read 'A' you get 'P'?  This is not a typo in your email, but this is actually happening?

 

Normally, I should have gotten one wchar per Cyrillics letter. 

Yes, that is what I would expect.

 

Having digged deeper inside, I found this in MSDN:

fgetws reads the wide-character argument str as a multibyte-character string or a wide-character string according to whether stream is opened in text mode or binary mode, respectively. 

And as a result in our case, that multibyte char conversion brought the issue. I tried to read the dictionary in a binary mode, yet it gave me a completely different result.  I made the code read the Russian dictionary correctly by replacing  "FILE *fp = dictopen(dict_name, "r")" to FILE *fp = dictopen(dict_name, "r, ccs=UTF-8"). After these changes the Russian dictionary was read smoothly. I got exactly one wchar per Russian letter.

Yay! 

Unfortunately, that was not the end. Now after these changes, I have a problem with the English dictionary!!!

 

Boo!

 

The source of the new issue is the following. The English dictionary contains a row of special symbols like this : ½ ⅓ ⅔ ¼ (fractions) and some others like the degree sign. After my changes have been made, this special symbols are read as one wchar, but in the original version they were read as several wchars (multibyte char conversion), and int nr = wcrtomb(s, wc, ps) returns -1, not converting single wchars like that : ½. I tried setting different locals, but nothing helped. 

Any ideas about that problem will be highly appreciated!

The link parser currently reads the dictionaries using fgetwc but perhaps it does not need to.  There are only two or three places where fgetwc is used; these could be replaced with fgetc(), although this would require a loop to read multiple bytes (i.e. to keep reading until char & 0x80 is false, and then fungetc that char).   These conversions are more complicated than what I can do in this email, but perhaps you could try them?

(There are other conversions to wide chars, in other places, to test for upper-case characters, etc.  These should be left alone, I believe. For example, a sentence that starts with an upper-case letter should still parse correctly.).

If this is too complicated,  I could try this myself later, but I'm rather busy this week.

-- Linas