Re: [Link Grammar] switching dictionary (e.g. en, de, ru) through the Java interface

[Linas Vepstas]

On 12 March 2013 08:55, Alexander Tkachuk <alexande...@gmail.com> wrote:

Hello,

Finally managed to get LinkGrammar DLL's working in Java x64 environment, and faced with another problem...

Is there any way to switch dictionary/language through the Java interface?

Good question. Looking at the code, I see LGConfig.getDictionaryLocation()  and

LinkGrammar.setDictionariesPath(config.getDictionaryLocation());  so it looks like the answer is 'yes'.  I'm not sure how to use it.

As far as I know, we don't have any Java Hello World examples for this API.  Care to write one?

-- Linas

[Linas Vepstas]

Hi Alexander,

Care to think about and propose a proper fix?

if I look at jni.c, which implements the java bindings, I see these lines:

   /* Default to the english language; will need to fix

    * this if/when more languages are supported.

    */

   ptd->dict = dictionary_create_lang("en");

Perhaps a new call, setLanguage(), should be introduced? And then delay the dictionary open until later on, instead of doing it in init()?

-- Linas

On 13 March 2013 06:07, Alexander Tkachuk <alexande...@gmail.com> wrote:

Hi Linas,

Thank you for your quick response!

I just posted in another thread very brief tutorial describing how to build LinkGrammar x64 DLLs and how to use them in Java.

I tried "setDictionaryPath()" already and it doesn't work for switching languages because it requires path to "data" folder, but not to "data/en" folder for example. If you will try to pass something like "data/en" you will get error message saying that dictionary was not found.

It looks like that simple copying files from "ru" to "en" work (at least LinkGrammar.init() doesn't claim about missing dictionary)...

Thank you again,

Alex

--
You received this message because you are subscribed to the Google Groups "link-grammar" group.
To unsubscribe from this group and stop receiving emails from it, send an email to link-grammar...@googlegroups.com.
To post to this group, send email to link-g...@googlegroups.com.
Visit this group at http://groups.google.com/group/link-grammar?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Alexander Tkachuk]

Hi Linas,

I think setLanguage() will be perfect solution!

Though, please wait for a couple days. I will make these changes by myself and come back to you with more thoughts:)

Thank you,

Alex

[Alexander Tkachuk]

Hi Linas,

Just added following functions and methods to jni-client.c, jni-client.h and LinkGrammar.java
 - setLanguage(String) switches between en, ru, etc.
 - setLocale(String) switches between en_EN.UTF-8, ru_RU.UTF-8, etc.

And of course made corresponding changes in the init() function in jni-client.c

It works well for English dictionary (i.e. language=en and locale=en_EN.UTF-8), but it fails loading Russian one (i.e. language=ru and locale=ru_RU.UTF-8) with following error message
 - link-grammar: Error parsing dictionary /ru/stem.dict.
Expecting ";" at the end of an entry.
     line 6140, tokens = "ЕДК,0>" ";" "бессмысленно.e" "зачем.e" "здесь.e"

My first though was that it is caused by wrong locale (that is why the setLocale() method was implemented), but it is not,,,

Could you point me to any example how to switch languages using LinkGrammar directly (without Java wrap)?

Thank you,

Alex

[Linas Vepstas]

On 14 March 2013 12:12, Alexander Tkachuk <alexande...@gmail.com> wrote:

Hi Linas,

Just added following functions and methods to jni-client.c, jni-client.h and LinkGrammar.java
 - setLanguage(String) switches between en, ru, etc.
 - setLocale(String) switches between en_EN.UTF-8, ru_RU.UTF-8, etc.

And of course made corresponding changes in the init() function in jni-client.c

Excellent!

 

It works well for English dictionary (i.e. language=en and locale=en_EN.UTF-8), but it fails loading Russian one (i.e. language=ru and locale=ru_RU.UTF-8) with following error message
 - link-grammar: Error parsing dictionary /ru/stem.dict.
Expecting ";" at the end of an entry.
     line 6140, tokens = "ЕДК,0>" ";" "бессмысленно.e" "зачем.e" "здесь.e"

My first though was that it is caused by wrong locale (that is why the setLocale() method was implemented), but it is not,,,

It has to be... maybe the locale setting is beig over-ridden, later on, by some other unrelated java component? 

Could you point me to any example how to switch languages using LinkGrammar directly (without Java wrap)?

The only way I've ever tested is via the command-line utility: 

link-parser  /some/path/to/data/ru

for you I guess this would be something like this:

cd C:\some\path\to\lg\source

link-parser .\data\ru

or just 

link-parser C:\some\path\to\lg\source\data\ru

-- Linas

[Linas Vepstas]

I should also mention that all of my testing of the ru dictionaries is done in the en_US.uft8 locale.   This is OK, because the only thing that is used out of the locale is the utf8 setting (which is identical between en_US.utf8 and ru_RU.utf8). The other settings, where these two locales differ:  currency symbols, date, time formats, etc. are not used by link-grammar. 

--linas

[Alexander Tkachuk]

Hi Linas,

Thank you very much for your answers and suggestions!

Just finished playing with LinkGrammarExe (build it successfully using Visual C++ 9 after couple changes) to be sure that my problem was not caused by Java environment. In order to exclude other system related problems I built and tested it both for Win32 and x64 platforms using two different RegEx libraries ( http://gnuwin32.sourceforge.net/packages/regex.htm and http://www.airesoft.co.uk/pcre) for Win32 build. It works well for English, German and Lithuanian languages, but it doesn't work for Russian one showing the same error messages as in Java.

The first error says "link-grammar: Error parsing dictionary /ru/stem.dict.
Expecting ";" at the end of an entry. line 6140..."

That line contains "бессмысленно.x зачем.x здесь.x :  <morph-ПРЕДК,0>;".

I managed to get rid of this error deleting all lines which contain "morph-ПРЕДК".

Then I god second error saying "link-grammar: Error parsing dictionary /ru/suffix.dict.
Expecting ";" at the end of an entry. line 153..."

That line contains "=.i :
  (LLAEN- or LLDTC- or LLDTF- or LLDTI- or LLDTJ- or LLDTK- or LLDTL- or LLDTM- or LLDTN- or LLDTO-)
    & <morph-СОЮЗ,0>;"

I tried to delete several lines following error messages, but was unable to find any relationship between error messages and lines...

I got your comment about UTF locale, but it still looks like locale-related problem...

I appreciate any thoughts!

Thank you again,

Alex

***



14.03.2013 21:23, Linas Vepstas пишет:

[Linas Vepstas]

On 18 March 2013 10:01, Alexander Tkachuk <alexande...@gmail.com> wrote:

tested it both for Win32 and x64 platforms using two different RegEx libraries ( http://gnuwin32.sourceforge.net/packages/regex.htm and http://www.airesoft.co.uk/pcre) for Win32 build.

The problems you are seeing are not regex related.  Currently, the russian dicts almost don't use regex at all -- there is only a single line for recognizing capilatlized words (in 4.0.regex)

 

The first error says "link-grammar: Error parsing dictionary /ru/stem.dict.
Expecting ";" at the end of an entry. line 6140..."

Have you tried setting the locale to en_US.UTF8 ?  Does it work then?

I don't know of any easy way of debugging this.  I guess you can do a bit of C programming, right? If not, then I'll have to send you modified files, and get you to compile them and send me the resulting output.

The error message that you see is printed at line 1500 of read-dict.c 

You can try adding this print just before line 1500:

wchar_t * ptr = dict->pin;   // the current input location.

ptr -= 80;                       // back up 80 chars to see what came before

int jjj;

for (jjj=0; jjj<100; jjj++) {

    wchar_t  c = *ptr;

    int ic = (int) c;

    printf("jjj=%d    char=%lc    val=0x%x\n", jjj, c, ic); 

    ptr++;

}

or better yet, insert that before line 352

This will show you the text leading up to the error.  Notice that the dicationary is read as wide characters (which are 16 bit or 32-bit characters, I forget). The conversion to utf8 happens at line 286  by wcrtomb()   Notice that the conversion uses a "mutli-byte shift state" to track where it is in the input stream.   It might be possible to fix the problem by adding this:

   memset(&dict->mbss, 0, sizeof(dict->mbss));

just before line 286.  However, this might introduce new errors, I am not sure.  On the other hand, it might be possible to fix your problem by *removing* the above memset from line 1386

If you do remove it from 1386, it might need to be moved to line 1400 and maybe line 1459 ... or something like that.

If you think you can experiment with the above to find the problem, that would be great. Otherwise, I would have to send you modified files, ask you to compile them, and then have you send me the output so that I can ponder it.

-- Linas

[Alexander Tkachuk]

Hi Linas,

Thank you very much for your suggestions and comments!

I tried all of them without any success, then I sent it to my colleague, and he found the problem.

The Cyrillic "Р" symbol is represented by 1056 and 160 wchar_t values, and it is treated as a delimiter in link_advance() at read-dic.c. As a result any token with mentioned symbol is split in to two tokens by mistake. This is cause the problem.

My colleague and me are still working on the problem trying to fix it quickly. Assuming that it works well on Linux the problem can be caused by Visual C++ compiler and/or Windows libraries. We are trying to find any information such as compiler options which can affect wchar_t related behavior.

If we will be unable to find solution using Visual C++ quickly, probably we will try to use MinGW and/or CygWin building LinkGrammar.

Please, let me know if you have any thoughts!

Thank you again,

Alex

***

18.03.2013 20:14, Linas Vepstas пишет:

[Linas Vepstas]

Hi,

On 25 March 2013 04:43, Alexander Tkachuk <alexande...@gmail.com> wrote:

Hi Linas,

Thank you very much for your suggestions and comments!

I tried all of them without any success, then I sent it to my colleague, and he found the problem.

The Cyrillic "Р" symbol is represented by 1056 and 160 wchar_t values,

1056 is hexadecimal 0x420

160 is hexadecimal 0xA0

OK, if I look at http://en.wikipedia.org/wiki/List_of_Unicode_characters#Cyrillic then I see that cyrillic P is U+0420 so that makes sense.

However U+00A0 is "non-breaking space" so I don't understand that statement.  If Windows is encoding P as 0xA0 then indeed, there is a problem.  Can you verify that this is not happening?

and it is treated as a delimiter in link_advance() at read-dic.c.

I assume you mean "whitespace".  I believe that  you are referring to line 321 of that file, which is

         if (iswspace(c)) {

            dict->token[i] = '\0';

            return TRUE;

         }

It should be the case that iswspace(0x420) returns false

I presume that iswspace(0xA0) would return true.   If it does on Windows, then clearly if P is encoded as 0xA0 then there is  .. a strange bug.  Can you verify this?

---------

I see one other place where there might be a bug.  If you look at utilities.h line 103 you will see this:

#define iswspace  isspace

Now 0x20 is ASCII space, so perhaps isspace (0x420) would return true, and that would explain your bug.  However, that #define is used only if you have __CYGWIN__ defined.  Perhaps this is accidentally defined ??  If so, that's the bug.  See line 90:

/*

 * CYGWIN on Windows doesn't have UTF8 support, or wide chars ... 

 * However, MS Visual C appearently does, as does MinGW.  Since

 * some users have both cygwin and MinGW installed, crap out the 

 * UTF8 code only when MinGW is missing.

 */

#if defined (__CYGWIN__) && !defined(__MINGW32__)

Can you verify that you do not have this symbol defined?

--linas

[Alexander Tkachuk]

Hi Linas,

So sad to hear about the Boston marathon, explosions, and victims. Hope you, your family, and your friends are safe.

I was living in the Boston area for several years, and it became my second native city in addition to my Russian one.

We gave up fighting with LinkGrammar in Windows and switched to Linux. Please, find below summary of found issues and our deeds.

1. Dictionary reading issue

As it was discussed before, we had problems reading the Russian dictionary from Windows. For every single Cyrillics letter we received two wchars, for example: for the letter "А":  1056 (which is "Р" itself) and another wchar (don't remember what exactly).

Alexey fixed it replacing  "FILE *fp = dictopen(dict_name, "r")" to FILE *fp = dictopen(dict_name, "r, ccs=UTF-8"). After that the Russian dictionary was read smoothly. We got exactly one wchar per Russian letter.  But then we got problems reading the English dictionary. The English dictionary contains a row of special symbols like this : ½ ⅓ ⅔ ¼ (fractions) and some others like the degree sign. After changes which have been made, these special symbols are read as one wchar, but in the original version they were read as several wchars (multibyte char conversion), and int nr = wcrtomb(s, wc, ps) returns -1, without converting single wchars like that : ½. Alexey played with it setting different locales, but it didn’t work.

2. Parsing issue

Alexey noticed that Visual Studio doesn't read hard-coded UTF-8 values correctly, like this one:  "const wchar_t wqs[] = L"\"\'«»《》【】『』‘’`„“""; from tokenize.c. The reason is that its compiler simply doesn't understand them as UTF-8 values even when the source file encoding is set to UTF-8 directly. As the result, the input strings are splitted into quotes incorrectly. Alexey managed to read the quote wchars correctly only by saving the tokenize.c file in the UTF-8 with BOM (without BOM it doesn't work) encoding and replacing:
static const wchar_t *list_of_quotes(void)
{
    /* Single-quotes are used for abbreviations, don't mess with them */
    /* const char * qs = "\"\'«»《》【】『』‘’`„“"; */
    const wchar_t wqs[] = L"\"\'«»《》【】『』‘’`„“";
    return wqs;
}

After that we were be able to see them in the debugger correctly. 

Link Parser executable didn't work because the Windows console doesn't support UTF-8 by default. Probably additional encoding is needed. That is why we tested it by passing the input string from a C-source file, which was encoded in UTF-8, but then there was another issue. In source files our input string stores as an array of chars, in Cyrillics case, it will be an array of multibyte chars.  For example, the word "получения" will transform to the char array "получения". Later in tokenize.c the mbrtowc function doesn't convert multibyte chars into wchars properly (I get exactly the same wchars "получения", presumably, one char corresponds to one wchar, which is not correct for Cyrillics). And as a result, the whole string is splitted as a quote, because it contains the "»" symbol.

After all these changes we managed to get the Russian dictionary loaded, but we were unable to get any meaningful results. All that we got for Russian sentences were linkages with 0 links.

Then, as I mentioned abive, we switched to Linux working on our project and everything works well there. Though, some of our colleagues are Windows persons unfortunately.

Also I added couple functions to jni_client enabling language choice and linkages count. You can find modified files attached.

Thank you,

Alex

***


25.03.2013 19:05, Linas Vepstas пишет:

[Linas Vepstas]

Hi Alexander,

On 17 April 2013 10:20, Alexander Tkachuk <alexande...@gmail.com> wrote:

1. Dictionary reading issue

As it was discussed before, we had problems reading the Russian dictionary from Windows.

Is there a utility, in Windows, to convert UTF8 files into the native format?  In unix, we have the "iconv" command which will convert between just about any format to any other.   Perhaps if you converted the link-grammar dictionaries into the windows native format, then it would work?

Alexey noticed that Visual Studio doesn't read hard-coded UTF-8 values correctly, like this one:  "const wchar_t wqs[] = L"\"\'«»《》【】『』‘’`„“""; from tokenize.c. The reason is that its compiler simply doesn't understand them as UTF-8 values even when the source file encoding is set to UTF-8 directly.

Based on your description, I made this change:

#ifdef _MSC_VER

   static const wchar_t* wqs = "\"«»《》【】『』`„“";

   return wqs;

#else

...

I'm not sure what to do about the BOM.

Link Parser executable didn't work because the Windows console doesn't support UTF-8 by default.

OK ... reads from the console are implemented in link-parser/link-parser.c line 156 and line 130 ... what if, instead, these read wchar_t and then did a wctomb before returning?

 

BTW, there is a special command to read input from a file:  it is !file   so, for example:

linkparser> !file stuff.txt

!help will explain more.

Then, as I mentioned abive, we switched to Linux working on our project and everything works well there. Though, some of our colleagues are Windows persons unfortunately.

I'm still thinking that an iconv of the dictionary files into windows native format, and a change of the console read to read wide chars might be able to solve these problems. 

Is there a way you could run this experiment? If it works, I would like to include it in 4.7.12

Also I added couple functions to jni_client enabling language choice and linkages count. You can find modified files attached.

Commited to svn. Will appear in 4.7.12   But you did not send the corresponding changes to LinkGrammar.java, LGConfig.java and LGService.java -- maybe you are not using these?  The java API there allows distributed networked parsing, so that it can be done in parallel over a cluster of machines.

--linas

[Alexander Tkachuk]

Hi Linas,

Thank you for your quick response!

Sorry, forgot to attach java files in my previous email.

I think it is better to use the same encoding for all platforms. UTF works perfectly fine (e.g. Java uses UTF strings by default) we just need to resolve some encoding issues.

Thank you again,

Alex

***

17.04.2013 23:48, Linas Vepstas пишет:

[Linas Vepstas]

On 18 April 2013 09:54, Alexander Tkachuk <alexande...@gmail.com> wrote:

I think it is better to use the same encoding for all platforms. 

I would not want to distribute the dictionaries in two different encodings ... but if there was a Windows tool to change the dictionary encodings for the Russian dictionaries, and this solved all the problems for Russian windows users ...  this would be a good, simple solution.

-- Linas

[Alexander Tkachuk]

Hi Linas,

I agree with you that all dictionaries should be distributed using the same encoding.

Now I faced with another issue, it looks like that your implementation and Sergey's one (http://slashzone.ru/parser/) work in different way (see the attachment). I got number of similar examples where the first one is unable to identify POS of a word near left wall. This issue appears in all generated linkages. Not sure that it is a bug or feature:)

By the way do you know any other teams which work with the Russian dictionary? Recently I tried to get in touch with Sergey by email without success.

Thank you,

Alex

***


18.04.2013 21:46, Linas Vepstas пишет:

[Linas Vepstas]

On 29 April 2013 10:46, Alexander Tkachuk <alexande...@gmail.com> wrote:

Now I faced with another issue, it looks like that your implementation and Sergey's one (http://slashzone.ru/parser/) work in different way (see the attachment). 

They work in a fundamentally different way.  Sergey's original design would have resulted in dictionaries that were gigabytes in size (maybe tens of gigabytes?) which is clearly impractical.  His solution was to create a set of perl wrappers that would examine one sentence, automatically generate a very small dictionary containing only those words for that sentence, and then invoke the link parser with that single-sentence dictionary.

This seemed like an awful hack to me; I wanted to distribute the russian dictionary, but not these hacky perl scripts. So I redid the design to use the link-grammar suffix mechanism to split Russian words into a stem and a suffix, with a link connecting the two.  This allowed the Russian dictionary to fit in less than a gigabyte.

My first cut had bugs and short-comings and mis-understandings in it, Sergey and I went through several iterations to get it to work mostly-right.  In his last email, he mentioned that there were still several bugs, and that he'd work to fix them ... I am hampered, because I don't know russian enough to be able to tell apart good and bad parses.  I think the bugs had something to do with words that had a "null" (empty) suffix.  Not sure.

In the example screenshot you show, it looks like the first letter is capitalized in one, but not the other sentence.  Try using a lower-case letter, and see if that fixes it.

By the way do you know any other teams which work with the Russian dictionary?

No.

 

Recently I tried to get in touch with Sergey by email without success.

I suggest offering to pay him some money ... I think he would be thrilled to be able to  work on this for a living.

-- Linas

[Alexander Tkachuk]

Hi Linas,

Thank you!

I just got in touch with Sergey! I guess with your help:)

I hope to get funding soon to involve more people into development.

You was quite right, the problem was in the capitalized first letter. Now it works well for my purpose!

Thank you again,

Alex

P.S. Will come back to the Russian dictionary issues under Windows soon.

***


29.04.2013 20:44, Linas Vepstas пишет:

[Linas Vepstas]

Hi,

On 30 April 2013 04:41, Alexander Tkachuk <alexande...@gmail.com> wrote:

You was quite right, the problem was in the capitalized first letter. 

In that case, there is some bug & I will try to fix it. The parser is supposed to automatically try to use the lower-case version of the first word in the sentence, so that you don't have this problem. Somehow, that didn't work right, I'm guessing due to unicode.

--linas