qa_enhancements branch ready to merge to dev...

[Jim Hargrave]

I've added a decent number of unit tests. More importantly the current
code should be backwards compatible. Biggest changes are in
languagetool-lib and languagetool-step. languagetool-lib is no longer
empty and contains base code for interacting with LT. languagetool-step
adds new options for "smart" term and blacklist checking. You can enable
all rules at once to get spell, grammar, bitext, term and blacklist
checking.

Yves - the languagetool-step parameters UI is configured for the new
options.

Let me know of any issues, otherwise I will merge in a few days.

Jim

[Yves Savourel]

Hi Jim,

I've tried building the branch: There are several unit tests of LanguageToolTest that pass when executed under eclipse, but they fail when trying the external rebuild.

I get:

-------------------------------------------------------------------------------
Test set: net.sf.okapi.steps.languagetool.LanguageToolTest
-------------------------------------------------------------------------------
Tests run: 9, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 6.248 sec <<< FAILURE! - in net.sf.okapi.steps.languagetool.LanguageToolTest
simpleTest2(net.sf.okapi.steps.languagetool.LanguageToolTest) Time elapsed: 0.099 sec <<< FAILURE!
java.lang.AssertionError
at net.sf.okapi.steps.languagetool.LanguageToolTest.simpleTest2(LanguageToolTest.java:97)

termCheckSdlTbx(net.sf.okapi.steps.languagetool.LanguageToolTest) Time elapsed: 0.438 sec <<< FAILURE!
java.lang.AssertionError: expected null, but was:<P67345789?its-lqi?lqiComment?s?The source term "font" is not translated with the recommended target term. "Schrift" [Suggestions: "Schrift"]?lqiEnabled?b?true?lqiXTrgStart?i?33?lqiXStart?i?-1?lqiSeverity?f?2.0?lqiXTrgEnd?i?38?lqiXIssueType?s?LANGUAGETOOL_ERROR?lqiXType?s?LANGUAGETOOL_ERROR?lqiXEnd?i?-1?lqiXSegId?s?0?lqiType?s?terminology?its-lqi?lqiComment?s?The source term "typeface" is not translated with the recommended target term. "Schrift" [Suggestions: "Schrift"]?lqiEnabled?b?true?lqiXTrgStart?i?4?lqiXStart?i?-1?lqiSeverity?f?2.0?lqiXTrgEnd?i?12?lqiXIssueType?s?LANGUAGETOOL_ERROR?lqiXType?s?LANGUAGETOOL_ERROR?lqiXEnd?i?-1?lqiXSegId?s?0?lqiType?s?terminology?its-lqi?lqiComment?s?The source term "typeface family" is not translated with the recommended target term. "Schriftfamilie" [Suggestions: "Schriftfamilie"]?lqiEnabled?b?true?lqiXTrgStart?i?4?lqiXStart?i?-1?lqiSeverity?f?2.0?lqiXTrgEnd?i?21?lqiXIssueType?s?LANGUAGETOOL_ERROR?lqiXType?s?LANGUAGETOOL_ERROR?lqiXEnd?i?-1?lqiXSegId?s?0?lqiType?s?terminology>
at net.sf.okapi.steps.languagetool.LanguageToolTest.termCheckSdlTbx(LanguageToolTest.java:166)

termCheckBlacklistJapanese(net.sf.okapi.steps.languagetool.LanguageToolTest) Time elapsed: 0.369 sec <<< FAILURE!
java.lang.AssertionError
at net.sf.okapi.steps.languagetool.LanguageToolTest.termCheckBlacklistJapanese(LanguageToolTest.java:256)

I'm under Windows (10).
I'll try to look at it more later.

Cheers,
-yves

[Jim Hargrave]

My command line builds are working. Try a raw maven build to force
updated of all the latest dependencies:

mvn clean install -U

Jim

[Yves Savourel]

No luck with -U.
Same errors.

I'll keep digging as soon as I have time.

-ys

--
You received this message because you are subscribed to the Google Groups "okapi-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to okapi-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jim Hargrave]

The only thing that comes to mind is that we are now using the
MultiThreadedJLanguageTool. Perhaps the windows JVM has a thread
implementation difference.

I even tried building/running on various combinations of Java 1.7 and
1.8, but all my tests still pass.

The errors almost look like old code - but when a pushed git said
everything is up to date on the "qa_enhancments" branch.

Jim

[Yves Savourel]

I've made sure I was working with the latest update from Git: I was.
I've tried different things without success.

The code is there and runs fine in eclipse.
For a time I thought maybe the Japanese issue was related to the encoding (we use the raw characters in the source code), but that can't be that since the French example also failing uses the \u notation.

I'm baffled.

[Jim Hargrave]

I just pushed some changes (1) I added a thread safe Language cache to
speed up processing and (2) removed the NFKC Unicode normalization.

No reason to think the changes will make any difference, but you never know.

I'm trying to reproduce the problem, but nothing yet.

Jim

[Yves Savourel]

Thanks.
I'll try the new code.

[Jim Hargrave]

That still looks like old code failing the tests. To be sure, on the command line, before running the build script, can you run:

git pull origin qa_enhancments

If you check the file TermCheck.java you should see this code for the match method:

@Override
    public RuleMatch[] match(AnalyzedSentence sourceSentence, AnalyzedSentence targetSentence) throws IOException {
        RuleMatch[] matches = srcRule.match(sourceSentence);
        if (matches.length > 0) {
            // if we can't find the specified target term we have an error
            if (trgRule.match(targetSentence).length <= 0) {
                return matches;
            }
        }

        // otherwise we found the source and target terms
        return new RuleMatch[0];
    }

Jim

[Yves Savourel]

Yes, I have that code (in TermCheckRule.java, not TermCheck.java).

And I got your latest changes.

Still passing when run from eclipse, but failing from the maven command-line (I do a full rebuild)

 

Could someone else on Windows try the Maven build for that branch?

 

The problem has to be either specific to Windows or to my machine’s configuration.

 

Thanks,

-yves

[Jim Hargrave]

The results you are seeing are exactly as if the rule below was reversed (trigger the rule if we *do* find the target term), which was the case a while back.

If I can figure out how to install everything on windows I will give it a try there and see what I find.

Jim

[Yves Savourel]

The code is the same as in your email, so it has to be the return of srcRule.match(sourceSentence) that is incorrect.

I’ll try different ways to build, it has to be a build problem, not a code issue, since it works in eclipse.

 

-ys

[Jim Hargrave]

On my Windows 10 VM I can successfully build with "mvn clean install"

Jim

[Yves Savourel]

Ah! I think I know why it doesn’t pass: the LT resources don’t get loaded properly.

From eclipse I get:

[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'en-US' ('English (US)') for locale 'en-us'
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'fr' ('French') for locale 'fr-fr'
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'en-US' ('English (US)') for locale 'en'
…
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'ja-JP' ('Japanese') for locale 'ja'
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'en-US' ('English (US)') for locale 'en-us'
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'fr' ('French') for locale 'fr-fr'

But from the command line I get:

[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'en-US' ('English (US)') for locale 'en-us'
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'default' ('Default') for locale 'fr-fr'
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'en-US' ('English (US)') for locale 'en'
…
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'default' ('Default') for locale 'ja'
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'en-US' ('English (US)') for locale 'en-us'
[main] INFO net.sf.okapi.steps.languagetool.LanguageTool - Using LT language 'default' ('Default') for locale 'fr-fr'

So the test don't get the right locale-specific resources and, naturally, fail.

What could cause this?
I'll start looking at the loading calls.

-ys

[Jim Hargrave]

LanguageTool loads whatever Languages are found on the classpath. If the
onlu locale that is loading is English US, then its acting like the
other Languages are not availible. Maybe only the core language tool is
installed or available? I expect German will be the same thing, though
not listed here.

One thing to try is deleting you local maven repo, or at least the
language tool maven artifacts.

Jim

[Yves Savourel]

> One thing to try is deleting you local maven repo,
> or at least the language tool maven artifacts.

I did that earlier this morning, to no effect.

I do have FR and JA, etc. (and EN) all version 3.1:

.m2\repository\org\languagetool\language-ja\3.1

So, somehow the command-line classpath is not getting those... mmm.

-ys


-----Original Message-----
From: okapi...@googlegroups.com [mailto:okapi...@googlegroups.com] On Behalf Of Jim Hargrave
Sent: Monday, March 28, 2016 3:18 PM
To: okapi...@googlegroups.com
Subject: Re: [okapi-devel] qa_enhancements branch ready to merge to dev...

[Yves Savourel]

Hi Jim, all,

I'm still not understanding why I can't access the LT resources from the build, but I can from eclipse.

We have one POM reference to LT which is:

<dependency>
<groupId>org.languagetool</groupId>
<artifactId>language-all</artifactId>
</dependency>

That dependency has a POM referring back to:

<parent>
<groupId>org.languagetool</groupId>
<artifactId>languagetool-parent</artifactId>
<version>3.1</version>
<relativePath>../../pom.xml</relativePath>
</parent>

which itself declares all the language resources as modules.

So eclipse gets the DE, JA, etc. JARs from that path.

But why the build from the command-line does not?

I would understand if the test was called like one of the Tikal tests which runs the built application: Since we don't put the LT resources in the built JAR (dependencies are ant-driven in the build if you recall), then missing them makes sense.

But the failing tests are unit tests run exactly like in eclipse, so why the LT dependencies are not fully taken into account? Is there some kind of flag to fetch dependencies of a dependency that we are missing?

And, Jim, I would be interested to know how you manage to have the resources in the classpath using the same build/pom files as I do. You've got to do something that is not in the build, no?

Cheers,
-yves

[Jim Hargrave]

Yves,

I was able to take a clean Windows 10 VM. Clone the okapi source and do a full build without errors. There was no modifications to the Windows 10 system (paths, classpath etc..) other than install Java 8 and maven.

This shows its not a problem with Windows. Have you tried on a clean system? You can set up a VM using VirtualBox for a quick test. If it works there then you know its a problem with your setup. I still think a classpath issue is the most likely problem. Somehow the language resources are being hidden from the classloader.

Another possibility (remote) is that the language jars downloaded, but are corrupt somehow.

Jim

[Jim Hargrave]

Yves I just created the branch "dev_java_8" which upgrades to the LT 3.3. This branch can only be built and run with Java 8.

Can you try this branch to see if you have any better luck? It should download all the newest 3.3 artifacts from maven central.

Jim

[Yves Savourel]

Will do. Thanks!

 

From: okapi...@googlegroups.com [mailto:okapi...@googlegroups.com] On Behalf Of Jim Hargrave
Sent: Wednesday, April 6, 2016 3:41 PM
To: okapi...@googlegroups.com
Subject: Re: [okapi-devel] qa_enhancements branch ready to merge to dev...

 

Yves I just created the branch "dev_java_8" which upgrades to the LT 3.3. This branch can only be built and run with Java 8.

--

[Yves Savourel]

No better luck with dev_java_8.

Same errors: no access to the LT resources, so the default EN resources are loaded and the tests fail with them.

 

The package that fails is the step. The library package complies fine and produces a JAR with the LT resources for all the languages (okapi-lib-languagetool-0.30-SNAPSHOT.jar).

 

There is something different in the generated JARs for that package compared to the other libraries packages: It also have an original-okapi-lib-languagetool-0.30-SNAPSHOT.jar in addition to the normal and test JARs. And there is a dependency-reduced-pom.xml. Those are related to the shade plugin: so we do get the LT resource in the library’s JAR, I don’t see why the runtime parts of LT would be accessible while the resources would not.

 

Tomorrow I should be able to try on a new Windows 7 setup to see if there is any difference, but IMHO it seems that none of that has anything to do with Windows or Linux: We’ve got to have some kind of difference in the maven settings.

 

Cheers,

-ys

 

 

 

From: okapi...@googlegroups.com [mailto:okapi...@googlegroups.com] On Behalf Of Jim Hargrave

Sent: Wednesday, April 6, 2016 3:41 PM

To: okapi...@googlegroups.com
Subject: Re: [okapi-devel] qa_enhancements branch ready to merge to dev...

 

Yves I just created the branch "dev_java_8" which upgrades to the LT 3.3. This branch can only be built and run with Java 8.

--

[Jim Hargrave]

You may be right about the maven difference, but it would have to be a previous merge issue or something like that. Delete, then re-clone the repository and start fresh - that should rule out any differences.

Jim

[Yves Savourel]

Hi Jim, all,

 

Some progress: I am able to pass the language-dependent tests only if I add the dependency to ‘language-all’ (or the three specific languages) in the POM of okapi-step-languagetools.

 

It seems having the inherited dependency to the language resources in the library is not working when running the command line.

[Jim Hargrave]

It still troubles me that everything works fine when run cleanly (fresh pull) on my Windows 10 VM. But if the changes you are making work generally then I'm OK with that (though it seems redundant).

Jim

[Yves]

Hi Jim, all,

So, as we discussed yesterday, I updated maven (got the latest 3.3.9) and tried without the extra dependency in the step...

And it worked: All tests pass now.

Apparently there is some behavior related to implicit dependencies that is different.

I guess you can merge the `qa_enhancments` branch.

Thanks,

-yves

[Jim Hargrave]

Sweet butter biscuits! Sorry this was such an issue. I will do the merge now.

btw: I'll upgrade my maven to the latest as well. Probably a good idea for everyone reading this post.

Jim

[Jim Hargrave]

dev has been updated with the qa changes and the original branch deleted. For those interested I will also update the dev_java_8 branch shortly.

Note that the dev_java_8 branch uses the latest language tool jar 3.3 (only supported on Java 8)

Jim

On 04/15/2016 11:37 AM, Yves wrote:

[Jim Hargrave]

I did the merge and now I get jenkins saying the build is unstable - no failing tests afaik. The latest version of maven is 3.0.3 I'm wondering now if this is enough to build this beast.

Do we know of a way to do a fresh build? That is, delete everything (m2 repo , target files etc..) and then build.

Jim

On 04/15/2016 11:37 AM, Yves wrote:

[Mihai Nita]

If we can figure out exactly at what version the failures start then we can put it in superpom/pom.xml as minimum requirement.
The one listed now is 3.0.5

Mihai

[Jim Hargrave]

Good point - but the latest maven in jenkins seems to be really early: 3.0.4. Is there any way to get a newer version? Or maybe if I change it in the pom it will magically work, let me try.

Jim

--

[Mihai Nita]

I don't think we should ask for the latest either.
Maybe some projects out there can't really update maven easily. They might use a "good enough version", no need to" break" them

I will try to detect what the minimal version is over the weekend.

Mihai

[Jim Hargrave]

For now I am using 3.3.3 in our pom which is pretty old. I found the config to upgrade our maven choices and made the default 3.3.3 as well.

Running the build now.

Jim

[Jim Hargrave]

Build is back to normal. Thanks Mihai!

Jim

[Mihai Nita]

I did some more digging to see what is the minimal version that works.
So I've downloaded and tried older maven versions (from https://archive.apache.org/dist/maven/maven-3/)

3.2.5 fails, 3.3.1 succeeds. It looks like there is nothing in between.

Then I spent some time trying to figure out the root cause.
Not that easy this time :-)
But I'm still digging, looks like an interesting challenge :-)

Anyway, for now I think that asking for maven 3.3.1 as minimal version in the superpom will do. I think it is friendlier to other okapi users to not force them to update maven if not really needed.

Mihai

[Jim Hargrave]

Thanks Mihai - I'll make the change in the pom.

Should we leave the jenkins maven version at the latest?

Jim

--

[Mihai Nita]

Got some more info on what is going on...

LanguageTool determines which languages it supports at runtime by reading them from a file META-INF/org/languagetool/language-module.properties in the classpath.
http://wiki.languagetool.org/java-api#toc4

The problem is that the .jar generated by maven-shade-plugin in lib-languagetool only contains English in that file.

So Japanese, French, German, others, map to Default.

Take a look at

~/.m2/repository/net/sf/okapi/lib/okapi-lib-languagetool/0.30-SNAPSHOT/okapi-lib-languagetool-0.30-SNAPSHOT.jar

If you unzip it and check
  META-INF/org/languagetool/language-module.properties

the content is
  languageClasses=org.languagetool.language.English,
      org.languagetool.language.AmericanEnglish,
      org.languagetool.language.BritishEnglish,
      org.languagetool.language.AustralianEnglish,
      org.languagetool.language.CanadianEnglish,
      org.languagetool.language.NewZealandEnglish,
      org.languagetool.language.SouthAfricanEnglish

(I've added line wraps for readability)

To test my "hypothesis" I've added all the other languages to that property file, and the test passed with older maven versions (only tried 3.2.5)

(the correct value for each language is in the matching jar under ~/.m2/repository/org/languagetool)

So I think that the shaded jar is incorrect, it might affect production systems, and needs fixing (might not be instantly visible, if default is used instead of French some things might still "kind of" work, like sentence / word breaks).

I have no explanation on why maven 3.3.x works. In my opinion it should fail.

And on Ubuntu it does fail! (with maven 3.3.9)

I build everything, everything passes, it's fine and dandy.

Then I go to okapi/steps/languagetool, run mvn install, and fails with exactly the same errors as Windows with maven 3.2.x:
  Failed tests:
    LanguageToolTest.simpleTest2:97 null
    LanguageToolTest.termCheckBlacklistJapanese:257 null
    LanguageToolTest.termCheckSdlTbx:166 expected null, but was:

My guess is that it has something to do with "from a file ... in the classpath"

Something with how maven sets the classpath for testing...

So I tried to see if I can change the shade settings.

There is a "transformer" that seemed promising:
https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#AppendingTransformer

And it seems to solve things, at least on Ubuntu.
I've commited the fix, and I will try it tonight on Windows with older maven versions.

If you run it problems please revert my "fix" and let me know.

Cheers,

Mihai

[Jim Hargrave]

Strange on my Ubuntu and maven 3.3.9 everything works fine :-)

I will take a look at the shaded jar to make sure all the languages are included. Thanks for tracking this down, ugly!

Jim

[Jim Hargrave]

Mihai, note that qa_enhancments  is now a dead branch as of last week. I deleted it when I merged into dev.

I'm not sure about the shaded changes, as I said when building on the latest dev everything works fine for me.

Jim

On 04/18/2016 11:59 AM, Mihai Nita wrote:

[Jim Hargrave]

Shaded jar looks correct to me with all languages represented:

On 04/18/2016 11:59 AM, Mihai Nita wrote:

[Mihai Nita]

On Ubuntu
* mvn clean
* I delete the .m2 repository
* mvn install (the full build) succeeds
* I go in okapi/steps/languagetools, run mvn install and it fails

Mihai

[Jim Hargrave]

Same here. And this is on Dev where the qa_enhacments have been merged. Can you add your build fix to dev?

Jim

--

[Jim Hargrave]

I'm not sure what the manifest listing does. For example I create my own LT language called "Default" and use it with the LT rules without a problem. It's not in the manifest, only in the classpath. Maybe the manifest is only to get a list of the supported languages quickly without searching the classpath?

No reason our manifest shouldn't be complete so I welcome your fix Mihai. Just still confused a bit :-)

Jim

On 04/18/2016 11:59 AM, Mihai Nita wrote:

[Mihai Nita]

Don't try creating default, try Japanese (or German, or French). They will fallback to default.

Mihai

[Jim Hargrave]

My tests do use japanese. And when I use the okapi sdk framework jar (also a fat jar) everything seems to work even with the same manifest having only the English variants.

That's why I'm still baffled.

Jim

On Apr 18, 2016 8:00 PM, "Mihai Nita" <mih...@gmail.com> wrote:

Don't try creating default, try Japanese (or German, or French). They will fallback to default.

Mihai

--

[Mihai Nita]

Yes, it is weird.
The steps I've described don't gain on dev.
Mihai

[Mihai Nita]

s/don't gain on dev/don't fail on dev/

[Mihai Nita]

I've merged my fix into main

To test it I have changed the superpom back to require maven 3.0.5, and did a full build with it.

Clean .m2 repository, Windows 7 x64, jdk 8 x64
(same configuration that was failed for me a day or two ago)

But I did not submit the superpom.
I can do it if you think it is a good idea (or anyone can do it :-)

Cheers,
Mihai

[Mihai Nita]

Another "quick and dirty" test.

Took the following code:

===============
import org.languagetool.Language;
import org.languagetool.Languages;

public class LanguageToolTest {

    public static void main(String[] args) {
        for (Language lang : Languages.get()) {
            System.out.println(lang);
        }
    }
}
===============

And compiled + run it against all the original languagetools jar files as downloaded by maven

I've dumped them all in one folder (also required org.apache.commons.logging & net.sourceforge.segment)

Compiled and run
    javac -cp .;libs\* LanguageToolTest.java
    java -cp .;libs\* LanguageToolTest

The list contains 45 locales.

Then I did the same thing, but with the classpath pointing to the shaded jar built yesterday, before my fix:
    javac -cp .;okapi-lib-languagetool-0.30-SNAPSHOT.jar  LanguageToolTest.java
    java -cp .;okapi-lib-languagetool-0.30-SNAPSHOT.jar  LanguageToolTest

The list contains 7 entries (English variants)

And finally the same test using the shaded jar built today, after my fix.

The result was the same list of 45 locales as resulted with the original jar files.

No maven involved, nothing.

It clearly shows that things were wrong before.

I am still baffled by the fact that things passed in some combinations...

My guess is that the classpath also pointed somehow to the original jar files,
not only to the shaded jar.
Maven (black) magic... :-)

Mihai

[Jim Hargrave]

Language tool step now works on my system too. Many thanks! This was a thorn for far too long.

Your comment below makes sense to me and probably explains how my production code was working.

cheers,

Jim