RE: [EXTERNAL] java errors during .mrc indexing in VUFind 5.1 and Ubuntu 18.04 LTS (openjdk 11.x)

[Demian Katz]

Theodoros,

 

I’m copying the solrmarc-tech list into this thread, in case this rings a bell for anyone there.

 

I do all of my VuFind development work with OpenJDK 11. Here’s the output of “java -version” on my development server:

 

openjdk version "11.0.4" 2019-07-16

OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)

OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3, mixed mode, sharing)

 

Is this significantly different in any way from the JDK 11 that you are using?

 

The only other thing I’m wondering is whether the relative path is coming from in your output (i.e. /home/ubuntu/../../data/vufind-prod/vufind/local/import). I ran into some problems in the past where duplicate home directory paths would cause weird problems with custom class loading. I added this commit to work around it, though I’m not convinced it’s the best solution:

 

https://github.com/solrmarc/solrmarc/commit/b6c34fad04fc2db7d79b2ddd4f9de403b120b569

 

I’m wondering if perhaps your problem is being caused by having the same home directory represented in two different ways in the list – once as an absolute path and once as a relative one that evaluates to the same thing.

 

I wonder if you can fix the problem locally by identifying and eliminating the source of that relative path… and perhaps we can fix it more robustly by forcing absolute path conversion inside SolrMarc somewhere.

 

But of course none of that explains why it would work in JDK 8 and not JDK 11 – but it may be that different code is executing to account for different class loading functionality in the different versions.

 

- Demian

 

From: Theodoros Theodoropoulos <th...@physics.auth.gr>
Sent: Friday, October 18, 2019 8:35 AM
To: Demian Katz <demia...@villanova.edu>
Subject: Re: [EXTERNAL] java errors during .mrc indexing in VUFind 5.1 and Ubuntu 18.04 LTS (openjdk 11.x)

 

Also please find included the two outputs of the indexing process, one with jdk8 the other with jdk11. BOTH use the same jars, folders and .properties file. The only change is in jdk.

 

 

On 18/10/2019 3:31 μ.μ., Theodoros Theodoropoulos wrote:

I agree its totally strange Demian.

(I also don't want to fill the mailing list with long emails. If you find that all this conversation could be of any use to others, feel free to forward it accordingly)

 

So, with openjdk11 in place and the new solrmarc:

 

- With "solrmarc.path = /data/vufind-prod/vufind/local/import" in ./local/import/import.properties the problem goes away.
[1] gives:

DEBUG [main] (Boot.java:638) - Number of homeDirStrs: 3

DEBUG [main] (Boot.java:641) - homeDirStrs[0]: /data/vufind-prod/vufind/local/import

DEBUG [main] (Boot.java:645) - homeDirStrs[1]: /home/ubuntu/../../data/vufind-prod/vufind/local/import

(only local import dir is passed to solrmarc_core.jar) - Maybe the directory where solrmarc_core.jar is stored is used always? See homeDirStrs that has a value of 3!

 

- With "solrmarc.path = /data/vufind-prod/vufind/import" in ./local/import/import.properties, the problem appears again (without any other changes!)

[1] gives:

DEBUG [main] (Boot.java:638) - Number of homeDirStrs: 3

DEBUG [main] (Boot.java:641) - homeDirStrs[0]: /data/vufind-prod/vufind/import

DEBUG [main] (Boot.java:645) - homeDirStrs[1]: /home/ubuntu/../../data/vufind-prod/vufind/local/import

(both default and local import dirs are explicitly passed to solrmarc_core.jar), but maybe the default dir is used anyways and this produces the error?

 

- With "solrmarc.path = /data/vufind-prod/vufind/local/import|/data/vufind-prod/vufind/import" in ./local/import/import.properties, the problem appears as well (without any other changes!)

[1] gives:

DEBUG [main] (Boot.java:638) - Number of homeDirStrs: 4

DEBUG [main] (Boot.java:641) - homeDirStrs[0]: /data/vufind-prod/vufind/local/import

DEBUG [main] (Boot.java:645) - homeDirStrs[1]: /data/vufind-prod/vufind/import

(both default and local import dirs are explicitly passed to solrmarc_core.jar, one of them twice)

 

Also, i've tested with all possible solrmarc/marc4j combinations and realized that they all produce the same results.

So the error appears ONLY when openjdk-11 is used!
With openjdk-8, no matter what solrmarc/marc4j you use, with any of the above three combinations, you don't get errors.

 

Does this give you any more clues? Have you tried indexing marc with openjdk-11?

 

Best,
Theodoros

 

[1] /usr/lib/jvm/default-java/bin/java -Xms512m -Xmx512m -DentityExpansionLimit=0 -Duser.timezone=UTC  -jar /data/vufind-prod/vufind/impt/solrmarc_core_3.3.jar /data/vufind-prod/vufind/local/import/import.properties -solrj /data/vufind-prod/vufind/solr/vendor/dist/solrj-lib /data/vufind-prod/vufind_marc_updates/updated_2019-10-17.mrc

 

On 18/10/2019 2:12 μ.μ., Demian Katz wrote:

Theodoros,

 

I’m glad you’ve found a workaround, but I don’t think the problem is fully explained yet. The syntax you identify is correct, and every VuFind installation builds an import.properties file like the one you had as part of the setup process. This is how SolrMarc knows to first look in $VUFIND_LOCAL_DIR and then fail over to $VUFIND_HOME when searching for import files. If you remove the $VUFIND_HOME piece, I imagine the importer may have trouble finding some things. In fact, I’m surprised that it works at all, unless you have copied a lot of files into your local directory.

 

I’d suggest taking another look at this. Please let me know if you need my help in further troubleshooting!

 

- Demian

 

From: Theodoros Theodoropoulos <th...@physics.auth.gr>
Sent: Thursday, October 17, 2019 4:02 PM
To: Demian Katz <demia...@villanova.edu>; vufin...@lists.sourceforge.net
Subject: Re: [EXTERNAL] java errors during .mrc indexing in VUFind 5.1 and Ubuntu 18.04 LTS (openjdk 11.x)

 

Culprit found!

 

Indeed I downloaded latest master, changed paths, and I didn't get that error (of course I got several other errors, but they were expected).

 

After some tests I found that the problem was hiding in my vufind/local/import/import.properties

 

More specifically in my vufind/local/import/import.properties, I had:

solrmarc.path = /data/vufind-prod/vufind/local/import|/data/vufind-prod/vufind/import (both valid directories, the first with custom files, the second with the solrmarc.jar file and default .properties files)

 

When I removed the part after | ,the problem was solved!

 

To my defense, I'm almost certain I saw this syntax somewhere (probably a long time ago) and it is used in my current production system without producing problems. On second thought, this variable was probably designed to hold only one path, so I wonder how it worked until now...

 

Anyways, it's highly unlikely that anyone out there will have a similar issue, but if so, this thread might come handy.

 

Thanks for your time and patience,

Theodoros

 

On 17/10/2019 8:23 μ.μ., Demian Katz wrote:

Theodoros,

 

The error should not be related to the Solr version – SolrMarc communicates with Solr over HTTP and so it is not tightly bound to the Solr version.

 

Out of curiosity, if you just check out the current master branch of VuFind to a separate directory and run the import script there, does it give the same error?

 

- Demian

 

From: Theodoros Theodoropoulos <th...@physics.auth.gr>
Sent: Thursday, October 17, 2019 12:14 PM
To: Demian Katz <demia...@villanova.edu>; vufin...@lists.sourceforge.net
Subject: Re: [EXTERNAL] java errors during .mrc indexing in VUFind 5.1 and Ubuntu 18.04 LTS (openjdk 11.x)

 

Yeap I deleted them!

 

I've also tried hardcoding the java version and the solrmarc version:

/usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms512m -Xmx512m -DentityExpansionLimit=0 -Duser.timezone=UTC  -jar /data/vufind-prod/vufind/import/solrmarc_core_3.3.jar /data/vufind-prod/vufind/local/import/import.properties -solrj /data/vufind-prod/vufind/solr/vendor/dist/solrj-lib /data/vufind-prod/vufind_marc_updates/updated_2019-10-10.mrc

and I still get the same error... Any chance it's related to the solr version (v7.3.1) that's included in vufind 5.1.1? If so,I can always downgrade and use java 8...

 

Or, if you have another idea, I would be happy to try it before giving up...

 

Theodoros

 

On 17/10/2019 4:27 μ.μ., Demian Katz wrote:

Did you delete the old files at the same time that you added the new ones? If the old versions are still there, the script might be loading the wrong .jar file.

 

- Demian

 

From: Theodoros Theodoropoulos <th...@physics.auth.gr>
Sent: Thursday, October 17, 2019 9:24 AM
To: Demian Katz <demia...@villanova.edu>; vufin...@lists.sourceforge.net
Subject: Re: [EXTERNAL] java errors during .mrc indexing in VUFind 5.1 and Ubuntu 18.04 LTS (openjdk 11.x)

 

Hmm... I'm probably missing something, because I've copied the newer files, switched to openjdk 11 and now I'm getting again the initial error...

Any ideas?

 

On 17/10/2019 3:47 μ.μ., Demian Katz wrote:

Yes, that is exactly right!

 

From: Theodoros Theodoropoulos <th...@physics.auth.gr>
Sent: Thursday, October 17, 2019 8:46 AM
To: Demian Katz <demia...@villanova.edu>; vufin...@lists.sourceforge.net
Subject: Re: [EXTERNAL] java errors during .mrc indexing in VUFind 5.1 and Ubuntu 18.04 LTS (openjdk 11.x)

 

Thank you Demian for your prompt reply!

 

Pull request #1402, upgrades the following two files:

import/lib/marc4j-2.8.3.jar → import/lib/marc4j-2.9.jar
import/solrmarc_core_3.2.jar → import/solrmarc_core_3.3.jar

 

So I suppose this should be enough, no? If anything else is required (like solrmarc configuration changes), please advise!

 

Thanks in advance,

Theodoros

 

 

On 17/10/2019 3:10 μ.μ., Demian Katz wrote:

Theodoros,

 

This is a known issue with VuFind 5.x and earlier – the SolrMarc that is bundled with those releases is not compatible with newer versions of Java. This was corrected in release 6.0, when we upgraded to a newer SolrMarc release.

 

It might be worth adding a note to the installation instructions indicating that for older versions of VuFind, it may be necessary to use older versions of Java; but this might be excessive, as my assumption is that the installation instructions are most commonly used to install the most current release.

 

If you need to run VuFind 5 with Java 11, it should be possible to manually upgrade SolrMarc by replacing a few .jar files – it’s not a particularly complicated procedure, and I’m happy to help with that if there is a need.

 

Thanks for bringing this up, and sorry for any inconvenience/confusion!

 

- Demian

 

From: Theodoros Theodoropoulos <th...@physics.auth.gr>
Sent: Thursday, October 17, 2019 7:54 AM
To: vufin...@lists.sourceforge.net
Cc: Demian Katz <demia...@villanova.edu>
Subject: [EXTERNAL] java errors during .mrc indexing in VUFind 5.1 and Ubuntu 18.04 LTS (openjdk 11.x)

 

Hello Demian and all,

Following the installation guide for Ubuntu, "apt-get -y install default-jdk" installs openjdk 11.0.4 in Ubuntu 18.04 LTS.

I've verified that (at least) for VUFind v5.1.x, during indexing of the .mrc files, this jdk produces the following fatal errors:

 INFO [main] (PropertyUtils.java:313) - Opening file (instead of 1 other options): /data/vufind-prod/vufind/import/vufind.properties

 INFO [main] (PropertyUtils.java:313) - Opening file (instead of 1 other options): /data/vufind-prod/vufind/import/vufind.properties

FATAL [main] (Boot.java:212) - ERROR: Error while invoking main method in specified class: org.solrmarc.driver.IndexDriver

java.lang.LinkageError: loader org.solrmarc.driver.Boot @4617c264 attempted duplicate interface definition for org.ini4j.Persistable. (org.ini4j.Persistable is in unnamed module of loader org.solrmarc.driver.Boot @4617c264, parent loader 'app')

        at java.base/java.lang.ClassLoader.defineClass1(Native Method)

        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)

        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:877)

        at org.solrmarc.driver.Boot.loadClass(Boot.java:141)

        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)

        at java.base/java.lang.ClassLoader.defineClass1(Native Method)

        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)

        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:877)

        at org.solrmarc.driver.Boot.loadClass(Boot.java:141)

        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:575)

        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)

        at java.base/java.lang.Class.getDeclaredMethods0(Native Method)

        at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)

        at java.base/java.lang.Class.getDeclaredMethods(Class.java:2309)

        at org.solrmarc.index.extractor.methodcall.MethodCallManager.add(MethodCallManager.java:115)

        at org.solrmarc.index.extractor.methodcall.MethodCallManager.add(MethodCallManager.java:106)

        at org.solrmarc.index.extractor.methodcall.AbstractMethodCallFactory.addMethodsFromClasses(AbstractMethodCallFactory.java:36)

        at org.solrmarc.index.extractor.impl.java.JavaValueExtractorFactory.<init>(JavaValueExtractorFactory.java:16)

        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)

        at java.base/java.lang.Class.newInstance(Class.java:584)

        at org.solrmarc.index.indexer.ValueIndexerFactory.createExtractorFactories(ValueIndexerFactory.java:516)

        at org.solrmarc.index.indexer.ValueIndexerFactory.initialize(ValueIndexerFactory.java:79)

        at org.solrmarc.driver.IndexDriver.execute(IndexDriver.java:92)

        at org.solrmarc.driver.IndexDriver.main(IndexDriver.java:69)

        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.base/java.lang.reflect.Method.invoke(Method.java:566)

        at org.solrmarc.driver.Boot.invokeMain(Boot.java:201)

        at org.solrmarc.driver.ConfigDriver.main(ConfigDriver.java:114)

        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

If one installs openjdk-8 (ie by using "apt install openjdk-8-jdk-headless") and make that the default java, the above errors dissapear.

(openjdk 9 and 10 are not available anymore, so I cannot verify which was the last openjdk that worked without issues...)

 

If you can verify my findings:

- The code part that produces the error should be fixed (as java 8 doesn't receive free public updates as of January 2019)

OR, if this is difficult (ie due to solrmarc dependencies),

- A note regarding the supported java version should be put in the installation guide for the latest Ubuntu LTS version (you have a similar warning in https://vufind.org/wiki/installation:fedora), as well as in the https://vufind.org/wiki/installation:requirements

 

Best regards,

Theodoros Theodoropoulos

 

 

 

--

Theodoros Theodoropoulos
Library Services Support Team
Library and Information Centre
Aristotle University of Thessaloniki
Tel. :+30 2310 995344
email:libsu...@lib.auth.gr

 

[Demian Katz]

Theodoros,

 

Thanks for this explanation – I’m glad we’ve gotten to the bottom of the problem, though it’s still a bit of a mystery why different JDK versions handle symlinks differently. In any case, when I have a bit more time, I’ll see if I can reproduce this problem on my test server now that we know what the root cause is, and maybe we can compensate for it in the code somehow. In any case, at least this is a warning that symlinks and SolrMarc may not always work together as expected, and we have a valid workaround for the moment.

 

- Demian

 

From: Theodoros Theodoropoulos <th...@physics.auth.gr>
Sent: Wednesday, October 23, 2019 5:51 AM
To: Demian Katz <demia...@villanova.edu>
Cc: solrma...@googlegroups.com
Subject: Re: [EXTERNAL] java errors during .mrc indexing in VUFind 5.1 and Ubuntu 18.04 LTS (openjdk 11.x)

 

Demian, I have the exact same openjdk and JRE!

 

The error is obviously related to paths found in ./import.properties, but does not relate to the directory the script is run from.
I changed my current dir to VUFIND_HOME/.. and run [1] again.
Relative paths in the debug messages were gone (see bold below), but I got the same errors for the same solrmarc.path combinations in ./local/import/import.properties

 

See:

DEBUG [main] (BootableMain.java:117) - Adding directory: /data/vufind-prod/vufind/import

DEBUG [main] (BootableMain.java:117) - Adding directory: /data/vufind-prod/vufind/local/import

DEBUG [main] (Boot.java:638) - Number of homeDirStrs: 3

DEBUG [main] (Boot.java:641) - homeDirStrs[0]: /data/vufind-prod/vufind/import

DEBUG [main] (Boot.java:645) - homeDirStrs[1]: /data/vufind-prod/vufind/local/import

[...]

FATAL [main] (Boot.java:212) - ERROR: Error while invoking main method in specified class: org.solrmarc.driver.IndexDriver

java.lang.LinkageError: loader org.solrmarc.driver.Boot @4617c264 attempted duplicate interface definition for org.ini4j.Persistable. (org.ini4j.Persistable is in unnamed module of loader org.solrmarc.driver.Boot @4617c264, parent loader 'app')

And then it struck me!

/data/vufind-prod/vufind is a symbolic link to /data/vufind-prod/vufind-5.1 (it has always been like that), so when I use

- "solrmarc.path = /data/vufind-prod/vufind-5.1/local/import|/data/vufind-prod/vufind-5.1/import"

Got:

DEBUG [main] (Boot.java:638) - Number of homeDirStrs: 3

DEBUG [main] (Boot.java:641) - homeDirStrs[0]: /data/vufind-prod/vufind-5.1/import

DEBUG [main] (Boot.java:645) - homeDirStrs[1]: /data/vufind-prod/vufind-5.1/local/import

And the problem is fixed in both JDKs!!

 

So, even though "solrmarc.path = /data/vufind-prod/vufind/local/import|/data/vufind-prod/vufind/import" ends up in the same directories, for some reason, solrmarc in JDK11 probably(?) follows the symlink, adding 'another' directory to homeDirStrs, that doesn't know it should be deduplicated and produces "attempted duplicate interface definition for org.ini4j.Persistable" error.

I don't know if any of this is valid or makes any sense or gives you and indication and if you can replicate it (I'm so confused with all those changes and checks), but it kind of explains why you never encountered this issue, although you do all your development in the same JDK environment.
If you can replicate it, then it should be handled in solrmarc (or explained somewhere in the docs and/or in import.properties file)... If not, then it's something specific to my installation/configuration and I give up. I found a workaround that works for me and I will not bother you anymore with this issue :)

Best,
Theodoros

[Demian Katz]

Hello again,

 

I was able to easily reproduce this problem on my test server, and I have opened a SolrMarc issue with detailed instructions on how I did it:

 

https://github.com/solrmarc/solrmarc/issues/87

 

I’m interested to know if anyone with more Java familiarity than me has an easy explanation/solution based on the stack trace and existing code… but if no one else is able to look at this, I’m willing to spend more time on it when my schedule opens up again!

 

- Demian

--
You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to solrmarc-tec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/solrmarc-tech/DM6PR03MB4922D9FBA67B7751194EADECE86B0%40DM6PR03MB4922.namprd03.prod.outlook.com.