Solr Indexing Hangs

[Micheal Grimard]

For some reason recently when indexing items into solr using a data import handler it seems to hang after finishing for a long period of time (in as few as 25 records being indexed).

In the catalina.out log file this is what it says right before hanging:

Apr 27, 2017 3:00:32 PM org.apache.solr.search.SolrIndexSearcher <init>

INFO: Opening Searcher@52ed2651 main

Apr 27, 2017 3:00:32 PM org.apache.solr.core.QuerySenderListener newSearcher

INFO: QuerySenderListener sending requests to Searcher@52ed2651 main{StandardDirectoryReader(segments_f6e:79154:nrt _kd9(4.2):C256418/25 _kda(4.2):C6 _kdb(4.2):C8 _kdc(4.2):C11)}

Apr 27, 2017 3:00:32 PM org.apache.solr.core.QuerySenderListener newSearcher

INFO: QuerySenderListener done.

Apr 27, 2017 3:00:32 PM org.apache.solr.core.SolrCore registerSearcher

INFO: [collection1] Registered new searcher Searcher@52ed2651 main{StandardDirectoryReader(segments_f6e:79154:nrt _kd9(4.2):C256418/25 _kda(4.2):C6 _kdb(4.2):C8 _kdc(4.2):C11)}

Apr 27, 2017 3:00:32 PM org.apache.solr.update.DirectUpdateHandler2 commit

INFO: end_commit_flush

I tried increasing the ram of the jvm (from 1gb to 4gb) but it doesn't reflect it in the solr admin page - does it use a seperate jvm than tomcat? Where do I change solr's xmx and xms settings?

Any ideas appreciated.

[dp...@metro.org]

Which version of Solr? If running 4.10 or less and a standard islandora installation, should be running by default inside the same tomcat JVM fedora, etc are running, assuming you have a single machine install of course. its a .war so if it shares the same deployment directory than fedora, gsearch, then you know for sure.

I feel what you are seeing is a hard commit. Means: when Solr writes its transaction log (it keeps all your updates in a log and memory) to Disk and spins a new searcher (fast disk?). By default (depending again on your config and version) Solr also does a core optimization, which will takes some extra time. I suggest doing some tunning in your solr config file, avoid a hard commit immediately, do soft commit and wait for more document updates before doing the hard commit, and don't ever optimize your core automatically, you can always trigger that via cron or manually if needed or a higher document count limit.

1 Gbyte is pretty low for Solr but a 4 Gbyte increase is huge, why that jump? Does that leave enough memory for other tasks?. Do you have enough memory? Want to share your JVM options? I personally feel safer when dealing with predefined memory limits: one of our servers.

-Xms7984m -Xmx7984m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=65 

you see upper and lower limits are the same, I use G1GC as garbage collecting algorithm/strategy and allow GC to pause only for 200 ms. Works pretty well but don't copy blindly, we did a lot of testing before coming to that solution.

If you don't see the changes, then chances are you are changing in the wrong spot or not restarting tomcat after the changes

some thoughts here also https://wiki.apache.org/solr/SolrPerformanceProblems

Best

[Micheal Grimard]

Yeah, it's as you say: A .war file inside the tomcat directory, but it is version 4.2.0.

This 'hang' after doing an import through solr manually (using solr's data-import) is fairly new - we didn't have an issue before, not really sure what made the change (maybe the index size past a critical mass for our settings?). We were just using the default 1gb before, but during large ingestion we were running out of memory so we increased the server's memory from 4gb to 8gb and wanted to increase the amount we assign to the jvm as well.

This is what I have set up in the server's profile to be used when starting up tomcat:

PATH=$HOME/.rbenv/bin:/opt/java/bin:$PATH:$HOME/bin

export FEDORA_HOME=/usr/local/fedora

export CATALINA_HOME=/usr/local/fedora/tomcat

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CATALINA_HOME/lib

export LD_LIBRARY_PATH

export JAVA_OPTS="-Xms2048m -Xmx4096m -Djavax.net.ssl.trustStore=/usr/local/fedora/server/truststore -Djavax.net.ssl.trustStorePassword=tomcat"

export JAVA_HOME=/usr/lib/jvm/java-8-oracle

export JRE_HOME=$JAVA_HOME/jre

export J2SDKDIR=$JAVA_HOME

export J2REDIR=$JAVA_HOME/jre

export KAKADU_LIBRARY_PATH=/opt/adore-djatoka-1.1/lib/Linux-x86-64

But it doesn't seem to be reflected in the solr admin page that the alloted memory has been increased. Definitely restarting tomcat - even tried restarting the whole server.

[dp...@metro.org]

Ok, 4.2 is slower but ok. Optimizing your Core takes more time as the index grows, so that could be the issue and for sure commits also.

Look at the  https://wiki.apache.org/solr/DataImportHandler (that is the one you are using right?) and look at the commit and optimize options in the full import section. Also, worth trying https://wiki.apache.org/solr/DataImportHandler#interactive debug option allows you to inspect further. Are you logging Solr correctly? Do you get enough info there?

About the memory increase, it is weird that your JAVA OPTS are not taking place, can you share a screenshot of your Solr admin Web interface (the dashboard part)? Its the OS tomcat right or the one shipped with fedora?

Best

Diego Pino 

Metro.org

[Micheal Grimard]

Thanks for the links and assistance. I've attached a screenshot. It's an ubuntu server, using the tomcat installation as explained in the Islandora Installation guide. I've used the commit option and the verbose, but I didn't realize about the debug option - I'll give that a try. It would be nice to figure out why the memory in the jvm isn't increasing.

[dp...@metro.org]

I suspect you are having issues with JAVA opts being overwritten in some later steps (like in some Catalina.sh file) because you are clearly getting 1024 max and min there or you are editing a not used JVM opts settings file (believe, happens). Using the standard islandora installation, normally means using the internal fedora provided Tomcat? usr/local/fedora/tomcat ? which means starting via /usr/local/fedora/tomcat/bin/startup.sh?

Make sure nothing is redefining those opts in some later steps. Give also https://github.com/Islandora-Labs/islandora_vagrant_base_box/tree/master/scripts a look. That one is a installation using OS tomcat (ubuntu) but a good place to look at all the possible places tomcat/jvm can interact (and to learn how to deploy a full fedora islandora mix when in doubt).

Good luck

Diego

[Micheal Grimard]

I verified that tomcat is loading the JAVA opts I have set - when I echo them, it's what I'd expect and when I run jps -lvm I get the following:

4099 org.apache.catalina.startup.Bootstrap start -Djava.util.logging.config.file=/usr/local/fedora/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms2048m -Xmx4096m -Djavax.net.ssl.trustStore=/usr/local/fedora/server/truststore -Djavax.net.ssl.trustStorePassword=tomcat -Djava.awt.headless=true -Dkakadu.home=/opt/adore-djatoka-1.1/bin/Linux-x86-64 -Djava.library.path=/opt/adore-djatoka-1.1/lib/Linux-x86-64 -DLD_LIBRARY_PATH=/opt/adore-djatoka-1.1/lib/Linux-x86-64 -Djava.endorsed.dirs=/usr/local/fedora/tomcat/endorsed -Dcatalina.base=/usr/local/fedora/tomcat -Dcatalina.home=/usr/local/fedora/tomcat -Djava.io.tmpdir=/usr/local/fedora/tomcat/temp

12122 sun.tools.jps.Jps -lvm -Dapplication.home=/usr/lib/jvm/java-8-oracle -Xms8m

I'm not sure exactly what is going on.

[Micheal Grimard]

Nope. You were exactly right. For some reason there was apparently and errant 'export JAVA_OPTS' in the start-up.sh file that was defaulting the values.

On Thursday, April 27, 2017 at 5:05:05 PM UTC-4, dp...@metro.org wrote:

[Micheal Grimard]

The bump in memory however doesn't seem to have fixed the initial problem of it hanging for a long period of time after indexing.

[dp...@metro.org]

Hi Michael, ok, one issue fixed =) I'm pretty sure your Solr config settings are forcing some type of core optimizing routine. If you want I can give your settings a review if you can share them and explain your data import handler worklfow. But also, if not, look how-when you are opening a new searcher. That is what takes more time in any indexing operation (other than the core optimize routine)

Best

Diego

[Micheal Grimard]

Which solr config settings would you need to see? We're using https://github.com/discoverygarden/basic-solr-config as a basis for ingesting drupal items into our index. I'm running the process through the solr admin webapp at the moment. To be honest, I'm a little out of my element when it comes to talk about a new searcher - other than following the instructions for setting up islandora, fedora, and solr and then using the starting point of the discovery garden stuff I don't think I've made any adjustments that would impact how the searcher would work from default.

[dp...@metro.org]

Michael, i missed your last message sorry. solrconfig.xml, https://cwiki.apache.org/confluence/display/solr/Configuring+solrconfig.xml

best