how to repair corrupt solr statistics

[Francis Brouns]

Hi all,

since we are being flooded by search requests, several SOLR errors started occuring. These errors occurs both in search and statistics and it seems like these got corrupted. If I understand the manual correctly, the search solr index will repair itself because we enabled the respective cron tabs. Is there a way to repair the statistics index?

• Error opening new searcher Caused by: java.io.IOException: Map failed
  
• java.io.IOException: Map failed: statistics/data/index/_1p90_Lucene410_0.dvd
  
•  auto commit error...:java.io.IOException: Map failed:
  

The files referenced in the log files (e.g./dspace/solr/statistics/data/index/_1p90_Lucene410_0.dvd) do exist and have the correct permissions.

Running DSpace 5.4, solr statistics, jspui, tomcat 7, java 7, Sles Linux, Oracle.

kind regards,

Francis Brouns

Errors thrown are:

2017-10-19 00:30:59,054 ERROR org.apache.solr.core.SolrCore @ org.apache.solr.common.SolrException: Error opening new searcher

at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)

...

Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/dspace/solr/search/data/index/_ggq.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 623947081 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]

at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)

at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:224)

2017-10-19 01:31:09,769 ERROR org.apache.solr.core.SolrCore @ java.io.IOException: Map failed: MMapIndexInput(path="/dspace/solr/statistics/data/index/_1osu_Lucene410_0.dvd") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 303329851 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]

at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)

at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:224)

at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:199)

2017-10-26 01:45:11,641 INFO  org.apache.solr.update.UpdateHandler @ start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}

2017-10-26 01:45:12,418 ERROR org.apache.solr.update.CommitTracker @ auto commit error...:java.io.IOException: Map failed: MMapIndexInput(path="/dspace/solr/statistics/data/index/_1p90_Lucene410_0.dvd") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 305990079 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]

at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)

at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:224)

at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:199)

at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:198)

at org.apache.lucene.codecs.lucene410.Lucene410DocValuesProducer.<init>(Lucene410DocValuesProducer.java:121)

at org.apache.lucene.codecs.lucene410.Lucene410DocValuesFormat.fieldsProducer(Lucene410DocValuesFormat.java:181)

at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:238)

[Tom Desair]

Hi Francis,

Try this:

1. Stop the traffic to your repository but leave Tomcat running. Then take of copy of the /dspace/solr/statistics/data/index directory while Tomcat is running to /dspace/statistics-backup/index (not sure if this is really necessary).

2. Run the following command on that copy:  java -cp /dspace/webapps/solr/WEB-INF/lib/lucene-core-4.10.2.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /dspace/statistics-backup/index -fix 

3. This will run for a while and needs a lot of disk space. You need at least the size of the statistics core as free disk space.

4. Once the CheckIndex tool is ready and repaired the index.Stop Tomcat again and move the repaired index (in /dspace/statistics-backup/index) to the SOLR statistics directory (/dspace/solr/statistics/data/index)

The CheckIndex tool will report how many records are lost when repairing the index. There is no way to restore them.

Best regards,

Tom

 

Tom Desair 250-B Suite 3A, Lucius Gordon Drive, West Henrietta, NY 14586 Gaston Geenslaan 14, Leuven 3001, Belgium www.atmire.com

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscribe@googlegroups.com.
To post to this group, send email to dspac...@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[Francis Brouns]

Dear Tom,

thanks for your quick reply. Unfortunately, the CheckIndex did not find any error in the index, not in statistics, not in search. Nevertheless, logs keep reporting errors both for search and statistics. We now also run into problems because the browse indexes no longer are up to date. For example author names being shown in the browse index, result in a No index error message when clicking the name. Similarly for items.

Is it safe to complete delete the solr/search directory and have generate all indexes again?

I guess that for statistics this is not an option. Should I try to export statistics and import statistics?

best wishes,

Francis Brouns

Open Universiteit of the Netherlands

[Francis Brouns]

Dear Tom,

in addition I notice that in statistics/tlog there are 168 logfiles, while in search/tlog there are 7 logs. That also indicates something is not working properly.

Is it correct that index files "disappear" or only exist for a particular day? I no longer can find back indexes that are mentioned in log files from last week. E.g. last week an error occurred on /statistics/data/index/_1p90_Lucene410_0.dvd. On that day we checked and the file was present. Today this file no longer exists. Similarly, for files in the search index. That makes it really hard to track down problems.

2017-10-26 01:31:09,010 ERROR org.apache.solr.servlet.SolrDispatchFilter @ null:java.io.IOException: Map failed: MMapIndexInput(path="/dspace/solr/statistics/data/index/_1p90_Lucene410_0.dvd")

at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)

Any pointers on how to figure out what is wrong, is highly appreciated.

kind regards,

Francis Brouns