error opening new searcher during crontab jobs optimising index and filtermedia

[Francis Brouns]

Hi all,

the upgrade from our DSpace 1.8.3 server to DSpace 5.4 seemed to go fine, except that I now see an error in the solr.log and dspace.log when crontabs are executed during the night. The crontabs are scheduled, following the example in the manual. It seems that the same error is repeated: error opening new searcher. Again it is referring to MMAP issues and ulimit memory settings.We encountered many unrecoverable MMAP issues when we tried the upgrade, using JDK1.8 and Tomcat 8. We now installed Oracle JDK 1.7.0_79 and Tomcat 7.0.64, Suse Enterprise Server, Oracle 11g, upgrading a DSpace 1.83 release without SOLR statistics (only using legacy statistics).

The errors occur during the crontabs to optimise the SOLR index (index-discovery –o; stats-util –o) and again during the filter-media crontab. It seems that the errors refer to writing the discovery index, but it seems that the index is continuously being updated and I can't detect any obvious problems with the handles that are reported in the errors. 
 

Our repository is small, only around 6000 items. The server has 5Gb RAM, 1GB allocated to Tomcat via catalina_opts and 512Mb to the dspace script. Should I allocate more to Tomcat?

errors in solr.log

2015-12-23 00:30:59,853 ERROR org.apache.solr.core.SolrCore @ org.apache.solr.common.SolrException: Error opening new searcher

               at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)

 

Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/dspace/solr/search/data/index/_sxp.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 423723626 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]

               at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)

               at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:224)

errors in dspace.log

2015-12-23 03:02:19,822 ERROR org.dspace.discovery.SolrServiceImpl @ Error while writing item to discovery index: 1820/5148 message:Map failed: MMapIndexInput(path="/dspace/solr/search/data/index/_sxp.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 423723626 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]

org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Map failed: MMapIndexInput(path="/dspace/solr/search/data/index/_sxp.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 423723626 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]

at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)

at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)

best wishes,

Francis Brouns