Gsearch setup: Caught between NoSuchDirectoryException and IndexNotFoundException

[Paddy McCann]

I've also asked this on StackOverflow at http://stackoverflow.com/questions/25178756/caught-between-nosuchdirectoryexception-and-indexnotfoundexception

Hi, I'm new to working with [Fe/Islan]dora and I'm trying to set up Islandora 7.1.3 as per https://wiki.duraspace.org/display/ISLANDORA713/Islandora. Fedora Commons and Drupal are working, but I'm having an issue with Solr 4.2/fedoragsearch 2.6.

Following the instructions athttps://wiki.duraspace.org/display/ISLANDORA713/Installing+Solr+and+GSearch, I ran into a problem at step 14.

When I tried to update the index at [url]/fedoragsearch/rest?operation=updateIndex I got this error:

IndexReader open error indexName=FgsIndex : ; nested exception is: org.apache.lucene.store.NoSuchDirectoryException: directory 'path/to/solr/collection1/data/index' does not exist

That seemed straightforward, so I shut down tomcat and created data/index within the existing collection1 directory. Restarting tomcat and trying again gave me:

IndexReader open error indexName=FgsIndex : ; nested exception is: org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.SimpleFSDirectory@/path/to/solr/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68aed52c: files: []

A quick search for this error suggested if the index directory is present Solr expects it not to be empty, with the solution being to delete the directory and let Solr create it. But that gives me the previous error.

I've tried with and without the collection1/data directory present (NoSuchDirectoryException either way) and I've set the permissions on collection1 (and collection1/data, if present) to 777 while I try to work this out.

Any help greatly appreciated.

[Linda Richter]

This is what works for me:

1. Stop tomcat
2. Delete the index directory
3. Start tomcat

The index directory and the segments file will be created on startup.

[Paddy McCann]

Thanks Linda

I tried that, but ended up with the NoSuchDirectoryException again.

Paddy

[Phil R]

Ok, I've spent a ton of time trying to figure out this issue myself.  I would try to reindex solr and get an error, so I would delete either the contents of index, or the index folder itself and then get another error.  This is the process I'm using now for reindexing solr.

{{{

1. tail -f catalina.out (I do this in a separate terminal so I can monitor solr/gs activity)

2. browse to example.com:8080/fedoragsearch/rest

3. click 'updateIndex'

4. click 'updateIndex createEmpty'

5. sudo service solr stop (stop solr if it isn't already.  hopefully you've set up init.d for solr by now, I don't remember the actual command for this without it)

6. sudo cp -Rf /path/to/index /path/to/backup/index (backup your index)

7. browse to example.com:8080/solr/update?stream.body=%3Cdelete%3E%3Cquery%3E*%3A*%3C%2Fquery%3E%3C%2Fdelete%3E (THIS IS THE MAGIC THAT PROPERLY DELETES YOUR SOLR INDEX.  make sure you have a backup, and are otherwise prepared)

8. sudo service solr start (start it up)

9. click 'updateIndex from FoxmlFiles'

10. watch your tail!

}}}

I'd highly suggest rebuilding fedora before you reindex solr.  It's easy and it cleans everything up:

sudo service fedora stop
sudo su - fedora
$FEDORA_HOME/server/bin/fedora-rebuild.sh (I do both fedora and mysql rebuilds)
sudo service fedora start

[Phil R]

I used the following code to set up solr init.d.

https://github.com/jbusby/solr-initd

[Paddy McCann]

Thanks Phil

There were some issues with regard to SSL in the logs, so I went back and reinstalled Fedora and Solr/Gsearch and eliminated those, but the main error as described above remains. I wondering now if this is related to messaging - ingested items are not being indexed by Solr and the following error is logged when shutting down tomcat:

ERROR 2014-08-13 10:59:07,881 (MessagingClient) Messaging Exception encountered attempting to stop Messaging Client: fedoragsearch0. Exception message: The JMS connection has failed: java.io.EOFException

com.yourmediashelf.fedora.client.messaging.MessagingException: The JMS connection has failed: java.io.EOFException

at com.yourmediashelf.fedora.client.messaging.JMSManager.close(JMSManager.java:680)

at com.yourmediashelf.fedora.client.messaging.MessagingClient.stop(MessagingClient.java:320)

at dk.defxws.fedoragsearch.server.UpdateListener.destroy(UpdateListener.java:127)

at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1436)

at org.apache.catalina.core.StandardWrapper.stop(StandardWrapper.java:1784)

at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4872)

at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:936)

at org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:1359)

at org.apache.catalina.startup.HostConfig.stop(HostConfig.java:1330)

at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:326)

at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)

at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)

at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1110)

at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:468)

at org.apache.catalina.core.StandardService.stop(StandardService.java:604)

at org.apache.catalina.core.StandardServer.stop(StandardServer.java:788)

at org.apache.catalina.startup.Catalina.stop(Catalina.java:662)

at org.apache.catalina.startup.Catalina.start(Catalina.java:629)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:616)

at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

Caused by: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1362)

at org.apache.activemq.ActiveMQConnection.stop(ActiveMQConnection.java:545)

at com.yourmediashelf.fedora.client.messaging.JMSManager.close(JMSManager.java:674)

... 23 more

Caused by: java.io.EOFException

at java.io.DataInputStream.readInt(DataInputStream.java:392)

at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)

at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:227)

at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219)

at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)

at java.lang.Thread.run(Thread.java:679)

My installation is very much as per the docs - even moreso than previously since I reinstalled Fedora and Solr/Gsearch. Not knowing too much about JMS, I accepted the default option for the messaging provider after choosing to enable messaging during Fedora installation. Any pointers on the messaging issue appreciated.

Thanks,

Paddy

[Phil Redmon]

So are you able to reindex solr? Have you followed the steps I indicated above?

> --
> For more information about using this group, please read our Listserv
> Guidelines: http://islandora.ca/content/welcome-islandora-listserv
> ---
> You received this message because you are subscribed to the Google Groups
> "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to islandora+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/islandora.
> For more options, visit https://groups.google.com/d/optout.



--
phil

[A L]

Hi Paddy,

I have seen that frustrating error before for several instances at different times with a different root cause.

1. One instance was when I copied solrconfig file from development server to a test server AS IS and I got all those errors that you got. After constant struggle, I realized I had data directory path listed as "C:" where in test server, I had all Solr configuration files located in directory "D:". So, make sure check the <dataDir> tag value in solrconfig file which should be located under $FEDORA_HOME/solr/collection1/conf

2. Another instance, solr.xml was missing in $FEDORA_HOME/tomcat/conf/catalina/localhost.

My solr.xml looks like this (source:Step-4 under Installing Solr section - https://wiki.duraspace.org/pages/viewpage.action?pageId=34644675)

<Context docBase="/opt/solr/apache-solr-1.4.1.war" debug="0" crossContext="true">

 

<Environment name="solr/home" type="java.lang.String" value="/usr/local/fedora/gsearch_solr/solr" override="true" />

 

</Context>




[Paddy McCann]

Hi,

Phil, I tried those steps. Currently watching catalina.out and fedoragsearch.daily.log in two terminal windows. Any of the updateIndex actions (at any of the points in the process you described) result in one of the two errors described in my original post, depending on the presence of the index directory. Full log entries (from catalina.out):

With index directory present:

dk.defxws.fedoragsearch.server.errors.GenericSearchException: Wed Aug 13 16:22:11 BST 2014 IndexReader open error indexName=FgsIndex :
; nested exception is:
 org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.SimpleFSDirectory@/home/islandora/fedora/solr/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@7df4591e: files: []
 at dk.defxws.fgssolr.OperationsImpl.getIndexReaderAndSearcher(OperationsImpl.java:719)
 at dk.defxws.fgssolr.OperationsImpl.updateIndex(OperationsImpl.java:267)
 at dk.defxws.fedoragsearch.server.GenericOperationsImpl.updateIndex(GenericOperationsImpl.java:556)
 at dk.defxws.fedoragsearch.server.RESTImpl.updateIndex(RESTImpl.java:311)
 at dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:147)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
 at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.SimpleFSDirectory@/home/islandora/fedora/solr/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@7df4591e: files: []
 at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:741)
 at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
 at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:65)
 at dk.defxws.fgssolr.OperationsImpl.getIndexReaderAndSearcher(OperationsImpl.java:717)
 ... 33 more

With index directory absent:

dk.defxws.fedoragsearch.server.errors.GenericSearchException: Wed Aug 13 16:23:45 BST 2014 IndexReader open error indexName=FgsIndex :
; nested exception is:
 org.apache.lucene.store.NoSuchDirectoryException: directory '/home/islandora/fedora/solr/collection1/data/index' does not exist
 at dk.defxws.fgssolr.OperationsImpl.getIndexReaderAndSearcher(OperationsImpl.java:719)
 at dk.defxws.fgssolr.OperationsImpl.updateIndex(OperationsImpl.java:267)
 at dk.defxws.fedoragsearch.server.GenericOperationsImpl.updateIndex(GenericOperationsImpl.java:556)
 at dk.defxws.fedoragsearch.server.RESTImpl.updateIndex(RESTImpl.java:311)
 at dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:147)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:247)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
 at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.lucene.store.NoSuchDirectoryException: directory '/home/islandora/fedora/solr/collection1/data/index' does not exist
 at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:217)
 at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:241)
 at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:679)
 at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
 at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:65)
 at dk.defxws.fgssolr.OperationsImpl.getIndexReaderAndSearcher(OperationsImpl.java:717)
 ... 33 more

No error is logged when running the update action on Solr.

A L, thanks for the suggestions. solr.xml is present and, I believe, correct:

<?xml version="1.0" encoding="UTF-8"?>
<Context docBase="/home/islandora/fedora/tomcat/webapps/solr.war" debug="0" crossContext="true">
        <Environment name="solr/home" type="java.lang.String" value="/home/islandora/fedora/solr" override="true" />
</Context>

dataDir is also correct in solrconfig.xml.

As I mentioned previously, solr appears to be running without errors, accessible at host:8080/solr, but is not indexing files ingested into Fedora. No errors are logged on ingest, but the following is logged in catalina.out when shutting down tomcat:

ERROR 2014-08-13 16:34:16,215 (MessagingClient) Messaging Exception encountered attempting to stop Messaging Client: fedoragsearch0. Exception message: The JMS connection has failed: java.io.EOFException

I assume the messaging error is related to the lack of indexing taking place - might it also be at the root of the other issues? Is there any additional configuration required around messaging beyond choosing the default URI after choosing to activate it while installing Fedora? I don't see it mentioned elsewhere in the docs.

Thanks again,

Paddy

[A L]

Paddy,

Have you restarted tomcat recently? I see Phil already mentioned about it. But try that again - 1. *empty* solr index directory (do not delete index folder) 2. restart tomcat 3. go to the solr rest url and hit updateIndexfromFoxml.

 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.<span

...

[A L]

Paddy, one correct to my previous post. For (1) that I mentioned, you can delete the index directory actually. You need not have to just empty it. So, delete index directory (back it up somewhere else if you need the data in it) and then restart tomcat.

Source: http://www.mail-archive.com/solr...@lucene.apache.org/msg06519.html

"Remove the index directory... it shouldn't be there if there is no index."

[A L]

Paddy - Did that work?

[Paddy McCann]

Hi A L

I've shut down and started up tomcat many times while trying to get to the bottom of this. I gave it another go just now - I deleted the index directory and restarted tomcat and when I tried to run the updateIndex action in gsearch I got the NoSuchDirectoryException. I then ran the solr update action, as suggested by Phil previously, and restarted tomcat. I still got the error.

The IndexNotFoundException that occurs when an empty index directory is present seems to be well documented. I'm really struggling to find anything to make any progress with this NoSuchDirectoryException.

[Paddy McCann]

Success!

It was, of course, typo.

solrconfig.xml specified a dataDir at solr/data. gsearch was looking in solr/collection1/data.

Not entirely sure how I missed it for so long

Now to work why objects aren't being indexed...

Thanks,

Paddy

[A L]

Awesome!! See..I told ya about the same exact thing that I ran into. ;)

>
1. One instance was when I copied solrconfig file from development server to a test server AS IS and I got all those errors that you got. After constant struggle, I realized I had data directory path listed as "C:" where in test server, I had all Solr configuration files located in directory "D:". So, make sure check the <dataDir> tag value in solrconfig file which should be located under $FEDORA_HOME/solr/collection1/

conf