Re: [Dspace-tech] Problem with "write.lock" file in /dspace/search - DSpace 1.5.1

[Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY]]

I was able to correct the problem of the email not being sent out whenever a lock situation occurred by adding this type of line to every place in DSIndexer.java where an error was being logged to dspace.log:

            emailException(e)   {where “e” is an “Exception” object/variable}

 

There were actually 9 places where I inserted this line, and while some of them really were “overkill” (there were a couple of places in the program that had comments to the effect that that section of code should never be executed) I added it in for consistency anyway.

 

The good news is that I recreated the error and I did get the error email this time.

 

Perhaps this should be included in a future release? 

 

Sue

 

From: Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY] [mailto:susan.m....@nasa.gov]
Sent: Thursday, May 20, 2010 5:30 PM
To: dspac...@lists.sourceforge.net
Subject: [Dspace-tech] Problem with "write.lock" file in /dspace/search - DSpace 1.5.1

 

Hello,

     I updated a Collection description in one of our DSpace instances, a few times this morning.  Each time I noticed that it took awhile for the update to complete.  I finally looked in the log and saw errors referring to “write.lock” in /dspace/search.  I looked in the /dspace/search directory and was surprised to find a “write.lock” file in the directory, but I was shocked to see that the search indexes had not been updated since March 29^th!  Long story short, somehow a failed transaction left a lock file in the search directory on March 29^th and then every time after that someone performed a transaction that attempted to update the search indexes, the application waited 30 seconds (explanation later) and then an error occurred.

 

     So basically my repository’s search indexes have been inaccurate since March 29^th.  That’s easy to fix by running “index-update”.    The problem is that I just sort of found the issue by accident while looking at something else in the dspace.log and was never notified about the errors.  I looked at DSIndexer.java and saw that in the documentation at the top of the program it says:

 

“It will attempt to attain a lock on the index in the event that an update is requested and will wait a maximum of 30 seconds (a worst case scenario) to attain the lock before giving up and logging the failure to log4j and to the DSpace administrator email account.”

 

My question is this:  Why wasn’t an email sent out when the “lock” errors occurred?  It looks as if the program was successfully writing to the log, but for some reason it didn’t send out the emails.  Is this an oversight, an issue that’s been corrected in a subsequent version, or a problem specific to our installation?  I feel like if I had received the email(s) back in March, I could have corrected it then.

 

Below is the error output from the dspace.log.  If any has any ideas, please let me know.  Thanks!

 

Best regards,

Sue

 

 

2010-05-20 11:32:36,815 INFO  org.dspace.app.webui.servlet.CommunityListServlet @ susan.m....@nasa.gov:session_id=AB9292929292929292929293939AFAFA:ip_addr=111.222.333.444:view_community_list:

2010-05-20 11:32:41,404 INFO  org.dspace.app.webui.servlet.DSpaceServlet @ susan.m....@nasa.gov:session_id=AB9292929292929292929293939AFAFA:ip_addr=111.222.333.444:view_collection:collection_id=91

2010-05-20 11:32:41,404 INFO  org.dspace.browse.BrowseEngine @ susan.m....@nasa.gov:session_id=AB9292929292929292929293939AFAFA:ip_addr=111.222.333.444:browse_mini:

2010-05-20 11:38:03,321 INFO  org.dspace.app.webui.servlet.DSpaceServlet @ susan.m....@nasa.gov:session_id=AB9292929292929292929293939AFAFA:ip_addr=111.222.333.444:view_community:community_id=34

2010-05-20 11:38:03,325 INFO  org.dspace.browse.BrowseEngine @ susan.m....@nasa.gov:session_id=AB9292929292929292929293939AFAFA:ip_addr=111.222.333.444:browse_mini:

2010-05-20 11:38:03,417 INFO  org.dspace.app.webui.servlet.RetrieveServlet @ susan.m....@nasa.gov:session_id=AB9292929292929292929293939AFAFA:ip_addr=111.222.333.444:view_bitstream:bitstream_id=652731

2010-05-20 11:41:43,673 INFO  org.dspace.content.Collection @ susan.m....@nasa.gov:session_id=AB9292929292929292929293939AFAFA:ip_addr=111.222.333.444:update_collection:collection_id=91

2010-05-20 11:42:13,731 ERROR org.dspace.search.DSIndexer @ Lock obtain timed out: SimpleFSLock@/export/home/dspace/search/write.lock

org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SimpleFSLock@/export/home/dspace/search/write.lock

          at org.apache.lucene.store.Lock.obtain(Lock.java:85)

          at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:691)

          at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:452)

          at org.dspace.search.DSIndexer.openIndex(DSIndexer.java:805)

          at org.dspace.search.DSIndexer.writeDocument(DSIndexer.java:877)

          at org.dspace.search.DSIndexer.buildDocument(DSIndexer.java:943)

          at org.dspace.search.DSIndexer.indexContent(DSIndexer.java:328)

          at org.dspace.search.SearchConsumer.end(SearchConsumer.java:207)

          at org.dspace.event.BasicDispatcher.dispatch(BasicDispatcher.java:171)

          at org.dspace.core.Context.commit(Context.java:384)

          at org.dspace.core.Context.complete(Context.java:350)

          at org.dspace.app.webui.servlet.admin.EditCommunitiesServlet.processConfirmEditCollection(EditCommunitiesServlet.java:693)

          at org.dspace.app.webui.servlet.admin.EditCommunitiesServlet.doDSPost(EditCommunitiesServlet.java:267)

          at org.dspace.app.webui.servlet.DSpaceServlet.processRequest(DSpaceServlet.java:147)

          at org.dspace.app.webui.servlet.DSpaceServlet.doPost(DSpaceServlet.java:105)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:816)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:917)

          at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:398)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)

          at org.dspace.app.webui.filter.RegisteredOnlyFilter.doFilter(RegisteredOnlyFilter.java:98)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:217)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)

          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)

          at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:255)

          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:188)

          at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:586)

          at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:556)

          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:187)

          at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:586)

          at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:556)

          at com.sun.webserver.connector.nsapi.NSAPIProcessor.service(NSAPIProcessor.java:160)

 

 

Sue Walker-Thornton

NASA Langley Research Center
Integrated Library Systems

Developer, Application & Database Administrator

ConITS Contract ~ NCI Information Systems, Inc.

130 Research Drive

Hampton, VA  23666

Office: (757) 224-4074 ~ Mobile: (757) 506-9903 ~ Fax: (757) 224-4001 

email:  susan.m....@nasa.gov

 

[Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY]]

I was able to correct this problem by inserting the following line in 9 different places in DSIndexer.java:

 

emailException(e)   {where “e” is an “Exception” object/variable}

 

What I did was insert this line every place where an error occurred and was logged to dspace.log, but the exception was not emailed.  So basically I found every place in the program where there was a line like this…

 

            log.error(e.getMessage(), e);

 

…that wasn’t followed by the “emailExeption(e)” statement, and I inserted it there.  Granted, some of this was probably overkill since there were areas of the program that had a comment saying “This code will never be executed, but it makes the compiler happy…” J.  The good news is that I recreated the problem and I did receive the error email this time.

 

Perhaps this fix should be included in the next release??

 

Thanks,

[Thornton, Susan M. (LARC-B702)[RAYTHEON TECHNICAL SERVICES COMPANY]]