Re: [Dspace-tech] indexing error: too many open files

[Joshua Gomez]

I discovered the solution to my indexing problem. To recap what happened: when I ran the dspace filter-media script it would crash after a while and complain about too many files being open. To fix that I edited the /etc/security/limits.conf file and added the following lines:

 

* hard nofile 65536

* soft nofile 65536

 

But the script still kept crashing.  Finally I discovered that the * wildcard does not apply to the root user, and I often login to our dspace server with my own username and run dspace scripts using sudo instead of doing things the proper way and running the scripts as the dspace user. By running the script as root, the new file limits did not apply.

 

Lesson learned, and I hope this message helps someone else in the future.

 

-Joshua

 

 

Joshua Gomez

Digital Library Programmer Analyst

Gelman Library

George Washington University

(202) 994-8267

jng...@library.gwu.edu

>>> Andrea Schweer <sch...@waikato.ac.nz> 7/21/2011 5:29 PM >>>
Hi Joshua,

On 22/07/11 02:20, Joshua Gomez wrote:
> Andrea, yes that worked. thanks. Of course I believe it has nothing
> to do with my original indexing problem, but at least I'm removing
> possibilities.

Good to hear it works and thanks for confirming -- I'll update the wiki
page.

So you're still getting your "too many open files" error with the Sun
JDK? The original error message you posted very much looked like a JDK
problem.

If you suspect it's actually related to Tomcat's memory settings, you
could try increasing its memory. There are some hints here:
https://wiki.duraspace.org/display/DSPACE/HowToPerformanceTuneForDspace.
You'd change JAVA_OPTS in /etc/default/tomcat6.

cheers,
Andrea

--
Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand

------------------------------------------------------------------------------
5 Ways to Improve & Secure Unified Communications
Unified Communications promises greater efficiencies for business. UC can
improve internal communications as well as offer faster, more efficient ways
to interact with customers and streamline customer service. Learn more!
http://www.accelacomm.com/jaw/sfnl/114/51426253/
_______________________________________________
DSpace-tech mailing list
DSpac...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

[Joshua Gomez]

Andrea, yes that worked. thanks. Of course I believe it has nothing to do with my original indexing problem, but at least I'm removing possibilities.

>>> Andrea Schweer <sch...@waikato.ac.nz> 7/20/2011 7:01 PM >>>
Hi Joshua,

On 21/07/11 02:36, Joshua Gomez wrote:
> I installed java-6-sun jdk and made it the default, but I am still
> getting the error with the index, though it now references the
> sunJDK instead of the OpenJDK.  However, when I run a list of open
> files, I see several files opened by jsvc (with dspace as the user)
> that are from the openjdk library. I don't understand why it would
> still be using those .jar and .so files if the sunjdk is the
> default.

Try putting the Sun JDK into /etc/default/tomcat6 and restarting tomcat
-- does that work for you?

JAVA_HOME=/usr/lib/jvm/java-6-sun

The tomcat 6 init script on Ubuntu has some 'clever' logic to determine
which JDK to use. Helpfully, this logic completely ignores what's set as
the system default.

cheers,
Andrea

--
Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand

------------------------------------------------------------------------------

10 Tips for Better Web Security
Learn 10 ways to better secure your business today. Topics covered include:
Web security, SSL, hacker attacks & Denial of Service (DoS), private keys,
security Microsoft Exchange, secure Instant Messaging, and much more.
http://www.accelacomm.com/jaw/sfnl/114/51426210/

[Joshua Gomez]

I installed java-6-sun jdk and made it the default, but I am still getting the error with the index, though it now references the sunJDK instead of the OpenJDK.  However, when I run a list of open files, I see several files opened by jsvc (with dspace as the user) that are from the openjdk library. I don't understand why it would still be using those .jar and .so files if the sunjdk is the default.

>>> Stuart Lewis <s.l...@auckland.ac.nz> 7/11/2011 6:06 PM >>>
Hi Joshua,

The error message below suggests that you are running the openjdk.  Instead, try installing the official Sun JDK:

- https://wiki.duraspace.org/display/DSPACE/Installing+DSpace+1.7+on+Ubuntu#InstallingDSpace1.7onUbuntu-%28OPTIONAL%29ChangetousingSun%2FOracleJavaJDK

This will hopefully help.

Thanks,


Stuart Lewis
Digital Development Manager
Te Tumu Herenga The University of Auckland Library
Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand
Ph: +64 (0)9 373 7599 x81928


On 12/07/2011, at 3:33 AM, Joshua Gomez wrote:

> I ran across the following error last week while trying to run the media filter (I'm using DSpace 1.7.2, on Ubuntu 10.04.2):
> 
> Exception in thread "Thread-2" java.lang.UnsatisfiedLinkError: /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libmanagement.so: /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libmanagement.so: cannot open shared object file: Too many open files
>         at java.lang.ClassLoader$NativeLibrary.load(Native Method)
>         at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1750)
>         at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1667)
>         at java.lang.Runtime.loadLibrary0(Runtime.java:840)
>         at java.lang.System.loadLibrary(System.java:1047)
>         at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:67)
>         at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:47)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at sun.management.ManagementFactory.<clinit>(ManagementFactory.java:485)
>         at java.lang.management.ManagementFactory.getPlatformMBeanServer(ManagementFactory.java:521)
>         at org.dspace.kernel.DSpaceKernelManager.unregisterMBean(DSpaceKernelManager.java:178)
>         at org.dspace.servicemanager.DSpaceKernelImpl.destroy(DSpaceKernelImpl.java:211)
>         at org.dspace.servicemanager.DSpaceKernelImpl.doDestroy(DSpaceKernelImpl.java:233)
>         at org.dspace.servicemanager.DSpaceKernelImpl$1.run(DSpaceKernelImpl.java:78)
> 
> It appears this error is not a problem with DSpace so much as the server environment, but I'm wondering if anyone else has come across such an error before?
> 
> This error occurred after the new items had been filtered and the script was in the process of updating the index. I recently added about a thousand items that had very poor OCR (they were very old documents) and our index word limit is not set, so I am wondering if the index has become too large due to all the new "words" from the bad OCR. Many of the pdfs we added are pretty large so I'm also wondering if it's just a memory issue.
> 
> Any thoughts?

> 
> Joshua
> 
> 
> Joshua Gomez
> Digital Library Programmer Analyst
> Gelman Library
> George Washington University
> (202) 994-8267
> jng...@library.gwu.edu

> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2_______________________________________________

[Andrea Schweer]

Hi Joshua,

On 22/07/11 02:20, Joshua Gomez wrote:

> Andrea, yes that worked. thanks. Of course I believe it has nothing
> to do with my original indexing problem, but at least I'm removing
> possibilities.

Good to hear it works and thanks for confirming -- I'll update the wiki
page.

So you're still getting your "too many open files" error with the Sun
JDK? The original error message you posted very much looked like a JDK
problem.

If you suspect it's actually related to Tomcat's memory settings, you
could try increasing its memory. There are some hints here:
https://wiki.duraspace.org/display/DSPACE/HowToPerformanceTuneForDspace.
You'd change JAVA_OPTS in /etc/default/tomcat6.

[Stuart Lewis]

Hi Joshua,

Do you have any cron jobs setup that might be using the openjdk?

Are you able to uninstall the openjdk library so that it can't be used any more?

Thanks,


Stuart


On 21/07/2011, at 2:36 AM, Joshua Gomez wrote:

> I installed java-6-sun jdk and made it the default, but I am still getting the error with the index, though it now references the sunJDK instead of the OpenJDK. However, when I run a list of open files, I see several files opened by jsvc (with dspace as the user) that are from the openjdk library. I don't understand why it would still be using those .jar and .so files if the sunjdk is the default.

[Andrea Schweer]

Hi Joshua,

On 21/07/11 02:36, Joshua Gomez wrote:

> I installed java-6-sun jdk and made it the default, but I am still
> getting the error with the index, though it now references the
> sunJDK instead of the OpenJDK. However, when I run a list of open
> files, I see several files opened by jsvc (with dspace as the user)
> that are from the openjdk library. I don't understand why it would
> still be using those .jar and .so files if the sunjdk is the
> default.

Try putting the Sun JDK into /etc/default/tomcat6 and restarting tomcat
-- does that work for you?

JAVA_HOME=/usr/lib/jvm/java-6-sun

The tomcat 6 init script on Ubuntu has some 'clever' logic to determine
which JDK to use. Helpfully, this logic completely ignores what's set as
the system default.

[Stuart Lewis]

[Joshua Gomez]

Thank you for the advice Stuart.  I installed the Sun JDK and got the same error (see below).  I have also tried changing the number of files allowed in the shell by adding the following lines to the /etc/security/limits.conf file:

 

* hard nofile 65536

* soft nofile 65536

 

I have read on other mailing list archives that I should adjust the kernel file limits to 200k, but when I check the current value, it is already much larger!

 

> cat /proc/sys/fs/file-max

799000

 

I'm wondering if I should try doubling the shell limit. Does anyone have any other suggestions?  Below is the error output:

 

Updating search index:
Exception in thread "Thread-2" java.lang.UnsatisfiedLinkError: /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libmanagement.so: /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libmanagement.so: cannot open shared object file: Too many open files
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1720)
        at java.lang.Runtime.loadLibrary0(Runtime.java:823)
        at java.lang.System.loadLibrary(System.java:1028)
        at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.management.ManagementFactory.<clinit>(ManagementFactory.java:467)
        at java.lang.management.ManagementFactory.getPlatformMBeanServer(ManagementFactory.java:504)

        at org.dspace.kernel.DSpaceKernelManager.unregisterMBean(DSpaceKernelManager.java:178)
        at org.dspace.servicemanager.DSpaceKernelImpl.destroy(DSpaceKernelImpl.java:211)
        at org.dspace.servicemanager.DSpaceKernelImpl.doDestroy(DSpaceKernelImpl.java:233)
        at org.dspace.servicemanager.DSpaceKernelImpl$1.run(DSpaceKernelImpl.java:78)

>>> Stuart Lewis <s.l...@auckland.ac.nz> 7/11/2011 6:06 PM >>>

[Joshua Gomez]