[Dspace-tech] Error: Too many files open

[Paul Marlow]

Hello!

First of all, forgive me as I'm new to DSpace, nor am I the one who
set up the system.

That being said, we are consistently receiving errors referring to
"Too many files open", in which we must restart the Tomcat service
two or more times a day (which has been happening for nearly a month
now). Searching on our system is fine, it occurs when our users are
attempting to edit the metadata and/or uploading/removing bitstreams
- or when the system attempts to send email alerts. At this point,
other than searching, the system is unusable until the services are
restarted.

I have done several searches in Google as well as in the Dspace
mailing lists, but I haven't found an actual solution to the problem,
although I have found discussions on the topic.

In relation to all the discussions I've read, I have provided most of
the information requested by them in a hope that others may have an
answer, or at least speed things up a bit.

DSpace version:
1.4.1

Operating System:
OS Version:
# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 4)
# uname -a
Linux <node name removed> 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16
17:17:21 EST 2007 i686 i686 i386 GNU/Linux

Cron Jobs:
dspace cron jobs
]# crontab -u dspace -l
# Send out subscription emails at 1:00 everyday
0 1 * * * /opt/dspace/bin/sub-daily
#Run the media filter at 2:00 every day
0 2 * * * /opt/dspace/bin/filter-media
#Run the checksum checker at 3:00
0 3 * * * /opt/dspace/bin/checker -lp
#Mail the results to the sysadmin at 4:00
0 4 * * * /opt/dspace/bin/dsrun org.dspace.checker.DailyReportEmailer
-c
#Run stat analyses
0 5 * * * /opt/dspace/bin/stat-general
10 5 * * * /opt/dspace/bin/stat-monthly
20 5 * * * /opt/dspace/bin/stat-report-general
30 5 * * * /opt/dspace/bin/stat-report-monthly

Notice that index-all is not included, as my understand is that
filter-media includes a rebuild of the indexes. I have manually
rebuilt the indexes which does not resolve the issue.

INodes:
# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/cciss/c0d0p1 524288 111084 413204 22% /
none 214062 1 214061 1% /dev/shm
/dev/cciss/c0d0p3 17137664 644078 16493586 4% /var

files-nr Information:
# cat /proc/sys/fs/file-nr
5120 0 580170

Example Error from Tomcat's catalina.out:
Feb 15, 2008 10:28:05 AM org.apache.tomcat.util.net.PoolTcpEndpoint
acceptSocket
SEVERE: Endpoint
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] ignored
exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(Def
aultServerSocketFactory.java:61)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoin
t.java:408)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll
owerWorkerThread.java:71)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:685)
at java.lang.Thread.run(Thread.java:534)
Feb 15, 2008 10:28:05 AM org.apache.tomcat.util.net.PoolTcpEndpoint
acceptSocket

Example corresponding errors in dspace.log:
java.io.FileNotFoundException:
/var/opt/dspace/assetstore/58/21/34/5821340475210663413811699708242116
0566 (Too many open files)
va.io.FileNotFoundException:
/var/opt/dspace/assetstore/58/21/34/5821340475210663413811699708242116
0566 (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
edu.sdsc.grid.io.local.LocalFileInputStream.open(LocalFileInputStream.
java:171)
at
edu.sdsc.grid.io.GeneralFileInputStream.<init>(GeneralFileInputStream.
java:145)
at
edu.sdsc.grid.io.local.LocalFileInputStream.<init>(LocalFileInputStrea
m.java:139)
at
edu.sdsc.grid.io.FileFactory.newFileInputStream(FileFactory.java:630)
at
org.dspace.storage.bitstore.BitstreamStorageManager.retrieve(Bitstream
StorageManager.java:553)
at org.dspace.content.Bitstream.retrieve(Bitstream.java:512)
at
org.dspace.app.webui.servlet.BitstreamServlet.doDSGet(BitstreamServlet
.java:205)
at
org.dspace.app.webui.servlet.DSpaceServlet.processRequest(DSpaceServle
t.java:159)
at
org.dspace.app.webui.servlet.DSpaceServlet.doGet(DSpaceServlet.java:10
0)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:210)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:174)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.pr
ocessConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoi
nt.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll
owerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:685)
at java.lang.Thread.run(Thread.java:534)
2008-02-15 10:28:06,006 WARN org.dspace.app.webui.util.UIUtil @
Unable to send email alert
java.io.FileNotFoundException:
/opt/dspace/config/emails/internal_error (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at java.io.FileInputStream.<init>(FileInputStream.java:66)
at java.io.FileReader.<init>(FileReader.java:41)
at
org.dspace.core.ConfigurationManager.getEmail(ConfigurationManager.jav
a:282)
at org.dspace.app.webui.util.UIUtil.sendAlert(UIUtil.java:533)
at
org.dspace.app.webui.servlet.InternalErrorServlet.doGet(InternalErrorS
ervlet.java:86)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:188)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispa
tcher.java:691)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicat
ionDispatcher.java:471)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDi
spatcher.java:403)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDisp
atcher.java:301)
at
org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.ja
va:364)
at
org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve
.java:213)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:135)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.pr
ocessConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoi
nt.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll
owerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:685)
at java.lang.Thread.run(Thread.java:534)


Any assistance would be greatly appreciated.

Thanks!

Paul Marlow

[Eric Luhrs]

Paul,
It sounds like you are running into a per-user file handle limit.
I've seen this with other application running on RHEL4 servers. It
looks like your system-wide limit is okay, which is the limit applied
to the root user. All other users are limited to 1024 files by
default.

Since tomcat is probably running as a different user, you'll need to
increase the total number of files that this user can have open at one
time. Keep in mind that to *nix, everything is considered a file,
including directories, libraries, etc. To allow another user to have
more than 1024 files open add a line like this to
/etc/security/limits.conf:

tomcat - nofile 4096

You can change the user (including *) and number of files as
appropriate. Login again (not via su) and check to see if the change
worked:

ulimit -n

This should report 4096 rather than the 1024 default. If you still
see 1024, try upgrading sshd since I think older versions don't
refresh this var correctly.

If you need more info, RedHat has a FAQ about all of this:

http://kbase.redhat.com/faq/FAQ_80_1540.shtm

I hope this helps,

Eric Luhrs
Digital Initiatives Librarian
Lafayette College

> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> DSpace-tech mailing list
> DSpac...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>

[Paul Marlow]

Randall,

To answer your questions, Tomcat is running under the tomcat user,
with ulimits as follows:

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 110591
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


I also read another post, indicating that there could be memory
problems with indexing a large repository, such that "the
org.dspace.core.Context object cache,
which in the case of a full re-index will currently try to get an
in-memory copy of every object in the DSpace instance."

It was suggested to add:

target.decache();

in the org.dspace.search.DSIndexer.indexAllItems(), as well as in
org.dspace.browse.Browse.indexAll().

Now we currently aren't experiencing memory problems while running
our indexing, so I'm not sure if this could be a potential cause or
not. Does anyone have any info on this suggestion, and if it could
be related to our issue?

Thanks,

Paul


---- Original Message ----
From: rdf...@indiana.edu
To: pma...@idrc.ca
Subject: Re: [Dspace-tech] Error: Too many files open
Date: Fri, 15 Feb 2008 17:08:01 -0500

>Paul,
>
>Who is the Tomcat container running as and what are their ulimits?
>If
>you become that user and do a 'ulimit -a' you will see a line like:
>
>open files (-n) 2048
>
>Maybe the defaults on your system and the user that you are running
>Tomcat as is too small.