[Dspace-tech] Tomcat memory leak warning prevents restarting
Jason Stirnaman
I get the error below when stopping Tomcat. In this particular case, I was trying to use Solaris' SMF to restart the Tomcat service. Instead, it never completed shutting down and it stopped answering requests. I got this error stanza repeated over and over in the log. Any ideas what's going on?
Thanks,
Jason
Details:
DSpace 1.7.1 w/ Discovery
Tomcat 6.0.32
Solaris 10
May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-80
May 21, 2011 12:00:08 AM org.apache.catalina.core.StandardService stop
May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause
May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-80
May 21, 2011 12:00:08 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [MultiThreadedHttpConnectionMa
nager cleanup] but has failed to stop it. This is very likely to create a memory leak.
May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@1de92e]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [o
rg.apache.xerces.parsers.SAXParser@17533df]) but failed to remove it when the web application was st
opped. This is very likely to create a memory leak.
May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@1b7fc4e]) and a value of type [org.apache.cocoon.environment.internal.Enviro
nmentStack] (value [[]]) but failed to remove it when the web application was stopped. This is very
likely to create a memory leak.
Jason Stirnaman
Biomedical Librarian, Digital Projects
A.R. Dykes Library, University of Kansas Medical Center
jstir...@kumc.edu
913-588-7319
Jason Stirnaman
I found a long, interesting thread from September 2010 at http://www.mail-archive.com/dspac...@lists.sourceforge.net/msg11969.html around this topic and an excellent description of the Tomcat MemoryLeakProtection at http://wiki.apache.org/tomcat/MemoryLeakProtection. I obviously didn't do my homework before posting. It looks like Tomcat 7.0.6 *might* "solve" this problem.
However, I'm still unclear whether there is anything else I can do now to work around it - to at least stop the leak detection from thwarting my restarts?
As Tim suggested in that earlier thread I'm including more details from my production server while Tomcat is up and DSpace is running OK.
Hardware:
Solaris 10 Zone virtual machine with 1167MHz processor
Output from top (I'm open to better methods, but I'm limited to the single Zone.)
40 processes: 39 sleeping, 1 on cpu
CPU states: 93.5% idle, 3.7% user, 2.8% kernel, 0.0% iowait, 0.0% swap
Kernel: 18117 ctxsw, 6164 trap, 8377 intr, 45826 syscall, 4 fork, 543 flt, 16 pgin, 40 pgout
Memory: 64G phys mem, 40G free mem, 4104M total swap, 4104M free swap
RAM:
"CATALINA_OPTS" value="-Xms256m -Xmx1028m -XX:MaxPermSize=256m"
I recently increased RAM to 1028 from (I think) 256M and I doubled PermGen. Since then the Java process has consistently been using around 490M and usually less than 1% CPU.
Postgres uses around 520M.
DSpace: version 1.7.1 with XMLUI and Discovery
Issues: Prior to my increasing the allotted Catalina RAM and Permgen, Tomcat would stop responding around every 3 days (? number of requests) due to the database pool filling up ("cannot get a connection - pool exhausted"). I decided to also restart Tomcat every night to prevent this from happening. That's when I encountered the memory leak warnings preventing my restarts.
Jason
Jason Stirnaman
Biomedical Librarian, Digital Projects
A.R. Dykes Library, University of Kansas Medical Center
jstir...@kumc.edu
913-588-7319
Vlastimil Krejcir
I'm sure that this issue has already been discussed here more than once
and as I can remember without any final and useful universal solution
(maybe there's not one). I have been fighting with it for years (and still
don't know where lies the "bug"). My solution is to restart Tomcat (via
cron) once a day and use a special piece of code to do this - I test if
Tomcat is really stopped correctly and if not I kill the Tomcat
process (bash script):
/opt/tomcat/bin/shutdown.sh
echo "Waiting 45 second for Tomcat shutdown..."
# This is necessary, because shutting down process can take some time,
# 45 second is enough according to my experience
sleep 45
RUNNING=`ps ax | grep opt\/tomcat | grep 2048`
if [ -z "$RUNNING" ]
then
echo "Tomcat has been stopped...";
else
echo "Shutdown failure... Trying to do hard kill."
PID=`echo $RUNNING | cut -d ' ' -f 1`
kill -9 $PID
# give system some time to kill it
sleep 3
# check it again
RUNNING=`ps ax | grep opt\/tomcat | grep 2048`
if [ -z "$RUNNING" ]
then
echo "Tomcat has been killed with -9."
else
echo "Cannot kill Tomcat. Administrator needed!"
exit 1
fi
fi
This "shutdown" Tomcat for sure.
Vlastik
----------------------------------------------------------------------------
Vlastimil Krejčíř
Library and Information Centre, Institute of Computer Science
Masaryk University, Brno, Czech Republic
Email: krejcir (at) ics (dot) muni (dot) cz
Phone: +420 549 49 3872
ICQ: 163963217
Jabber: kre...@jabber.org
----------------------------------------------------------------------------