[Dspace-tech] memory problem and 'Exception Processing ErrorPage'

[Martin Musacchio]

Hi all.

I have about 8 or 9 JSPUI dspace installations in a server which has 3
GB of RAM. Four of them are active and the other ones are smaller
projects that mostly don't get used. After our last install, 1.6.2
JSPUI, which is an image repository, I started getting tomcat hangs
almost hourly, or at least a few times a day, and a restart is
required. Sometimes even not related to the usage of that repository
but when another one is used (perfectly working before). I was
wondering if this new image repository, precisely because it hosts
images, thumbnails, etc, may be hogging the resources. The strange
thing is that the hangs sometimes happen even when another repo is
being used. Some hangs are "PermGen" but others just don't output
anything in the logs.

Also, I sometimes leave "parallel" webapps in the tomcat deploy dir,
which are copies of a webapp with some new modification I want to try.
And sometimes I leave a few directories like that in there. Sort of a
primitive version control system :). I was wondering if maybe tomcat
tends to try to deploy everything it sees in the deploy directory, and
maybe those parallel webapps are contributing to the hangs. I removed
them all and things seem to have improved some, but this is not
definitive. Is this possible? Does it make sense?


Another important part of this problem is that I can't seem to assign
specific memory parameters to tomcat. I've read the Performance Tuning
tips


https://wiki.duraspace.org/display/DSDOC18/Performance+Tuning+DSpace

and I see that the variable to modify is CATALINA_OPS, but they don't
say in what file. I guess it depends on the operating system. I'm on
Debian. I tried adding something like

CATALINA_OPTS="-Xms1025m -Xmx1024m -XX:PermSize=1024m
-XX:MaxPermSize=2048m -XX:+DisableExplicitGC XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled"

in

[tomcat]/bin/catalina.sh
[tomcat]/bin/setenv.sh
[tomcat]/bin/startup.sh

and inside

/etc/init.d/tomcat

but none seems to work.

When I run

ps aux | grep tomcat

I don't see the new parameters added.

How could I make this work?







Lastly, here's a stacktrace from the most common error I get (although
this doesn't hang tomcat)


19-mar-2012 1:13:57 org.apache.catalina.core.StandardHostValve custom
GRAVE: Exception Processing
ErrorPage[exceptionType=java.lang.Exception, location=/internal-error]
java.lang.IllegalStateException
at org.apache.coyote.Response.reset(Response.java:297)
at org.apache.catalina.connector.Response.reset(Response.java:653)
at org.apache.catalina.connector.Response.reset(Response.java:920)
at
org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:357)
at
org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:213)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:662)





--
-------------------------------------------------------------------------------
Martín Musacchio
Centro de Telemática - UNL
Pje. Martinez 2652 - S3002AAB Santa Fe - Argentina
tel +54(342)455-4245 - FAX +54(342)457-1240

[Martin Musacchio]

As of now I also tried adding the memory parameters of CATALINA_OPTS
to the root .profile file and the tomcat user .profile but after a
"source" command for reloading and a tomcat restart, I then run

ps aux | grep tomcat

and see the same as before; not my parameters.

Note that if I do

echo $CATALINA_OPTS, I can see it properly set. But somehow tomcat is
not picking it up...

Any help please?

[Peter Dietz]

Hi Martin,

Sorry to hear about your tomcat performance woes.

What is working for me to set some properties for tomcat, is to alter /opt/apache-tomcat-6.0.35/bin/catalina.sh

And at about line 87 (right after the comments, and before anything else gets set, I add:

JAVA_OPTS="-Xms512m -Xmx1024m -XX:MaxPermSize=512m"

I'm not sure about the details of why I chose to set JAVA_OPTS, instead of CATALINA_OPTS, but this does show up in ps aux | grep tomcat.

peterdietz      5494   0.0  5.8  4521128 484720 s011  U     3:18PM   0:58.23 /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home/bin/java -Djava.util.logging.config.file=/opt/apache-tomcat-6.0.35/conf/logging.properties -Xms512m -Xmx1024m -XX:MaxPermSize=512m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/apache-tomcat-6.0.35/endorsed -classpath /opt/apache-tomcat-6.0.35/bin/bootstrap.jar -Dcatalina.base=/opt/apache-tomcat-6.0.35 -Dcatalina.home=/opt/apache-tomcat-6.0.35 -Djava.io.tmpdir=/opt/apache-tomcat-6.0.35/temp org.apache.catalina.startup.Bootstrap start

I'm running this on development on OSX, and I locally installed tomcat6 to /opt/.

-----

Regarding some of your other issues, I've somewhat noticed some bad-behavior with Tomcat when I push its limits. I wanted to be able to have my University's local instance of DSpace with webapps (XMLUI, JSPUI, SOLR), and to also run a clean version of DSpace checked out from DSpace's trunk/master. That usually works fine, but when I also try to run the dspace rest-api, or another dspace webapp like webmvc, things tend to bog down. Upon restarting the server it lags for quite a while as its trying to deploy all of these webapps. 

I don't know if this advice would be useful at all (i.e. untested). But you could consider having separate installations of tomcat, and putting certain webapps in each one. i.e. /opt/tomcat-university has /dspace-myuniversity/webapps/xmlui, and have the run on port 8080, and /opt/tomcat-master has /dspace-master/webapps/xmlui run on port 8888. That might use more overall memory, but problems should hopefully be isolated to particular be webapps/sites. (Then your image repository won't take down your document repository).

Peter Dietz

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
DSpace-tech mailing list
DSpac...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

[Martin Musacchio]

Hi Peter, thanks for your reply.

Actually I could never make those parameters work in the catalina.sh
file. I tried a lot. Finally today after days of trying I managed to
do some trickery to make them work inside the /etc/init.d/tomcat
script from the OS (Debian). I know it's not elegant and I'll have to
take note of it for future tomcat upgrades but that's the only way I
got it to work. I used JAVA_OPTS as well. Are these parameters too
much for the mere 3 GB or RAM in this server? (there's a memory
upgrade coming soon)

-Xms512m -Xmx1024m -XX:PermSize=512m -XX:MaxPermSize=1024m

I'm thinking in the worst case both max heap and max perm will go high
enough to reach 2 GB leaving 1 extra GB for the other services of the
server.

Also, is it too much load to have, say, 8 dspace jspui apps? Currently
we have 4 working, and in the next year will probably reach to 6.



About the multiple tomcat installs... that sounds very interesting.
How could I manage to do that? If I install from debian repository, I
wouldn't be able to install twice unless I use a different version,
right?

[Peter Dietz]

Hi Martin,

re: 3GB memory

That might be ok, its unfortunate that Java apps are memory expensive. Our production system hosts one instance of DSpace, and that is running well on 2.5GB.

My development computer has 8GB, and have several instances, but has virtually no load. I've heard of some places that have dozens upon dozens of JSPUI instances. But I know very little about the infrastructure required to pull that off.

re:Multiple Tomcats..

I wouldn't attempt to install tomcat multiple times from the debian repository. That won't work. maybe you could get tomcat6, and tomcat7 installed simultaneously, but thats not the intention.

The process is more-or-less to download the tomcat zip from tomcat's website, unpack it, and create multiple copies of that folder. I like to install this kind of thing to /opt. 

Then you'd end up with:

/opt/apache-tomcat-6.0.35-client1

/opt/apache-tomcat-6.0.35-client2

...

You have to modify each of those to run on different ports. tomcat-client1 = 8080, tomcat-client2 = 8081, ...

And if you proxy apache's port 80 to tomcat, you'll have to setup rules for that to go to these different tomcats.

For starting / stopping your tomcats, you won't do /etc/init.d/tomcat6 start, but instead /opt/tomcat-client1/bin/startup.sh

I found an FAQ from Atlasssian about them recommending you not to put multiple Atlassian applications into a single tomcat instance. 

http://confluence.atlassian.com/display/JIRA044/Deploying+Multiple+Atlassian+Applications+in+a+Single+Tomcat+Container

I'm not at all saying that you can't put a single DSpace instance (jspui + xmlui + solr + sword) into a single instance of tomcat. Just that if you've got multiple instances of DSpace (client1-jspui, client2-jspui, ...), and you are running into tomcat performance issues, that multiple tomcat instances might be a way to go.

Peter Dietz

[helix84]

On Tue, Mar 20, 2012 at 18:36, Peter Dietz <pdie...@gmail.com> wrote:
> re:Multiple Tomcats..

Having Tomcat installed from the repository gives the convenience of
easy updates. You might want to check this out:

http://code.google.com/p/debian-tomcat-scripts/source/browse/#svn%2Ftrunk%2Fmultiple-instances

Or just search for "debian multiple tomcats".

One more thing I'd like to point out is that you don't have to isolate
_each_ webapp into its own container. Maybe just isolate the images
repository if you suspect it hogs all the memory while the others
worked fine without it.

Regards,
~~helix84