Improving Tomcat memory parameters

[Ellen Ball]

We've experienced the pain of tomcat crashing on an OpenMRS 1.9.3 production server with this memory error:  "Java.lang.OutOfMemoryError:  PermGen"

• The server has 32GB of RAM.  
• tomcat is  using JAVA_OPTS = "-Xmx512m -Xms512m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=128m"

Has anyone else seen this error?  Has anyone experimented with increased memory parameter values?  It seems that the Xmx and Xms values should be the same for a server ("In a server environment, you normally want Xms and Xmx set to the same value to avoid heap thrashing.").  The heap space shouldn't be too high, but if anyone know the optimal size that would be helpful.

Thanks,

Ellen Ball

Partners In Health

 

[Saptarshi Purkayastha]

Firstly expecting that this is a 64-bit OS with 64-bit JVM,

I'd have PermSize and MaxPermSize different, probably double for a server with 32GB RAM
   -XX:PermSize=256m -XX:MaxPermSize=512m

If Java activities and MySQL and equally prioritized on the server. I'd still give 4Gb max heap size for Java.
   -Xmx4G -Xms1024m

If you have enough RAM for everything else running on the server and its not paging, 4Gb heap size shouldn't be a problem.
Heap thrashing is a misnomer coming from early IBM implementations. Sun/Oracle JVM is really well managed

You are also probably using sun java6 (hotspot jvm), I'd enable the CMS (Concurrent Mark & Sweep Garbage collector)
   -XX:+UseConcMarkSweepGC

---
Regards,
Saptarshi PURKAYASTHA

My Tech Blog:  http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

--
OpenMRS Implementers: http://go.openmrs.org/implementers
Post: implem...@openmrs.org
Unsubscribe: implementers...@openmrs.org
Manage your OpenMRS subscriptions at https://id.openmrs.org/
 
 

[Saptarshi Purkayastha]

To add to what I wrote earlier, so that this is not considered Voodoo knowledge, lemme quote references:
http://www.oracle.com/technetwork/java/javase/6u18-142093.html
Server JVM heap configuration ergonomics are now the same as the Client, except that the default maximum heap size for 32-bit JVMs is 1 gigabyte, corresponding to a physical memory size of 4 gigabytes, and for 64-bit JVMs is 32 gigabytes, corresponding to a physical memory size of 128 gigabytes.

So, by a calculation that the default JVM memory allocations have some science behind it, they allocate 1/4 total physical memory size. On your 32GB server, lets consider MySQL has been given 16GB, so 1/4th of the remaining I'd recommend 4G for heap.

Also one should move to JDK7 for the G1 garbage collector's efficiency (once we figure out why those 2 tests randomly end in errors, but not fail)
http://www.oracle.com/technetwork/java/javase/tech/g1-intro-jsp-135488.html

---
Regards,
Saptarshi PURKAYASTHA

My Tech Blog:  http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

[Rowan Seymour]

@Saptarshi maybe you could update https://wiki.openmrs.org/display/docs/Troubleshooting+Memory+Errors 

I think every implementation runs into these kind of problems eventually and just googling for answers gives all sorts of conflicting advice.

--
Rowan Seymour
tel: +250 783835665

[Mark Goodrich]

Saptarshi,

Thanks so much!  This is very helpful...

+1 to Rowan's suggestion of updating the wiki... if you don't have time, let me know, and I can at least paste in what you've written here.

This is a good page as well:

https://wiki.openmrs.org/display/docs/Performance+Tuning

Do you have any experience with these parameters?

-XX:+CMSPermGenSweepingEnabled

-XX:+CMSClassUnloadingEnabled

It looks like we are experiences a classloading leak due to the user of Groovy in the UI framework... I checked yesterday and there were 4,000+ dead classloaders on our production most of which were Groovy classloaders.  Darius put a tweak into the UI Framework last night, but I don't think it has resolved it.

Mark

[Jeremy Keiper]

AMPATH's production machine uses these values, and we have fought with the PermGen errors before:

-Xmx3584M -Xms512M  -XX:PermSize=512M -XX:MaxPermSize=512M -XX:NewSize=256M

We also turned on -XX:+UseConcMarkSweepGC 

Right now we are fighting with CPU load, and have not explored the options Mark just mentioned.

Jeremy Keiper
OpenMRS Core Developer
AMPATH / IU-Kenya Support

[Saptarshi Purkayastha]

Hi Mark,

If you are using CMS for the JVM, you should ensure that your processor has enough cores, which I expect most modern servers would.
But please check... or else your system would hang which GC goes on.

I'll be a little scared to enable PermGenSweeping on OpenMRS or for that matter any system that uses Spring, lazy loading and localization.
From my use of profiling OpenMRS core, PermGen creates a sawtooth graph if GC is enabled on PermGen. Unless you are really constrained on RAM, please do not enable it.

On the other hand, class unloading might work much better. This is especially true for scripting languages that run on JVM. Rhino for sure speeds up quite a bit with CMS class unloading enabled. I've not tried JRuby or JPython, but I've read somewhere that these languages are the biggest gainers from G1 style compacting. CMS is not a compacting garbage collector and thus, doesn't do class unloading.

So I'd recommend that you try enabling it and running YourKit or another profiler.
Simplest I'd suggest using jdk7_update 4 and higher, which enables G1 by default

---
Regards,
Saptarshi PURKAYASTHA

My Tech Blog:  http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

[Mark Goodrich]

Thanks for the tips!
Mark

[Ellen Ball]

Dear Saptarshi, Rowan, Jeremy and Bill -

This is a perfect example of our fantastic global OpenMRS community.  

Thanks for all the help,

Ellen

"Men anpil, chay pa lou." 

Many hands make the load light.

-- Haitian Proverb

[Mark Goodrich]

A couple follow-on questions... (forgive my ignorance! :)

How many cores would you recommend for using CMS?

You said that CMS does not do class unloading... but yet the parameter I saw was "CMSClassUnloadingEnabled".  Do you mean that CMS doesn't do class unloading by default?

I'll talk with others about moving up to jdk7, but I'm not sure if we want to risk the potential instability at this point...

Take care,
Mark



On 04/12/2013 05:30 PM, Saptarshi Purkayastha wrote:

[Saptarshi Purkayastha]

If you are running dual processor system, definitely use CMS. If its a single processor with quad-core still use it. I've seen people set one core affinity to tomcat/java process, but I would recommend that in experimental cases. If its a dual-core or worse, one of those fake-extra thread Intel processors (I forget their name), don't enable CMS.

yes, I meant by default... It is also not meant to be compacting GC and hence it's time to GC is unpredictable

The problem with jdk7 at the moment in OpenMRS is only compiler related.
The runtime issues with the infamous PorterStemmer bug or the Lucene/Solr bugs were with the first couple of releases. Since, then I haven't seen any in-the-face stability problems. Performance gains are quite good. If the benchmarksgame@alioth wouldn't have removed java6, I could have shown some evidence :-/

---
Regards,
Saptarshi PURKAYASTHA

My Tech Blog:  http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

[James Arbaugh]

Hi Ellen,

 

We ran into those painful problems in the past too.  One thing we did was added a crontab job to restart tomcat automatically during the night twice a week as “preventative” maintenance.

 

FYI: our production server (with 46GB physical memory) runs stable with the following tomcat configuration…

CATALINA_OPTS="-Xms10240m -XX:PermSize=6144m -XX:MaxPermSize=6144m -XX:NewSize=4096m"

 

Thanks,

James

--