OldGen memory increasing too much, then out-of-memory errors

[Chris]

We have a situation where OldGen memory usage is increasing until it's too close to maximum, and applications get out-of-memory errors if they try to run then. The JVM does a GarbageCollection, and typically gets back most of the OldGen memory (more than half the Heap memory). Then applications can run without out-of-memory errors.

We saw this behavior in the Heap memory (thanks FusionReactor!) and narrowed it down to the OldGen memory. This happens in both Intranet and Internet Production, but not in the other environments - Dev, Test, PreProd.

Could I get some thoughts of how best to proceed?

Ideas:
1. Identify what is so big in the OldGen
This seems like going to the source of the issue. I understand that ColdFusion scoped variables live in the OldGen. This is primarily the Application, Session, and Server scopes, due to their longevity. However, I'm not sure how to narrow the focus. There are too many applications to just start doing code reviews.

2. Invoke Garbage Collection sooner.
There is a GC about every 10 minutes now. I figure this is a Full GC. It would be nice to trigger the GC sooner, before there is so little memory remaining. However, this seems like treating the symptom and not the cause.

3. Change the YoungGeneration to OldGeneration ratio.
I've read that objects can be put in the OldGen too soon if they're the wrong size. We'd like to experiment with Young/Old Ratio of size, but since this happens only on Production, we're not real keen on experimenting there ...

many thanks,
Chris

[David Stockton]

Hello Chris,

1 - You can install FREC ( http://www.fusion-reactor.com/fr/plugins/frec.cfm ) and use FusionAnalytics ( http://www.fusion-analytics.com/fa/ ) which could help you identify what's taking up the memory. The Adobe CF monitor has similar functionality in too if you have access to that. The only real way though is to take a heap dump and perform an heap analysis on it.

2 - You could set JVM arguments to request the JVM to GC sooner. However, I don't think that's the best way to go.

3 - Similarly to above, you could forcefully tune this but you're not necessarily attacking the root cause.

With #2 & #3, I'd suggest avoiding where possible. Unless you're specifically seeing confirmed issues with the JVM's own self-tuning I'd avoid trying to give it any tuning hints. The JVM is a complex and very capable piece of code and with respect to the GC engines, self tunes depending on the usage. Even if you go so far to speak to the engineers of the GC engines, many will tell you about the wonderful tuning possibilities but finish with a statement like "don't bother - let it take care of itself".

I'd use a combination of information from FusionReactor to see what's running at the time of the OOM errors and #1 to see what's happening and why you don't have enough memory. It could be that your application simply needs more memory to support your user-load. However, if you're seeing lots of available memory after a GC that's unlikely. What I suspect you'll find is you've got a long running query/page that's holding on to a bunch of memory until it completes. Only after completion will a GC be able to release the memory and thus this is why you're seeing the issue. Therefore attacking the slow query/page will release the memory sooner and keep the system stable.

I'd also mention here that we (makers of FusionReactor) have consulting services that are very likely the fastest and most cost effective route to resolution for you. Further details are on our dedicated consulting brand website: http://www.cfconsultant.com/

Best regards,

David Stockton

Fusion Team

[Chris]

Thanks David, very helpful. I know it was a really big question.

We'll stick with analyzing the issue (no. 1 below). It looks like a heap dump and analysis is our best way forward at present.

Interestingly, we increased the Java Heap to 1280MB recently (with a 192MB PermGen) on a Test server -- and the OldGen is no longer showing the memory increase until almost max. then a GC. And the YoungGen (Eden) is showing much less spiky GCs (Eden formerly looked almost like the teeth of a comb -- many GCs, with little space between them).

So maybe the YoungGen just needed a little more space!

Regards,
Chris

--
You received this message because you are subscribed to the Google Groups "FusionReactor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fusionreacto...@googlegroups.com.
To post to this group, send email to fusion...@googlegroups.com.
Visit this group at http://groups.google.com/group/fusionreactor?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.