[cfaussie] JVM Server Memory Management

[amay0048]

Hi All,

Trying to get some input on a memory management problem we're having
on our server.

Been getting strange out of memory errors, under minor server load -
about 10-30 requests per second. What we saw via fusion reactor
monitoring was the memory slowly increasing over a period of about 5
-10 minutes under load until the JVM ran out of memory.

I've isolated a particular cfc which will cause the out of memory
behaviour under load testing, and I've temporarily patched it by
calling the java runtime and manually garbage collecting when this cfc
is called. (Not sure if this is the only problem, but its the best
replication of the problem I have).

I've also moved the JVM across to Jrocket from Oracle and tried a few
tweaks to the JVM memory management settings to settle the process
down. This had made a significant problem, and my best guess right now
is that its something to do with the memory management for the various
generations.

I've been digging through memory dumps to find the problem, but I
can't find anything suspect in relation to coldfusion classes.

I've digging around blogs/forms/adobe documentation and I haven't been
able to find too much info on similar problems.

The most useful post I've found has been this,
http://www.schierberl.com/cfblog/index.cfm/2006/10/12/ColdFusion_memoryLeak_profiler
but the suggestions made do not seem to be the cause of the problem.

I'm wondering if anyone else has had similar problems, and can point
me in the right direction with regards to some possible longer term
solutions.

--
You received this message because you are subscribed to the Google Groups "cfaussie" group.
To post to this group, send email to cfau...@googlegroups.com.
To unsubscribe from this group, send email to cfaussie+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en.

[Kai Koenig]

Hi ,

I guess for a substantial analysis of the problem you'd have to provide more (detailed) info, u.e. actual memory figures, how your JVM is setup, what your system/OS/JRun/CF config, error messages etc etc etc etc. Important would also be what the CFC is doing - calling GC manually is a bad solution - but you have figured that out yourself as it appears :-)

Just as a side note: I'm pretty sure JRocket is not a supported JVM (it will work in most parts but might give you random behaviour and/or non-working features depending on the CF and JVM version).

I assume you've done JVM and GC profiling and logging over a few days and have GC logs available as you're mentioning memory dumps. There are a variety of different types of out of memory errors that could stem from different parts of the JVM. Also have a look into the group archive, we had a bunch of discussions in the last few months in which people have posted links to tools, blog posts, general thoughts and approaches that might help you.

Cheers
Kai

--
Kai Koenig - Ventego Creative Ltd
ph: +64 4 476 6781 - mob: +64 21 928 365 / +61 450 132 117
web: http://www.ventego-creative.co.nz
blog: http://www.bloginblack.de
twitter: http://www.twitter.com/agentK

Hands-on Regular Expression training @ webDU 2010
http://bloginblack.de/agentk/workshop-befriending-regular-expressions/

Hands-on Flash Catalyst and Flex 4 training @ Webinale 2010
http://bloginblack.de/agentk/workshop-rias-with-flash-catalyst-and-flex-4/
--

[Barry Chesterman]

Hello,

If you've isolated a particular CFC that you know causes out of memory errors under load then you should be able to isolate what section of code within the cfc is doing that. If you're not sure how to do this, then I recommend you set up a standalone test where you use just that CFC and do whatever you normally do with it, run it for a while and once your memory usage increases (preferably to as large amount as you can get it) then you need to do a memory dump (jmap will do this for you if you don't want to use some fancy tool, but you may need to dig that tool out of the Java SDK as I don't think it's included in the standard JRE), but google will tell you how to do memory dumps and analysis. Use a memory analysis tool (I use eclipse memory analyzer - MAT) to see the result and the thing that uses up most of your memory heap is the likely cause of your memory issue.

As Kai said there are other recent threads that cover memory diagnostics that you can look at, and you should also provide us with what your actual memory configuration values are, it could be a simple case that your memory is not enough to accommodate the data sets you're loading into it (though not very likely :))

Barry.