[mule-user] OutOfMemory (GC overhead limit exceeded) and workCompleted exception
Jon Smith
OS: RHEL 5.5
JDK: OpenJDK 1.6.0
Mule: 226
The exception to the console:
Exception in thread "JMS_QA0 Inbound Service.2" org.mule.api.MuleRuntimeException: Component that caused exception is: JMS_QA0 Inbound Service
at org.mule.model.seda.SedaService.handleWorkException(SedaService.java:495)
at org.mule.model.seda.SedaService.workCompleted(SedaService.java:465)
at org.mule.work.WorkerContext.run(WorkerContext.java:367)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:987)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:528)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.TreeSet.<init>(TreeSet.java:124)
at org.mule.transport.MessagePropertiesContext.<init>(MessagePropertiesContext.java:77)
at org.mule.transport.AbstractMessageAdapter.<init>(AbstractMessageAdapter.java:61)
at org.mule.transport.DefaultMessageAdapter.<init>(DefaultMessageAdapter.java:70)
at org.mule.DefaultMuleMessage.setPayload(DefaultMuleMessage.java:575)
at org.mule.DefaultMuleMessage.applyAllTransformers(DefaultMuleMessage.java:663)
at org.mule.DefaultMuleMessage.applyTransformers(DefaultMuleMessage.java:606)
ERROR 2011-03-09 18:31:46,379 [JMS_QA0 Inbound Service.4] org.mule.model.seda.SedaService: Work caused exception on 'workCompleted'. Work being executed was: org.mule.model.seda.SedaService$ComponentStageWorker@3fa5e1cc
Exception in thread "JMS_QA0 Inbound Service.4" org.mule.api.MuleRuntimeException: Component that caused exception is: JMS_QA0 Inbound Service
at org.mule.model.seda.SedaService.handleWorkException(SedaService.java:495)
at org.mule.model.seda.SedaService.workCompleted(SedaService.java:465)
at org.mule.work.WorkerContext.run(WorkerContext.java:367)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:987)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:528)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3221)
Exception in mule.log:
Exception in thread "ActiveMQ Transport: tcp:///127.0.0.1:61616"
java.lang.OutOfMemoryError: Java heap space
at org.apache.activemq.openwire.v2.BaseDataStreamMarshaller.tightUnmarshalByteSequence(BaseDataStreamMarshaller.java:412)
at org.apache.activemq.openwire.v2.MessageMarshaller.tightUnmarshal(MessageMarshaller.java:72)
at org.apache.activemq.openwire.v2.ActiveMQMessageMarshaller.tightUnmarshal(ActiveMQMessageMarshaller.java:66)
at org.apache.activemq.openwire.v2.ActiveMQTextMessageMarshaller.tightUnmarshal(ActiveMQTextMessageMarshaller.java:66)
at org.apache.activemq.openwire.OpenWireFormat.tightUnmarshalNestedObject(OpenWireFormat.java:428)
at org.apache.activemq.openwire.v2.BaseDataStreamMarshaller.tightUnmarsalNestedObject(BaseDataStreamMarshaller.java:118)
ERROR 2011-03-09 18:04:34,345 [JMS_QA0 Inbound Service.4] org.mule.transport.jms.activemq.ActiveMQJmsConnector: Work caused exception on 'workCompleted'. Work being executed was: org.mule.transport.AbstractConnector$DispatchWorker@2e2cab11
at org.apache.activemq.openwire.v2.MessageDispatchMarshaller.tightUnmarshal(MessageDispatchMarshaller.java:71)
at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:347)
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:273)
at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:156)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:136)
I've updated ${MULE_HOME}/conf/warpper.conf to bump up the Xms/Xmx to 3072m ( I can see via ps -ef that the settings have been picked up), it
made no difference.
In the mule-config.xml, pulling messages from ActiveMQ queue and passing back to another ActiveMQ queue. Very simple configuration.
<jms:activemq-connector
name="jmsConnector"
specification="1.1"
brokerURL="tcp://127.0.0.1:61616" />
<model name="JMS_QA0">
<service name="JMS_QA0 Inbound Service">
<inbound>
<jms:inbound-endpoint queue="INQUE"
connector-ref="jmsConnector" />
</inbound>
<outbound>
<pass-through-router>
<jms:outbound-endpoint queue="OUTQUE"
connector-ref="jmsConnector" />
</pass-through-router>
</outbound>
</service>
</model>
This happens at peak times when I need to push tens of thousands of messages through the ESB.
On the ActiveMQ side, no messages are being generated. If I send the same # messages directly to ActiveMQ
and pull them out, no errors occur. So the error seems to be specific to the Mule <-> ActiveMQ transport.
I see error on array copy, so not sure if Mule is building an array based on the # of messages coming in/out.
I've also enabled -verbose:gc ( I can see setting via ps -ef ), but I see no GC output being generated to the console
or the mule.log or any other file in mule. The "GC" seems to be a "red herring". I don't think this has any thing to
do with the heap, but instead, possibly due to running out of bounds of any array.
Any tips on what can be tuned/changed to get passed this issue?
Thanks
Jon
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
Jon Smith
I have tried heap sizes up to 8gb for a single instance, and the problem continues.
If I break apart the inbound data into multiple queues with the same instance, the problem continues.
If I break apart the inbound data into multiple queues and divide the workload between multiple instances. It works.
So, the problem does appear to be with some internal limitation on # of inbound messages that a single Mule ESB
can handle. Maybe associated with an internal array.
What are the limits for inbound messages for Mule ESB?
Any takers?
Andrew Perepelytsya
There's nothing like such a limit in Mule, it only depends on your hardware. Are you sure the gc overhead limit is not coming from the active mq usage? Your next step would be profiling your app to indentify root causes.
HTH,
Andrew
Jon Smith
> Are you sure the gc overhead limit is not coming from the active mq usage?
- If you read my last post.
Single Mule ESB instance -> Single ActiveMQ instance - DOES NOT WORK
Dual Mule ESB instance -> Same Single ActiveMQ instance - DOES WORK
When I go from single Mule to dual Mule instance, all I did was split my # of queues between the two.
ActiveMQ generates now errors. And since it can handle the same # of messages from Dual Mule instances
obviously its not having the memory issues.
> Your next step would be profiling your app to indentify root causes.
I'm running this as a standalone Mule instance where I just have the simple configuration that routes data from one
ActiveMQ queue to another ActiveMQ queue. See my original post. I use that same configuration but just
specify different pairs of IN/OUT queues.
I really have not app.
Its the Mule ESB standalone server running a simple pass-through configuration.
Not sure how I can debug it.
> There's nothing like such a limit in Mule, it only depends on your hardware.
Humm .. its a 12 core ( x 2 socket ) 2.x GHz Nehalem with 114 GB RAM.
For the single mule instance, I've tried Java heap values up to 8 GB, the problem always occurs.
For the dual mule instance, I can run with 3 GB heap each, and the problem does not occur.
I've tried various GC tuning options ( Parallel, SurvirorRatio, etc,etc ). No matter the parameter, the
single Mule esb does work.
Andrew Perepelytsya
Raising the heap size will only worsen the problem if other GC params aren't tuned accordingly (more data for GC to process = more time in GC).
Finally, profiling is NOT debugging, and should be your first tool of choice for such cases. You can download Mule's profiler pack (but have to use trial or buy YourKit profiler to analyze results) or use a profiler of your own.
HTH,
Andrew
Jon Smith
Yes I've read many articles.
Reducing the heap doesn't heap either.
I actually started with 1GB and worked my way up to 8 GB.
I've tried to use -XX:-UseGCOverheadLimit to see what the impact would be. No change.
A little more investigation for a single Mule instance. Take the following scenarios
- Single ActiveMQ inbound queue -> Mule ESB -> Single Active outbound queue
- 5 ActiveMQ inbound queues -> Mule ESB -> 5 ActiveMQ outbound queues
- 10 ActiveMQ inbound queues -> Mule ESB -> 10 ActiveMQ outbound queues
No matter how many queues I split into, once I reach a same total # of messages into a single Mule ESB, it
starts to throw the messages. If I take 1/2 of the queues and move into second instances.
If I switch this to an ActiveMQ only scenario ( writing and reading from the same queue ) and not use the Mule ESB,
There are no problems. So its seems that ActiveMQ can keep up with the loaded.
If I change my Mule configuration to read from ActiveMQ and just delete the message. The problem continues to
occur. So there is an issue with Mule and inbound messages.
Any thoughts ?
Mario Flores
I remember I ran into an exception similar to yours a lot time ago.
You should investigate about OOME(Out of Memory Exception) from ActiveMQ.
I know the error comes on MULE, but is something with AMQ also.
In that time we made an analisys, monitoring the threads on both process (AMQ and MULE) and what we realize is that something in my code was leaving a thread alive on AMQ, until it grows to 800 or 1000 threads alive on AMQ and then the OOME rise.
I suggest you do something similar: for example we use 'prstat -p PID' and monitor the threads parameters (NLWP or LWPS) and you could see how that parameter value increase.
I hope it helps.
Regards
Mario
-------------------------| MaYaCaYo | maya...@hotmail.com | ICQ #23831520 | -------------------------|
> Date: Thu, 10 Mar 2011 08:55:57 -0800
> From: mule.us...@mulesource.com
> To: us...@mule.codehaus.org
> Subject: [mule-user] Re: OutOfMemory (GC overhead limit exceeded) and workCompleted exception
Jon Smith
So I ran the two tests using 10 queues
2 Mule ESB instances ( 5 queues per instance )-> Single ActiveMQ WORKS
1 Mule ESB instance ( 10 queues )-> Single ActiveMQ FAILS
and I monitored the thread activity for ActiveMQ. For both situations, the threads and
connections behaved about the same.
So this seems to still point back to a limitation within the single mule instance.
Maybe it has to do with the # of connections that can be supported in a single mule instance.
Since this is throwing a GC error and the Java heap is not being exhausted ( since I can
go to higher heap sizes and problem occurs at the same spot ), its probably not due to
lack of the general heap but one of the heap pools .. maybe for permanent objects. I did try
MaxPermsize for Mule .. I'll try a higher value.
Otherwise, it looks like I've stumble across a limit in Mule.
Jon Smith
After looking, it may actually have to do with threading on the Mule side.
With the single mule instance, obviously its spawning more threads per instance than if I use the dual mule instance.
I noticed that Mule is spawning 2K # of threads for the single instance. I think this is the problem.
I believe the threads memory comes from permanent heap portion of Java heap.
I increase maxperm to 1G (ouch) allowed it to run a little longer, but stilled failed.
I attempted to keep the threads limited using :
maxThreadsActive="200"
but I still get 2K threads.
So, how does one go about limiting the # of threads that get spawned in Mule ?
Is there a tunable or is this the purpose of maxThreadsActive? If not maxThreadsActive, then what.
Mario Flores
Did you solve your problem with the OOME?
Maybe this article can help you setting the proper parameters. I hope it helps you.
http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html
Regards.
Mario
> Date: Thu, 10 Mar 2011 12:45:00 -0800
> From: mule.us...@mulesource.com
> To: us...@mule.codehaus.org
> Subject: [mule-user] Re: OutOfMemory (GC overhead limit exceeded) and workCompleted exception
>