Memory Leak with 4.0.6?

[Bilal]

Team:

I have run into a memory issue on one of our servers (using Adobe CF 9.0.1 on Windows) where within minutes available memory would be consumed.

FusionReactor will then attempt to shutdown further request due to limited memory availability. The only way to restart CF is to restart the server.

I created a heap dump and ran initial analysis using MAT. 

It seems like there is large character array filled with a starting 1 followed by many zeros (600 MB of them): "100000000...". There were no request running when I did the dump.

The likely thread that MAT is pointing me to is occupying 78% of memory shows integral plugins.

Has anyone seen this behavior?

Thread Report

--------------------

Leak Suspects» Leaks» Problem Suspect 1» Description» Thread Stack Thread Stack

jrpp-546

  at java.math.BigInteger.<init>([C)V (BigInteger.java:380)

  at java.math.BigDecimal.bigTenToThe(I)Ljava/math/BigInteger; (BigDecimal.java:3363)

  at java.math.BigDecimal.bigMultiplyPowerTen(I)Ljava/math/BigInteger; (BigDecimal.java:3491)

  at java.math.BigDecimal.setScale(II)Ljava/math/BigDecimal; (BigDecimal.java:2373)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:488)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:408)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:458)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:382)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:408)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:458)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:408)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:458)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:356)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materializeForUx(Ljava/lang/Object;Ljava/lang/Object;)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:321)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.createUxReturnList(Ljava/util/LinkedList;Z)Lcom/intergral/fusionreactor/plugin/caps/PayloadList; (AMFDecorationDescriptor.java:265)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.materialize()V (AMFDecorationDescriptor.java:168)

  at com.intergral.fusionreactor.plugin.amf.AMFDecorationDescriptor.freeze()V (AMFDecorationDescriptor.java:564)

  at com.intergral.fusionreactor.plugin.amf.AMFStreamProcessor.finishing()V (AMFStreamProcessor.java:107)

  at com.intergral.fusionreactor.plugin.amf.AMFDecoratorManager.finishing(Ljava/lang/Integer;)Lcom/intergral/fusionreactor/plugin/caps/IDecorationDescriptor; (AMFDecoratorManager.java:82)

  at com.intergral.fusionreactor.plugin.amf.AMFPlugin.finishing(Ljava/lang/Integer;)Lcom/intergral/fusionreactor/plugin/caps/IDecorationDescriptor; (AMFPlugin.java:74)

  at com.intergral.fusionreactor.filter.FusionReactorFilter.c(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V (FusionReactorFilter.java:526)

  at com.intergral.fusionreactor.filter.FusionReactorFilter.d(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V (FusionReactorFilter.java:262)

  at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (FusionReactorFilter.java:171)

  at jrun.servlet.FilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (FilterChain.java:94)

  at coldfusion.filter.FlashRequestControlFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (FlashRequestControlFilter.java:71)

  at coldfusion.bootstrap.BootstrapFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (BootstrapFilter.java:46)

  at jrun.servlet.FilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (FilterChain.java:94)

  at jrun.servlet.FilterChain.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (FilterChain.java:101)

  at jrun.servlet.ServletInvoker.invoke(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljrun/servlet/InvokerChain;)V (ServletInvoker.java:106)

  at jrun.servlet.JRunInvokerChain.invokeNext(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (JRunInvokerChain.java:42)

  at jrun.servlet.JRunRequestDispatcher.invoke(Ljrun/servlet/ServletConnection;)V (JRunRequestDispatcher.java:286)

  at jrun.servlet.ServletEngineService.dispatch(Ljrun/servlet/ServletConnection;)V (ServletEngineService.java:543)

  at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(Ljava/lang/Runnable;)V (JRunProxyService.java:203)

  at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(Ljava/lang/Runnable;)V (ThreadPool.java:320)

  at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(Ljava/lang/Runnable;)V (ThreadPool.java:428)

  at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(Ljava/lang/Runnable;)V (ThreadPool.java:266)

  at jrunx.scheduler.WorkerThread.run()V (WorkerThread.java:66)

[charlie arehart]

Good news, Bilal. Yes, this is a known issue, with a solution. It came up in fact in another thread here today, but it was before you joined the group today (and welcome!)

You can find the discussion in this thread (via the Google Groups web site):

http://groups.google.com/group/fusionreactor/browse_thread/thread/0373d4f738cd05f9?hl=en_US#

As the thread explains (a few messages down), the problem is not a bug in FR but rather a bug in the JRun connector that FR is tripping over. I recall our discussion last week of your work with a Tomcat connector for IIS that you wrote for the Railo project (http://tomcatiis.riaforge.org/), and you mentioned how you had found many problems with the CF web server connectors. This is another of them. :-)

The good news for you is that the FR folks did implement a workaround, and you can get it free by updating your FR 4.0.6. While you could go with 4.0.10, you may as well go for 4.5 (released just last week), which is free for all FR 4 users. That should solve your problem (and offer a few more nice touches. There’s a technote about 4.5 here, http://www.fusion-reactor.com/support/kb/FRS-291.cfm, and I am also working up a blog entry to highlight things a little differently.)

Let us know if that helps.

/charlie

 

--
You received this message because you are subscribed to the Google Groups "FusionReactor" group.
To view this discussion on the web visit https://groups.google.com/d/msg/fusionreactor/-/zJGB33jCU5QJ.
To post to this group, send email to fusion...@googlegroups.com.
To unsubscribe from this group, send email to fusionreacto...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fusionreactor?hl=en.

[David Stockton]

Hi All,

Whilst your issue sounds very similar and could very well be the same
problem, the evidence posted so far doesn't 100% confirm that. Having
said that, in the first instance I would still request you upgrade
your FR installation to the latest 4.5 release (which is a free update
for all FRv4 users).

If the upgrade is something you're able to do, it'd be great to hear
back from you as to whether this resolves your problem. Otherwise
please post again or contact sup...@fusion-reactor.com and we'll see
how we can help you further.

Best regards,
David Stockton
Fusion Team

On May 30, 10:14 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> Good news, Bilal. Yes, this is a known issue, with a solution. It came up in
> fact in another thread here today, but it was before you joined the group
> today (and welcome!)
>
> You can find the discussion in this thread (via the Google Groups web site):
>

> http://groups.google.com/group/fusionreactor/browse_thread/thread/037...
> cd05f9?hl=en_US#

>
> As the thread explains (a few messages down), the problem is not a bug in FR
> but rather a bug in the JRun connector that FR is tripping over. I recall
> our discussion last week of your work with a Tomcat connector for IIS that
> you wrote for the Railo project (http://tomcatiis.riaforge.org/), and you
> mentioned how you had found many problems with the CF web server connectors.
> This is another of them. :-)
>
> The good news for you is that the FR folks did implement a workaround, and
> you can get it free by updating your FR 4.0.6. While you could go with
> 4.0.10, you may as well go for 4.5 (released just last week), which is free
> for all FR 4 users. That should solve your problem (and offer a few more

> nice touches. There's a technote about 4.5 here,http://www.fusion-reactor.com/support/kb/FRS-291.cfm, and I am also working

> To view this discussion on the web visithttps://groups.google.com/d/msg/fusionreactor/-/zJGB33jCU5QJ.

[Bilal]

@Charlie:

Thanks for all the feedback Charlie. Digging into the info.

@Dave:

We will try this in a two step approach: 

a) Disable the FusionReactor AMF Dissector Plugin which seems to be a large part of the problem call chain/stack

b) if a) does not resolve the issue upgrade to 4.5

[Bilal]

Team:

just feedback so far.

We have disabled this the  FusionReactor AMF Dissector Plugin and the memory error has not re-occurred.

We had also noted this plug-in in some cases where we saw high CPU utilization without any user or scheduled CF request running. On a cursory attempt of analysis, when digging into Thread dump this secondary issue appeared to be with due to unclosed network streams (java.net.DatagramSocket), forcing CPU cycles into wait (java.lang.Object.wait ). This secondary issue is still under observation. Will post if I have more.

[David Stockton]

Hi,

Thanks for the feedback. Let us know how it goes.

Best regards,
David Stockton
Fusion Support Team