Sudden increase in memory used by CF

[hay...@gmail.com]

Hello all. I am using FR 3.0.0 on CFMX7 / IIS and our server
sometimes experiences spikes in CF memory allocation and usage but the
memory footprint does not go back down. After a cursory examination
of the logs, I can tie the memory spikes to CPU spikes and long
running CF pages, but I am curious why the footprint stays the same
even hours after these offending pages are done processing. Is there
way to view the contents of the memory allocated to CF?

Thanks,
haylo75

[Douglas Spooner (Home)]

This sounds like the problem we have just discovered in CF 6.1 MX which is
also present in CFMX 7

This explains the problem and luckily there is a hotfix for 7.1 MX, but
unlucky for us there isn't one so we are moving to CF8.

http://www.digitalmethod.co.uk/cf/

I've been testing CF8 over the past few days and this problem no longer
exists the memory usage barely budges even when I upload a 800mb file.

Doug

[charlie arehart]

Yep, file uploads are a common killer. Unfortunately, folks often miss the
hot fix because they assume the cumulative hotfixes take care of all of
them. Sadly, they don't, and this one needs to be added manually (as Doug
notes, it's only available for 7, not 6, and it's not needed in 8 as the
problem is resolved.) Also, the title of the Adobe technote and its
description could lead some to presume it doesn't apply to them. Forget
about CFCs (as hinted at in the title and description). If you do file
uploads on CF 7, apply the hotfix.

/charlie

[hay...@gmail.com]

Thanks to both of you for your replies! My FR memory log looked much
like the spikes on the page you linked, Doug. My first question was
going to be about whether the cumulative hotfixes including this
particular item. Thanks to Charlie for beating me to the punch. I
will install the hotfix when I get into work this morning and see what
happens.

On Jul 14, 5:07 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> Yep, file uploads are a common killer. Unfortunately, folks often miss the
> hot fix because they assume the cumulative hotfixes take care of all of
> them. Sadly, they don't, and this one needs to be added manually  (as Doug
> notes, it's only available for 7, not 6, and it's not needed in 8 as the
> problem is resolved.)  Also, the title of the Adobe technote and its
> description could lead some to presume it doesn't apply to them. Forget
> about CFCs (as hinted at in the title and description). If you do file
> uploads on CF 7, apply the hotfix.
>
> /charlie
>
> -----Original Message-----
> From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
>
> On Behalf Of Douglas Spooner (Home)
> Sent: Monday, July 14, 2008 4:41 PM
> To: fusion...@googlegroups.com

> Cc: spoon...@rsc.org; hayl...@gmail.com

[hay...@gmail.com]

I got the hotfix installed and not five minutes after I restarted CF I
saw a memory spike. It's related to a page that ran for almost 5
minutes which also spiked CPU during its execution. I am positive
there are no file uploads in this page, so it appears the issue is
related to something else in that page.

[charlie arehart]

Well, be patient grasshopper. :-)

I know you said there are no file uploads, but even if there were, the fix
wouldn't stop the memory spike from happening. It would just stop CF from
holding the memory usage indefinitely. Instead, the memory (while used) is
now released for garbage collection after the request is complete.

The JVM still then is free on its own to decide when to collect that garbage
(unused memory). How it works depends on the GC algorithm you may choose in
the setup of CF's jvm.config (best for more to leave it as is), and it may
depend also on the version of the JVM in use (1.4 by default in CF 6, 7, and
1.6 by default in CF 8). Sometimes the GC won't take place until total
memory used by CF increases to a high percent of available memory.

You can see both the use and the total memory in the various memory graphs
in FusionReactor. And in FR 3, you can even request a GC to take place in
the System Metrics page's memory graph. (Again, it's just a request. It may
not honor it.)

Bottom line: the spike itself isn't necessarily a problem. The question is
does it remain spiked to the point that subsequent increases in memory use
take the total memory used to or near the limit. In that case, eventually CF
will die.

If you'd like, do let us know if you see memory continuing to grow, and what
percent is used (that's reported on the FR memory pages as well). It will be
interesting to see how things go.

[hay...@gmail.com]

It was hard to track down, but I just happened to be looking at FR the
other day when the memory spike occurred. It turns out a long running
page was trying to loop through a massive query, generating more html
content than I can dream of, then returning a 500 code after awhile.
I had the developer make some changes to the code in that page, so we
haven't seen the spikes to the extent we have before. I still wonder
why the spike in memory use is never cleaned up by GC.

On Jul 15, 9:52 am, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> On Behalf Of hayl...@gmail.com

[charlie arehart]

Well, again, a spike itself isn't unusual. And it could hold it long after
it's no longer needed, if the GC decides it's not yet time to collect it. Of
course, it could also be that the error leaves things in a state that CF
doesn't mark the memory as available, so that would indeed appear to you as
a "leak". You say it's "never cleaned up by GC". Are you saying that in this
instance, if left on its own, it did eventually fill memory and crash? Or is
this more speculation (not that there's anything wrong with that). Also, did
you try to use FR's feature to request a GC? Did that help? If so, then I'd
again argue it would have been cleared up at some point automatically. It
would be interesting to understand this, if you have more thoughts on the
matter.

/charlie

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of hay...@gmail.com
Sent: Monday, July 28, 2008 9:25 AM
To: FusionReactor
Subject: FusionReactor Group: Re: Sudden increase in memory used by CF

[hay...@gmail.com]

Charlie,

Agreed, spikes themselves aren't unusual... when they're spikes. The
spike caused by the failed request in question was never deallocated
from memory. I've tried the FR manual GC request a number of times to
no effect. I have a feeling that it has something to do with
ColdFusion eventually returning a 500 code, but that's just a feeling
with no supporting evidence. Really, I'm not the most conversant
person around when it comes to CF memory management. About 3 of these
"spikes" cause ColdFusion to become very slow to respond, consuming
around 600MB system memory (of 2GB total).

On Jul 29, 12:02 am, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> Well, again, a spike itself isn't unusual. And it could hold it long after
> it's no longer needed, if the GC decides it's not yet time to collect it. Of
> course, it could also be that the error leaves things in a state that CF
> doesn't mark the memory as available, so that would indeed appear to you as
> a "leak". You say it's "never cleaned up by GC". Are you saying that in this
> instance, if left on its own, it did eventually fill memory and crash? Or is
> this more speculation (not that there's anything wrong with that). Also, did
> you try to use FR's feature to request a GC? Did that help? If so, then I'd
> again argue it would have been cleared up at some point automatically. It
> would be interesting to understand this, if you have more thoughts on the
> matter.
>
> /charlie
>
> -----Original Message-----
> From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
>

> On Behalf Of hayl...@gmail.com

[charlie arehart]

But just to be clear, even if the GC request doesn't do anything in the
moment, does the memory eventually go down? Or does the server eventually
crash after these spikes (whether for being out of memory or some other
reason)? That's all I was getting at: whether the sudden memory hit was
permanent (a leak) or just long-lasting that would eventually pass.

I'm not denying that it would be useful to learn even if just was temporary,
as such a large spike (even if temporary) could be a problem for systems
already running at high memory use. It's also not clear: you had said that
your spike was seen to be due to a request looping over a large query and
creating a huge amount of HTML output. You also said it reported a 500 error
after a while. In the case where you tried to do the GC, was the request
completed? If you looked in FR, was the request gone from the running
requests list? If you look at the history (or the request log), can you see
if FR stepped in to try to terminate the request via crash protection? It
just may be useful to understand more about the state of things regarding
the triggering request, to understand better if/why it may have been holding
memory.

[hay...@gmail.com]

The memory usage spike is permanent in this scenario, regardless of
what I try.

The request returning the 500 error did show as complete and was in
the FR request log.

On Jul 30, 8:25 am, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> But just to be clear, even if the GC request doesn't do anything in the
> moment, does the memory eventually go down? Or does the server eventually
> crash after these spikes (whether for being out of memory or some other
> reason)? That's all I was getting at: whether the sudden memory hit was
> permanent (a leak) or just long-lasting that would eventually pass.
>
> I'm not denying that it would be useful to learn even if just was temporary,
> as such a large spike (even if temporary) could be a problem for systems
> already running at high memory use. It's also not clear: you had said that
> your spike was seen to be due to a request looping over a large query and
> creating a huge amount of HTML output. You also said it reported a 500 error
> after a while. In the case where you tried to do the GC, was the request
> completed? If you looked in FR, was the request gone from the running
> requests list? If you look at the history (or the request log), can you see
> if FR stepped in to try to terminate the request via crash protection? It
> just may be useful to understand more about the state of things regarding
> the triggering request, to understand better if/why it may have been holding
> memory.
>
> /charlie
>
> -----Original Message-----
> From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
>

> On Behalf Of hayl...@gmail.com

[charlie arehart]

Thanks. Just to be clear: was the request (as shown as complete in the
request log) showing any indication of having been terminated by FR's Crash
Protection? I just recall some issues in the past where threads were
terminated such that they no longer appeared in the running requests, but
their thread was still running in the background. Perhaps someone from
Intergral can step in if I'm misrepresenting things.

It just seems odd that a page with an error would continue to hold
memory--unless the page put stuff into a shared scope, like session,
application, or server. Can you confirm that that wasn't the case? (If you
happen to be on CF8 Enterprise, you could actually view what is in those
scopes with the Server Monitor or Admin API. Sadly, this just isn't
something FR can tell you.)

/charlie

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of hay...@gmail.com
Sent: Wednesday, July 30, 2008 11:18 AM
To: FusionReactor
Subject: FusionReactor Group: Re: Sudden increase in memory used by CF

[Greg Spencer [FusionReactor Team]]

Hi guys,

Charlie is right. There was an older version of FusionReactor which
had a known issue whereby killing a request which was stuck in a
native method call would incorrectly move it to the history then it
was really still running (this being a limitation of the underlying
Java framework). You can find various articles in our knowledge base
relating to this, eg:

http://www.fusion-reactor.com/support/kb/FRS-25.cfm
http://www.fusion-reactor.com/support/kb/FRS-102.cfm
http://www.fusion-reactor.com/support/kb/FRS-114.cfm
http://www.fusion-reactor.com/support/kb/FRS-149.cfm

With more recent versions of FusionReactor you will find that requests
which are suffering from this problem will remain in the running
requests list and will be marked as awaiting termination... (Please
note that due to the low-level nature of this problem FusionReactor is
not always able to do anything to resolve it but at least you'd be
aware of it and can do something liek schedule a restart at a
convenient time.)

To be honest though, this would be the first time that I've seen this
native method problem linked with high memory usage. (Although I'm
willing to be proven wrong! :) Usually the only symptom is long
running requests which slowly eat up the available threads. If the
memory really is being completely swallowed by this page then could we
once again be hitting a limitation of the underlying Java framework?
If everything is working normally, CF should be able to clear up all
the memory it has allocated (known issues aside), but if Java itself
ever run out of memory then we often see the server as a whole
becoming unpredictable in it's behaviour. Perhaps, when this page eats
all the memory, the CF memory management becomes corrupt and as a
result it becomes impossibe to correctly clean up afterwards?

This is just off the top of my head, but hopefully it's at least food
for thought...


Kind Regards,


Greg
FusionReactor Team





On Jul 30, 6:22 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> Thanks. Just to be clear: was the request (as shown as complete in the
> request log) showing any indication of having been terminated by FR's Crash
> Protection?  I just recall some issues in the past where threads were
> terminated such that they no longer appeared in the running requests, but
> their thread was still running in the background.  Perhaps someone from
> Intergral can step in if I'm misrepresenting things.
>
> It just seems odd that a page with an error would continue to hold
> memory--unless the page put stuff into a shared scope, like session,
> application, or server. Can you confirm that that wasn't the case? (If you
> happen to be on CF8 Enterprise, you could actually view what is in those
> scopes with the Server Monitor or Admin API. Sadly, this just isn't
> something FR can tell you.)
>
> /charlie
>
>
>
> -----Original Message-----
> From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
>

> On Behalf Of hayl...@gmail.com

> Sent: Wednesday, July 30, 2008 11:18 AM
> To: FusionReactor
> Subject: FusionReactor Group: Re: Sudden increase in memory used by CF
>
> The memory usage spike is permanent in this scenario, regardless of
> what I try.
>
> The request returning the 500 error did show as complete and was in

> the FR request log.- Hide quoted text -
>
> - Show quoted text -

[hay...@gmail.com]

I'm on CFMX7, though I'm uncertain whether the thread actually kept
running after FR reports it as complete. That would make sense,
thought i am using FR Revision: 3.0.0, Build: FR-HEAD.967.8763.

The real issue at hand is the long running page itself, which should
never be executed in a manner that causes this to happen. I am
attacking it from that angle right now.

Greg - thanks for the links, I will check them out.

Thanks to all for your valuable input and help!

On Jul 31, 4:54 am, "Greg Spencer [FusionReactor Team]"

<greg_spen...@intergral.com> wrote:
> Hi guys,
>
> Charlie is right. There was an older version of FusionReactor which
> had a known issue whereby killing a request which was stuck in a
> native method call would incorrectly move it to the history then it
> was really still running (this being a limitation of the underlying
> Java framework). You can find various articles in our knowledge base
> relating to this, eg:
>

> http://www.fusion-reactor.com/support/kb/FRS-25.cfmhttp://www.fusion-reactor.com/support/kb/FRS-102.cfmhttp://www.fusion-reactor.com/support/kb/FRS-114.cfmhttp://www.fusion-reactor.com/support/kb/FRS-149.cfm

[hay...@gmail.com]

I found an alarming issue yesterday morning. Someone disabled the
Request Timeout in the CF Administrator and we were also not using the
FR Crash Protection either. I re-enabled the Request Timeout and
things have been running okay for now.

Question: Does FR Crash Protection obey requesttimeout overrides that
are set using the cfsetting tag?

On Jul 31, 8:49 am, "hayl...@gmail.com" <hayl...@gmail.com> wrote:
> I'm on CFMX7, though I'm uncertain whether the thread actually kept
> running after FR reports it as complete.  That would make sense,
> thought i am using FR Revision: 3.0.0, Build: FR-HEAD.967.8763.
>
> The real issue at hand is the long running page itself, which should
> never be executed in a manner that causes this to happen.  I am
> attacking it from that angle right now.
>
> Greg - thanks for the links, I will check them out.
>
> Thanks to all for your valuable input and help!
>
> On Jul 31, 4:54 am, "Greg Spencer [FusionReactor Team]"
>
> <greg_spen...@intergral.com> wrote:
> > Hi guys,
>
> > Charlie is right. There was an older version of FusionReactor which
> > had a known issue whereby killing a request which was stuck in a
> > native method call would incorrectly move it to the history then it
> > was really still running (this being a limitation of the underlying
> > Java framework). You can find various articles in our knowledge base
> > relating to this, eg:
>

> >http://www.fusion-reactor.com/support/kb/FRS-25.cfmhttp://www.fusion-...

[charlie arehart]

I think I can answer this one: No, FR would not pay attention to the
CFSETTING REQUESTTIMEOUT, if you mean that you're wanting it to "let a
request through" that might otherwise fail the CP Timeout protection. FR
operates outside of CF (as a wrapper). It never really "sees your code", but
just watches the request going in and coming out of CF, tracking the details
that it does. The code itself doesn't influence how it works.

That said, there are two things you could consider, if you want to declare a
certain file to not be timed out by FR CP. First, you could add that page as
an exception in the CP Restrictions page.

Second, if you felt that you want to control things programmatically, not by
entry into the Admin, you could look at the FRAPI, which allows you to do a
lot of things programmatically. See the FRAPI guide on the FR site (or
linked to from within the FR help.)

Hey, BTW, care you let us know who this is behind the haylo75 address? I've
looked over the past dozen or so emails from you and see no name. No problem
if you choose to remain anonymous, but if it's just something you'd not
considered and you don't mind sharing, I'm sure some of us would appreciate
knowing who we're talking to when we're replying. :-)

/charlie

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of hay...@gmail.com
Sent: Tuesday, August 05, 2008 9:52 AM
To: FusionReactor
Subject: FusionReactor Group: Re: Sudden increase in memory used by CF

[hay...@gmail.com]

Charlie - that's kind of what I figured on the requesttimeout item.
Thanks for the FRAPI suggestion, I will check that option out. The
good news is I've managed to keep memory footprint in check after
enabling the CF request timeout. I'm keeping my fingers crossed!

No problem, my name is Kerr; I forgot to set that in my group
membership, which I will do now!



On Aug 5, 2:56 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> I think I can answer this one: No, FR would not pay attention to the
> CFSETTING REQUESTTIMEOUT, if you mean that you're wanting it to "let a
> request through" that might otherwise fail the CP Timeout protection. FR
> operates outside of CF (as a wrapper). It never really "sees your code", but
> just watches the request going in and coming out of CF, tracking the details
> that it does. The code itself doesn't influence how it works.
>
> That said, there are two things you could consider, if you want to declare a
> certain file to not be timed out by FR CP. First, you could add that page as
> an exception in the CP Restrictions page.
>
> Second, if you felt that you want to control things programmatically, not by
> entry into the Admin, you could look at the FRAPI, which allows you to do a
> lot of things programmatically. See the FRAPI guide on the FR site (or
> linked to from within the FR help.)
>
> Hey, BTW, care you let us know who this is behind the haylo75 address? I've
> looked over the past dozen or so emails from you and see no name. No problem
> if you choose to remain anonymous, but if it's just something you'd not
> considered and you don't mind sharing, I'm sure some of us would appreciate
> knowing who we're talking to when we're replying. :-)
>
> /charlie
>
> -----Original Message-----
> From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
>

> On Behalf Of hayl...@gmail.com

[charlie arehart]

Thanks for that. :-)

/charlie

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of hay...@gmail.com
Sent: Wednesday, August 06, 2008 10:33 PM
To: FusionReactor
Subject: FusionReactor Group: Re: Sudden increase in memory used by CF