Where/how to look for 'lost' memory?
Peter Boughton
FusionReactor is highlighting that we appear to have a memory problem
- we can get large spikes as memory usage goes up and comes down, but
it doesn't come down all the way to how it was before, and we're not
sure why.
This happens for requests which are not storing anything in persistant
scopes, but obviously something is holding on to that memory.
We found something that suggests there's a CF bug about clearing the
Variables/Request scopes at the end of a request, so we've added
<cfset StructClear(Variables)/> and <cfset StructClear(Request)/> to
the end of onRequestEnd() which has helped a bit, but hasn't solved
the problem.
We've got -XX:+AggressiveHeap in the JVM args, which again appeared to
help but hasn't solved the problem.
There's certainly some inefficient code issues we're looking into, but
would these be causing memory to be continually used even after the
request has ended?
One thing I thought might be helpful is if FR had a report that listed
memory before and after a request started, and then we could
investigate those with the biggest increases - but I think FR only has
either start/finish memory, not both?
Does anybody have any other ideas on where to look for the missing
memory, and/or suggestions on how to ensure things are correctly
cleaned up?
Thanks.
Peter
Charlie Arehart
target the cause of such issues), and I'll look forward to hearing the
thoughts of others here.
But as for a possible explanation, I'll ask if your code uses CFCs. There
have been some recent debates (and even a hotfix) proposing to address
issues related to memory leaks asserted to be in CFCs. I say asserted,
because there are those arguing for and against. Just do a search on:
cfc memory leak
And read the various takes. Still, your current solution may well be
resolving that. If you're not using CFCs, then it might be interesting to
hear what *IS* in your variables and request scope before you clear it. That
should give a clue.
Of course, some things increase memory but are not tied to variables (local
or shared), such as cached queries. Are you using any of those?
As for FR helping track memory issues, one challenge is that it doesn't
really know what memory was used BY a given request but rather just shows
what total memory was before or after a given was run. It makes it hard to
discern whether a given request is the cause of such a memory change, as it
could have been caused by other requests.
Indeed, the problem could be related not to any one request, but many small
ones. Are you confident you're not being hit by spiders, bots, web services
request, RSS feed readers, CFHTTP requests, and the like? Those can cause
problems by not presenting cookies, which means if requested repeatedly,
they could create new session and client variables on each page request
(where normal browsers would share one set of these across many requests).
Just a thought.
I, too, will look forward to any more thoughts from the bright bulbs at
Intergral, who have certainly seen and helped solve many such problems.
/charlie
Peter Boughton
>> I'll ask if your code uses CFCs. There
>> have been some recent debates (and even a hotfix) proposing to address
>> issues related to memory leaks asserted to be in CFCs.
Yep, we've got CFCs, and I think the StructClear thing we did was in
response to those memory leak claims - after googling for cfc memory
leak I came across the pages someone showed me when we were discussing
that.
Most of the CFCs are persisted in the Application scope, but don't (or
shouldn't) have any data stored during normal events, because they're
essentially function libraries, not data objects.
We will be doing a code review of the CFCs shortly, so if there is
anything dodgy happening in them hopefully we'll pick it up.
>> cached queries. Are you using any of those?
Not many. We've got a handful using cf caching, and maybe a handful
manually cached in Application scope (Due to them using
cfqueryparams), but not a huge number, and nothing particularly big.
(probably the biggest is the country list, which has a handful of
columns and a couple of hundred rows max)
>> Are you confident you're not being hit by spiders, bots, web services
>> request, RSS feed readers, CFHTTP requests, and the like?
Pretty much, yeah - it's a backoffice system with IP restrictions, so
spiders/rss/etc shouldn't be able to get to it.
Don't think we've got any web services or cfhttp - certainly nothing
I've noticed in Request History.
We don't use client vars (and they're switched off), but it could be
useful to add something to onSessionStart to verify what/when sessions
are created and see if that shows anything useful.
Thanks,
Peter
Charlie Arehart
your question. I know I didn't answer it directly, so it still stands.
As for your reply on client vars being "switched off", I do want to ask one
point of clarification for everyone following along (for their own sake as
well as yours). Do you mean you use clientmanagement="no" in
application.cfm/cfc? I've just seen so often that folks say "we don't use
client vars", but they leave that enabled, which still causes CF to create a
set of built-in client vars regardless of code referencing any "client."
vars. I sense you mean you do not use that attribute, just wanted to be
clear. :-)
Here's where I wrote more on that challenge, for those interested:
http://carehart.org/blog/client/index.cfm/2006/10/4/bots_and_spiders_and_poo
r_CF_performance
Peter Boughton
\ \
Peter Boughton
blog.bpsite.net
/ /
Charlie Arehart
To: fusion...@googlegroups.com
Peter Boughton
Unfortunately fixing the null query thing wasn't the total solution,
but it does appear to have helped (the server doesn't run out of
memory /every/ day now).
If anyone has any more suggestions they would be appreciated.
Thanks,
Peter
On Aug 7, 4:56 pm, "Charlie Arehart" <char...@carehart.org> wrote:
> Thanks, but I'm glad we had this exchange, because it gives me the chance to
> clarify to all that just "setting it to none in the Admin" does not disable
> it. It's a frequent and unfortunate misconception.
>
> While the enabling/disabling of application and session vars (on their
> respective page) does override whether one can enable them (and the timeouts
> they can set) in code, the same is not true of client vars.
>
> The client var page is just the default storage if none is specified in
> code. So if one did happen to have clientmanagement="yes/true" in code, that
> would still enable it. Now, it is true that by setting no default in the
> admin (which would otherwise default to the registry at installation), this
> does mean that one in code must also set clientstorage as well. But again,
> one could, and it would not be disabled by any setting in the Admin.
>
> As for the large query result, I think the Intergral guys will tell you
> they've seen some memory problems with very large queries. I'll let them
> speak to any more detail.
>
> /charlie
>
> _____
>
> From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
> On Behalf Of Peter Boughton
> Sent: Tuesday, August 07, 2007 11:38 AM
> To: fusion...@googlegroups.com
> Subject: FusionReactor Group: Re: Where/how to look for 'lost' memory?
>
> Yep, we've got it turned off in CF Administrator (storage set to "none"),
> and it wasn't mentioned in Application.cfc so should default to "no" - I've
> just explicitly set it to false to be sure.
>
> We've found something that might have been causing problems - a query that
> in some circumstances could return several thousand nulls (!). Haven't had
> any significant memory jumps since fixing that, but that might be a
> coincidence; it's too early to tell if that was the sole cause.
>
> On 8/7/07, Charlie Arehart <char...@carehart.org> wrote:
>
> Sure, and I hope that others (from Intergral or else wise) will please note
> your question. I know I didn't answer it directly, so it still stands.
>
> As for your reply on client vars being "switched off", I do want to ask one
> point of clarification for everyone following along (for their own sake as
> well as yours). Do you mean you use clientmanagement="no" in
> application.cfm/cfc? I've just seen so often that folks say "we don't use
> client vars", but they leave that enabled, which still causes CF to create a
> set of built-in client vars regardless of code referencing any "client."
> vars. I sense you mean you do not use that attribute, just wanted to be
> clear. :-)
>
> Here's where I wrote more on that challenge, for those interested:
>
> http://carehart.org/blog/client/index.cfm/2006/10/4/bots_and_spiders_...
Darren Pywell
Here are a few ways in which you can get further with such problems:
1. FusionReactor Request Log Files
FusionReactor logs all of the requests to log files so you can see the
Memory when the request started and when the request finished by
watching the log entries. As Charlie correctly mentions, FR logs the
total memory (not the memory used by just this request), but if you
are using a system where you can control the requests being executed
then you can be fairly sure of the memory usage of a request;
You can analyze the log files easily in Excel; this is a great way to
visualize memory issues using graphs:
http://www.fusion-reactor.com/support/kb/FRS-161.cfm
2. FusionReactor Resource Log Files
FusionReactor also logs the Memory and CPU used at a user defined
interval. By changing the sampling interval (lowest value is 100ms)
you can get more detailed insight into memory usage of the server
while request(s) are running.
3. Adding Memory Trace Markers to the Request
Using FusionReactor's API you can add trace statements to the request
that is running at specific points. These trace statements are added
to the Request Details "Markers" tab and via the JDBC tab if the JDBC
requests are being stored "In Order".
<cfscript>
frapiClass = createObject("java",
"com.intergral.fusionreactor.api.FRAPI");
frapi = frapiClass.getInstance();
RuntimeStatic = createObject("java", "java.lang.Runtime");
rt = RuntimeStatic.getRuntime();
maxmem = rt.maxMemory()\1024;
freemem = rt.freeMemory()\1024;
totalmem = rt.totalMemory()\1024;
usedmem = totalmem-freemem;
frapi.trace( "Used Memory: " & usedmem);
</cfscript>
http://www.fusion-reactor.com/support/kb/FRS-110.cfm
So you can get much more detailed information by adding such calls
before and after queries are run and at other points in your code
where you believe the problems can be.
I've logged an enhancement request for FR (#FR-949) to also show the
memory when started on the history screens; and in the next version of
FR the memory data can be accessed directly from FRAPI without the
need for the extra object create and call to the Java Runtime
#FR-722).
Darren
Darren Pywell
Peter Boughton
I tried using Excel to look at the log files, but I couldn't get it to
produce a sensible graph - trying to plot memory over time with a line
graph just doesn't work as it should. (Obviously that's an Excel
problem not an FR one though)
Is it safe to assume that increasing the log file size to something
like 5-10MB as well as increasing interval to give more detail over
time would be fine so long as we have enough disk space to store the
data? Or are there any side effects to large log files?
Adding some of those markers should definitely be useful - if I put
them in the various Application.cfc/on~Start & End functions it will
probably give us some interesting information, and there's certainly
some big functions it'd be nice to check.
Thanks!
Peter
Darren Pywell
Increasing the size of the log files or the number of log files should
be fine;
I'm wondering why you don't get any nice graphs out of excel; I do
this a lot because it often helps me to find the issues and lets you
get a good long term overview. Perhaps you could send a sample
screenshot to the sup...@fusion-reactor.com inbox and I can take a
look at what you see.
The markers are great stuff. You can learn a lot instrumenting your
code this way. Here is a tip for you: if you place the calls to
FusionReactor FRAPI in CFTRY/CFCATCH blocks then the code will also
run on servers that don't have FusionReactor installed. The CFCATCH
block can simple ignore any errors. That way you can move the code
between machines that may not have FR onboard without worries...
Darren
Peter Boughton
I've just used Google Docs and managed to get a graph fine, so the
data is okay.
I'll fiddle about some more with Excel tomorrow, and if I can't get it
working then I'll email you.
The try/catch is a good point - we've only got FR on our live
production machines, not on our staging or individual dev machines.
Thanks again,
Peter
On Aug 9, 4:51 pm, Darren Pywell <dapyw...@googlemail.com> wrote:
> Peter,
>
> Increasing the size of the log files or the number of log files should
> be fine;
>
> I'm wondering why you don't get any nice graphs out of excel; I do
> this a lot because it often helps me to find the issues and lets you
> get a good long term overview. Perhaps you could send a sample
> screenshot to the supp...@fusion-reactor.com inbox and I can take a