Seefusoin Vs Fusion Reactor
Steve Onnis
Kai Koenig
--
You received this message because you are subscribed to the Google Groups "cfaussie" group.
To post to this group, send email to cfau...@googlegroups.com.
To unsubscribe from this group, send email to cfaussie+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en.
Steve Onnis
From: Kai Koenig [mailto:k...@koeni.de]
Sent: Thursday, 18 February 2010 12:24 PM
To: cfau...@googlegroups.com
Subject: Re: [cfaussie] Seefusoin Vs Fusion Reactor
Kai Koenig
Dave
If you want to see what's in memory in terms of what objects, data,
etc are on the Heap then you will need take a heap dump and analyse
it. I use tools like VisualVM and MAT to do this. See:
http://www.alagad.com/blog/post.cfm/troubleshooting-coldfusion-performance-analysis-part-ii
for details
It takes a while to get the hang of it, but AFAIK, it's the only way.
dave
On Feb 18, 1:46 pm, "Steve Onnis" <st...@cfcentral.com.au> wrote:
> Yeah i can see that but i want to see whats in memory which it cant seem to
> do...or am i missing something?
>
> _____
>
> For more options, visit this group athttp://groups.google.com/group/cfaussie?hl=en.
charlie arehart
OK Steve, there are really multiples issues there: can you
determine what memory is used per request, and whether you can install FR and
SF together. Let me take the last first: that’s easy. The other
deserves more discussion (for those interested) and I’ll offer it in
another note.
First, as for installing SF and FR at the same time, there is absolutely no
problem with that at all. I do it and have seen it done at many sites. Whatever
problem you’re having with the SeeFusion installer would not seem at all
related to FR. I’m sure the SF folks can help with whatever is the issue.
/charlie
--
charlie arehart
Following up my last note, as for determining memory used per
request, I would say there is more to that question than just the simple one
asked. Besides addressing if FR/SF can do it, I would also ask if it’s
really what is needed most times.
FR/SF cannot track memory per request:
------------------------------------------------
First, no, neither FR nor SF have what no feature to track memory use per
request. They operate as servlet filters, watching the requests going in and
out of CF. Despite what it may seem, they have no insight into what goes on in
the request while it’s running (other than taking a stack trace, or
tracking SQL statements).
Only the CF 8/9 Monitor can:
-----------------------------------
For what you want, being able to see memory use per request, the only monitor
offering that is the CF 8/9 Enterprise Monitor, using its “start memory
tracking” feature. As some here will warn, that’s not generally
something that most will want to enable on a production server (the “memory
tracking”, I mean), as it can have substantial overhead (and even add to
CF’s use of memory)—not always, as some have had no problem.
Forewarned is forearmed. And even then, I’ve found that even with memory
tracking turned on, it’s still not always possible to determine what
requests are really using memory.
Is memory per request really the problem?
----------------------------------------------------
Indeed, the fact is (in my experience), the real cause of a lot of use of
memory (that should raise an alarm) is not “per request” use. In
such cases, when the requests finish, they give up their memory they used to be
garbage collected. Now, memory may indeed show as growing even after they
finish, but such “released” memory will eventually be garbage
collected. Now, if memory is NOT GC’ed, then something is holding on to
memory. It’s not been released, and that may cause problems over time.
The far more common cause of memory that can’t be GC’ed is memory
used that’s living beyond the life of any requests, such as sessions, the
application or server scopes, cached queries, and so on. There can also be
memory use (not released) due to a true leak, but those are rare. A common one
is the var scoping bug.
Is an increase in memory really a problem?
----------------------------------------------------
As important, I see some people “sweating” because they see memory
rising in various tools. In most cases, it may not be anything to worry about.
Memory WILL increase, as I said above, when requests start, use memory, and
end. The memory used will be marked to be released, but the JVM may not choose (and
often WILL not choose) to do a GC until memory use approaches the total. People
freak when they see memory “rising”, but it may just be normal.
What matters is if it CANNOT do a GC. That, then, is when you’ll have
problems.
I’ll add that another reason I see some sweat it when it seems memory is “rising”
is that in a tool like SeeFusion or the CF 8/9 Monitor, they may see their memory
as being always nearly full. But let me clarify something: if you have min heap
< max heap, those tools are (unintentionally) fooling you. They are tracking
the percent of used memory to allocated, NOT total memory (max heap).
FusionReactor, instead, does track all three individually: used, allocated, and
total (using total memory as the top of the graph) and computing the percent of
memory used as the ratio of used to total. Since SF and the CF Monitor track
the ratio of used to allocated, and the top of their memory graphs is the
amount allocated, that can be very misleading. Again, for those who do set min
heap=max heap, this is not an issue. That causes the JVM to pre-allocate all
the memory requested, and then the top number (allocated) is the same as used
(the true point at which memory maxes out) in SF and CF.
But back to the point above: with any of these tools (and whether min heap=max
or not), you may well see memory rising and approaching the total limit. I’m
saying: don’t always presume that that’s necessarily something to
worry about. In fact, all 3 tools have in them a button you can click to
request a GC. If you do that, and memory drops like a stone, then it’s as
I said: this memory that was used, released to be GCed, but the JVM simply hadn’t
gotten around to it. If that’s the case, then don’t worry.
But sure, if you’re getting outofmemory errors that are for sure due to
the heap space filling (not all outofmemory errors are about that), then yes
you want to find what is holding on to memory. It may not be any one request,
but the other uses of memory above.
Tracking memory use in Task Manager, other tools
--------------------------------------------------------------
Finally, Dan mentioned watching the memory in task manager (or other OS memory
monitoring), but that is less reliable, as it tracks more than just heap memory
used in the address space. And in fact using those to monitor memory suffer an opposite
impact if you DO set min heap=max heap: since the JVM then pre-allocates all
the memory, the OS perspective shows that all the memory is used, and you no
longer will see it raise/fall with GCs.
Really, you need to use one of the 3 monitors (or as Dave mentioned, a Java monitoring
tool) to really see the heap memory use.
Apologies
------------
I apologize both to those who hate long emails. I’ve added the headings
to try to help. I apologize as well to those who may regard me a tall poppy for
prattling on with all this information, and worse, for correcting assertions
made by others. I realize it’s a risky business, and I really mean no
offense. I just spend my day helping people troubleshoot their CF servers, and
I’m nearly always having to counter these and similar misperceptions in
the community. When you come to understand some of these things, it really
helps put a new perspective on CF. It so often gets blamed for problems that
are really not its fault, or people spend hours/days digging through code to
optimize it, when there’s some larger root cause that may have nothing to
do with that. I’m just trying to help stop the madness. :-)
Hope that’s helpful. Feel free to give me feedback, pro or con, on thoughts
like these whether on-list of off (cha...@carehart.org).
/charlie
From:
cfau...@googlegroups.com [mailto:cfau...@googlegroups.com] On Behalf Of Steve
Onnis
Sent: Wednesday, February 17, 2010 7:41 PM
To: cfau...@googlegroups.com
Subject: [cfaussie] Seefusoin Vs Fusion Reactor
What i am looking for is something that will show me requests based on memory usage to i can see what is consuming memory and how much of it.
--
Steve Onnis
From: charlie arehart [mailto:charli...@carehart.org]
Sent: Friday, 19 February 2010 12:51 PM
To: cfau...@googlegroups.com
Subject: RE: [cfaussie] Seefusoin Vs Fusion Reactor
Kai Koenig
Steve Onnis
To: cfau...@googlegroups.com
Subject: Re: [cfaussie] Seefusoin Vs Fusion Reactor
Kai Koenig
Steve Onnis
Kai Koenig
Brett Payne-Rhodes
http://www.ghidinelli.com/2009/07/16/finding-memory-leaks-coldfusion-jvm
B)
Steve Onnis wrote:
> In which part of your jvm? "Dont know what you mean "
>
> Heap? "huh?"
>
> PermGen? "more huh?"
>
> Premature ageing of objects? "more huh?"
>
> How soon after restart? "Depends on the load. Somtimes can be ok for
> days and other times like now....a few hours"
>
> How often? "Depends on the load"
>
> What else happens on the machine? "Nothing."
>
> What's the CPU load at the time? "CPU is fine so doesnt seem to be any
> endless loops or anything like that"
>
> How does it "die"? "Exhausts the amount of RAM allocated to the CF
> server JVM and just stops responding"
>
> Have you created and analysed stack traces and heap dumps? "No not yet
> because i cant make heads or tails of them anyway"
>
> etc
> etc
> etc
>
> I'm afraid there's no magic bullet such as a "my server hangs - show me
> in which template and why"-button.
>
> :-)
>
> K
>
>
--
Brett Payne-Rhodes
Eaglehawk Computing
t: +61 (0)8 9371-0471
m: +61 (0)414 371 047
e: br...@ehc.net.au
w: http://www.yoursite.net.au
charlie arehart
Steve, I appreciate your frustration. We don’t mean to
suggest that CF troubleshooting is a priestly art. It can certainly seem confusing
to take it all in when you’re under pressure, and worse when it seems that
CF is just a black box which is either up or down. We’re trying to
communicate that you simply have to dig a little further than rely solely on such
gross measures as a rise in the memory use as shown in Task Manager.
Using the tools you have or can get
-------------------------------------------
You mentioned having FusionReactor, and that you couldn’t get what you
wanted (memory per request), but I’ll say again that even without that,
there’s a lot it can help you understand. Beyond that, though, there are indeed
still some other things it can’t help with.
The good news is that with the right tools (some free, some commercial) you can
get a far greater insight into the details like Kai has alluded to. Just
yesterday, I did a CFMeetup session on this very topic: “CF911: ColdFusion
tools for when the stuff hits the fan” (and the recording is now online: http://www.carehart.org/presentations/#cf911_tools).
Now, it was intended to point out tools for a range of possible problems, so I
don’t specifically show how to target a memory problem. That’s a
topic for another session. But at least the few tools I offer may be helpful as
a starting point.
But until I do that other talk, and to try to help you now, here are some more
thoughts.
You’re still wondering “where is memory being held?”. I asserted
that it’s more often in things other than running requests, because in
most cases, people don’t have really long-running requests that run hours.
Yep, it happens sometimes.
Long-running requests
--------------------------
The first step is to determine if you do have any such long-running requests.
In the talk, I showed things as simple as CFSTAT which can help by showing how
many requests are running and the average request time (and I explain that it
does not work in a multiserver/multiple instance deployment).
I then pointed out the CF Server Monitor and how it can help (and without a lot
of overhead) if you’re on CF 8/9 Enterprise. If neither can work, then
either FusionReactor or SeeFusion can, and there are free trials of each. And
the latter three tools go beyond just showing how many requests are running to
actually showing exactly WHAT requests are running (and how long, and what
queries they’ve run, and so on).
After finding out if you have lots of really long-running requests, and I’ll
guess you maybe don’t, you’ll still be left then wondering, “then
what’s still holding memory?”
Tracking sessions, with and without the CF monitor
--------------------------------------------------------------
I did offer a couple of common causes, especially the shared scopes, session,
application, and server. In all three, one can be putting queries, CFCs, or
other potentially large things in, and they will last as long as those last.
Worse, in the case of sessions, you could have spiders, bots, or other
automated tools (even your own ping tools or internal search spiders) that
create an inordinate amount of sessions.
If you have the CF 8/9 Enterprise Server Monitor, there is an “active
sessions” page with a nice graph option (on the right of the page) that shows
active sessions. Many miss that. I’ve helped people who were shocked to
learn they had tens of thousands currently active sessions. That’ll use a
little memory (and more, of course, as you put more things in the session
scope).
But many do not have Enterprise. For them, I then recommend Aussie Mark Lynch’s
awesome ServerStats program, open source, which uses the internal undocumented CF
java objects such as the sessiontracker to tell you how many sessions (and
applications) you have at a moment. Again, yes, I realize some run on servers
where either all java objects or those internal ones have been disabled. Can’t
help you then, of course, but most shops do not have those restricted, so give
it a try. I link to it and offer more info on this problem here: http://www.carehart.org/blog/client/index.cfm/2009/1/22/tracking_count_of_sessions_per_application
Tracking query caching
----------------------------
The other common use of memory I’ve seen is CF’s query caching.
Sadly, the CF Admin only offers a limiter to the number of them. Not their
size. The default is 100. What’s yours set at? I’ve seen shops that
suffered because it was set to 2000—I’ve seen shops that suffered for
a different reason: the setting was 100 but it really needed to be 2000 because
they were using caching and were thrashing the cache, and they had the memory
so should have let more be stored there.
In both cases, the challenge is knowing what’s in the query cache (and
how it’s being used). If you have the CF Server Monitor, it has a page
just for this, the query cache status page. Even if you don’t turn on any
of the “Start” buttons, it still will be helpful, showing the
number of items in the cache, and the hit ratio. Again, sadly, it won’t
tell you the total size of the use of the cache (unless you turn on memory
tracking, which for many is risky). But at least seeing the total count will
give you an indication of whether you’re using it at all. As mentioned
above, the ratio can help with understanding other issues as well.
If one does not have the CF Server Monitor, it’s harder. No one has yet
posted some good code on tracking the query cache using the undocumented
functions. It should be possible to get some info, if not all the details
offered by the CF Server Monitor.
But there are some starting points, Steve.
What’s the real cause of crashing?
-----------------------------------------
As for CF crashing on you, what is your max heap size? You say you’re on 64
bit with 2 GB real memory. Also, what is the *first* error at the time
of your crashes, in the –out.log in the runtime logs ([cf]\runtime\logs\ (or
[jrun]\logs if on multiserver)? That could give a clue as to the real error. Believe
it or not (and I’m speaking to all readers, not just you, Steve), not all
outofmemory errors are due to the heap not being large enough. The details may
help.
Challenges getting help on a mailing list
--------------------------------------------------
All that said, it’s notoriously difficult to help solve such problems on
a mailing list. For one thing, it’s tedious going back and forth. For
another, some on the list may not appreciate the extended thread (sure, they
can ignore it). It’s also difficult to do well, in terms of communicating
things both from the person with the problem and from those trying to help. But
worse, for you, is that it takes a long time to get to the bottom of things,
even if people try to help. As you can see, I’m willing to try.
But I will follow up with Kai’s point and say that sometimes it will be a
challenge for some people to solve such problems on their own, simply for a
lack of experience doing it, and not knowing how to connect all the dots. There
are also a wide variety of opinions and recommendations out there on the net,
and some conflict. I’ll just say again that there are several
people or companies that provide CF troubleshooting assistance. I list several
of them, including myself, at http://www.cf411.com/#cfassist
(or for those on Safari, which doesn’t pass anchors during a redirect, http://www.carehart.org/cf411/#cfassist).
If you want to slog it out here, as you can see, I’m willing to try. Hope
something above helps you.
/charlie
Andrew
I've just been looking at the post below by Chris Peterson and I'm
very interested in this bit in particular:
========================================================
Before: cfset request.leaky = application.leaky
After: cfset request.leaky = duplicate( application.leaky )
This simple example is just an illustration of how you can get
yourself into trouble when pointing a reference to objects from a
shared scope like application or server, into a local scope like
request. When we make a link from the request scope into the
application scope, it maintains a reference such that the request
variable would never go away until the referencing application
variable gets garbage collected (which would wait for the application
timeout). By duplicating the object into the request scope instead,
this ensures there is no reference to the application scope variable,
which allows it to be cleaned up after the request completes.
========================================================
If the code in index.cfm had been
<cfset leaky = application.leaky>
Is the same issue present?
I guess what I really want to know is:
1. What is the default scope in a cfm if you don't specify a scope?
2. What is the best practice for referencing app scope cfc's from a
cfm page?
Many thanks,
Andrew.
On Feb 19, 7:15 am, Dave <davidame...@gmail.com> wrote:
> Steve,
>
> If you want to see what's in memory in terms of what objects, data,
> etc are on the Heap then you will need take a heap dump and analyse
> it. I use tools like VisualVM and MAT to do this. See:http://www.alagad.com/blog/post.cfm/troubleshooting-coldfusion-perfor...