Could FusionReactor be the cause of my memory issues?

[Daniel Short]

Hi,

I'm working on tracking down some memory leaks in our application,
where we routinely have a 6 gig instance completely jammed, and
nothing short of restarting the CF server will clear out the memory. I
took a heapdump and analyzed it in MAT and found that I had nearly 2
million retained heap objects that were FusionRequests.

You can see a screenshot of the analyze here:
https://dl.dropbox.com/u/2586403/Heapdump.png

So 80% of the used memory was retained URLClassLoader objects from
FusionReactor. Is there a setting in FR that might be causing this, or
a bugfix that might correct this issue? I'm currently running
Revision: 3.5.1, Build: FusionReactor.78.14986.

I'd really appreciate any information you can offer on what might be
causing this.

Thanks,

Dan

[David Stockton]

Hi Dan,

There is an issue that sounds similar to this that we've seen. From
memory it's related to a bug in the JRun connector that can sometimes
cause this (though I'm not certain of that). I've searched our
ticketing system but without some specific phrases to search on I'm
getting too many results to sort through and find the details. Again
from memory I *think* this is solved in 3.5.5 (the latest 3.5.x
release) but is definitely solved in 4.5 (the latest version) - I
think we were able to find a workaround the JRun issue.
However, assuming it's the issue I'm thinking of (apologies - 3.5 is a
fairly old release and my memory isn't great!) I know the engineer
responsible for that item returns from annual leave tomorrow and so
will be able to shed more light on this.
If you're able to upgrade to 3.5.5 (or preferably the latest version -
4.5 at the time of writing) then I would definitely recommend that as
you'll not only get a bunch of bug-fixes & performance improvements
but also lots of new features including more memory space details in
v4.5.

Upgrade pricing is available on our website if you don't have a valid
maintenance plan (which would entitle you to a free upgrade) -
http://www.fusion-reactor.com/fr/update.cfm

Apologies for not being able to give you an immediate solution but as
mentioned above I'll have one of the engineering team pitch in ASAP.

Best regards,
David Stockton
Fusion Team

[Daniel Short]

I'm going to install 3.5.5 tonight, and hopefully that will fix some
of these long-standing issues on our servers.

Dan

> --
> You received this message because you are subscribed to the Google Groups "FusionReactor" group.
> To post to this group, send email to fusion...@googlegroups.com.
> To unsubscribe from this group, send email to fusionreacto...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/fusionreactor?hl=en.
>

[David Stockton]

Hello Dan,

I'm afraid I only get half marks on this...
The issue is essentially as I discussed (I'll post some more technical
details lower down) BUT the fix is only in 4.0.9 and above.

The issue (for reference it's ticket FR-2661 in our tracking system)
is essentially a bug in the way JRun (and specifically the connector)
handle calls to the standard Java servlet function
HttpServletRequest.getPathTranslated(). Without going into too much
detail, at the "end" of the request you can't call this method in JRun
because JRun calls back to the connector which has prematurely already
started consuming bytes related to the following request. This results
in the connector looking at byte data it believes is a string
(length). As the byte data is long, it's the equivalent of a large
number and JRun connector tries to assign LOTS of memory to fit the
string into memory. This isn't normally possible and typically leads
to server failure. However, on your setup you have a rather large heap
space configured which is probably why you're only seeing massive
memory utilization (rather than actual process failure).

To be clear, this is a JRun/Adobe bug that we have no control over.
However, we have been able to put a work-around in place in
FusionReactor.

I hope that helps!

Best regards,
David Stockton
Fusion Team

[Daniel Short]

So just to be clear, my choices at this point are to either upgrade to
FR 4, or stop using FusionReactor completely right? I understand that
the root cause is a JRun issue, but this problem doesn't manifest
itself if I'm not using FusionReactor, correct?

Dan

[David Stockton]

Hi Dan,

The problem can occur without FusionReactor - it's a generic call that
is used in several pieces of the standard CF system or by other third-
party tools. Whilst FusionReactor v3 did exacerbate the problem, our
work-around in v4.0.9+ stops this.

If you would contact our sales team on sa...@fusion-reactor.com and
mention this thread we'll be happy to offer a discount on your upgrade
to help make that a more attractive option. We would hate to lose you
as a customer and we truly believe the benefit of having FusionReactor
on a server would far out-weigh a very small upgrade charge based on
the feature-set and quality of our product - I hope you agree!

Best regards,
David Stockton
Fusion Team

[charlie arehart]

Not to mention all the great features of FR4 that you'd be getting in the
upgrade. :-)

Dan, I'm just an outside observer so please don't hear me wrong. I
appreciate the situation you're in, and yes it would be nice if it could be
fixed in FR 3, but it's nice to see David's make that generous offer to
offset the need to go to FR4.0.9+ (or the free 4.5 upgrade over FR 4) to get
that problem fixed.

And though he didn't mention it, there really are quite a lot of nice
benefits of FR 4, and indeed more in 4.5, which should also take the sting
out of having to upgrade to solve this problem. You can find a list of
features here:

http://www.fusion-reactor.com/fr/whatsnew4.cfm
http://www.fusion-reactor.com/fr/featurefocus/index.cfm

Hope that's helpful.

/charlie

[Daniel Short]

I appreciate the responses, and there are no hard feelings, just facts
I need to contend with :).

I unfortunately didn't see much in the way of killer features that
would drive me to upgrade to 4.5, but a bugfix that may keep my
servers from crashing would definitely qualify :). If we do move
forward with an upgrade, I will definitely take David and his team up
on their generous offer.

Dan

[charlie arehart]

I hear you. I can think of one feature that may alone be worth it. But
before I share that that, let me just highlight (for you and others here) a
few FR 4 features that I find myself mentioning often to folks, as I help
them make the most of FR in my consulting practice:

- the improved log file archiving is valuable for many, going from managing
by x rotations up to y megs to instead archiving them off every hour into a
ZIP that's kept for 31 days (fully configurable)
- many love the added new insight into JVM metrics
- those who would install FR into more than one thing on a given box should
also appreciate the new FRAM feature
- those using Flash Remoting will benefit from the new AMF dissector feature
- those using FR 4 Enterprise will likely love the new free mobile apps
- those interested in FusionAnalytics should note that only FR 4 logs can be
used by it
- and if one wants to run with CF10 that will only work with FR 4.5
- and still a few more things

Still, I grant that in each of those cases, one may go, "yeah, but I don't
see using those features myself". Fair enough. :-)

Here's something else you may have not appreciated from a surface reading of
the new feature pages, which I think EVERYONE using FR with CF should
appreciate: the FREC plugin (free, but only with available with FR4). This
mild-mannered, perhaps geeky-sounding feature is amazing in the simple new
capability it adds:

- It allows FR to now capture in its logs much of the info that was
previously available only in the CF Enterprise Server Monitor, which itself
logs nothing at all
- And note that it gets that data even for those on CF Standard, who don't
have the Server Monitor at all. That's awesome!
- It works against CF 7, 8, 9, or 10.
- For instance, it tracks (in some new FR logs, primarily realtimestats.log)
such things as the count of CF sessions, count of cached queries (and hit
ratio), count of template cache entries (and hit ratio), a breakdown of all
4 kinds of request queues (in Enterprise, at least), DB Pool statistics, and
much more. These are all stats that CF tracks for free (no overhead, nothing
to "enable"). FREC just grabs and logs them
- FREC also tracks CF request queuing, which FR itself has not previously
had insight into (and again, if on Enterprise, it tracks the queued requests
in the 4 request types), which is also really valuable
- Remember that this is all for CF Standard as well as Enterprise. And as
for Enterprise, some additional info is gathered depending on whether you
enable the one or more of the "start" buttons in the CF Enterprise Server
Monitor, but all that I list above is available even if none of them are
turned on. (And FREC's realtimestats log even tracks whether those 3 buttons
are enabled!)

Note that those of you using FR4+ do need to install it separately, after
installing FR, so you won't see this new info in FR's logs (as new log
files) until you do that. There's nothing changed in the FR interface based
on adding FREC

Yes, I would say that for many, that alone would be reason enough to upgrade
to FR 4 (or the new free upgrade from that to 4.5). But I realize that many
either may fail to appreciate it from reading those pages, or might even
have been using FR 4 for some time and never bothered to add it. Go do it.
It doesn't add any measurable overhead to FR at all, in my experience. For
more info, check out http://www.fusion-reactor.com/fr/featurefocus/frec.cfm.

Still, Dan, I realize you may still not find that compelling enough. As they
say, we can only lead a horse to water... :-) But I offer all this as much
for others reading the thread.

/charlie


> -----Original Message-----
> From: fusion...@googlegroups.com
> [mailto:fusion...@googlegroups.com] On Behalf Of Daniel Short
> Sent: Wednesday, May 30, 2012 12:09 PM
> To: fusion...@googlegroups.com
> Subject: Re: [fusionreactor] Re: Could FusionReactor be the cause of my
memory
> issues?
>

[David Stockton]

@Charlie - Thanks for the great insights and list of notable features!

@Dan - Please feel free to contact me directly on
david_s...@intergral.com and I'd be happy to help with the upgrade
discount offer.

For others reading this, the FR3.5ent -> FR4.5ent upgrade is (at the
time of writing) $499.50 and can be done directly through our website.
Note that with the upgrade you'd also be getting a free/bundled FR AIR
Dashboard license, access to our free iOS & Android apps plus a
version of FR that supports ColdFusion 10 and has lots more features
and performance improvements!

Best regards,
David Stockton
Fusion Team

On May 30, 6:07 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> more info, check outhttp://www.fusion-reactor.com/fr/featurefocus/frec.cfm.