Memory problems when Dashboard's Exception Viewer enabled

[Joe Roberts]

Hi,

We have a fairly large MachII app, running on Railo 3.1, using MachII
1.6.1.8 and the Dashboard 1.0.0.8

When we enable the "Exception Viewer" on the "Debugging" tab of the
dashboard, the servers suffer from a memory 'leak' and eventually
crash.

I have annotated a Fusion Reactor memory graph which illustrates the
problem: http://www.twitpic.com/o0n8n/full

I looked at the dashboard logging code, and could see no reason why
this was happening, but suspect that there may be a bug?

How can I ensure that the Exception viewer is never enabled, is there
a property I can define in our app's MachII config xml?

Thanks

Joe

[Peter J. Farrell]

Are you consistently reloading the application using the Dashboard?
There is a known memory leak there somewhere that we cannot track down
at the moment -- but it only occurs in certain circumstances.

Somehow I doubt it is the exception viewer. It only stores a maximum
of 10 exceptions. However, I suggest you enable it, let your
application run for a while and send me a dump of it -- maybe
something is going wrong in the management of it. It's located here:

application[getAppKey()]._MachIIExceptionLoggerData.data

Are you logging any special data to the logs using getLog().debug()
etc.? I think a dump of what is in the exception logger would help.
Please send that to me off list.

How are you creating the AppKey in your Application.cfc? If you are
not setting MACHII_APP_KEY and you are accessing the application from
different URLs, different app keys can be dynamically generated which
could possibly create a memory leak. This was fixed in the 1.8
release:

https://greatbiztoolsllc.trac.cvsdude.com/mach-ii/ticket/373

Best,
.Peter

[Joe Roberts]

Hi, thank you for your quick response.

We reload the app at least once a day, but not via the Dashboard.

We do not log any special data to the logger. Currently, we do not
have a "Logging" <property> definition in our MachII config xml. Would
defining one, and specifying it as disabled, have the equivalent
effect disabling the exception viewer in the Dashboard? It is
important that I work out how to keep this, as a temporary solution.

We define the appKey as a MachII config property <property
name="appKey" value="store" />, does this rule out the possible issue
with dynamically created MACHII_APP_KEYs?

I found the logger data in application.MachIIExceptionLoggerData.data,
the only item I had in application.myappkey is appLoader. I will email
this dump to you off list, it is quite large!

All the best

Joe

[Joe Roberts]

Quick update...

- I updated our MachII app to the newer (recommended) Application.cfc
bootstrap method, ensuring that the app key is defined as
MACHII_APP_KEY
- I added a logger <property> and set loggingEnabled to false
- I deployed the changes and saw the same problem of rapidly rising
memory usage

I have had to remove the Dashboard module from the application, for
the time being.

Joe

[Peter J. Farrell]

Joe,

I have the dashboard deployed on several Adobe 8 and Open BD servers. I'm beginning to think this has to do with Railo.

When the memory runs up, are there more than 10 items in the exception viewers? Or is it limited to 10 items.

.Peter

[Joe Roberts]

Normally 10 items, but bizarrely, I have noticed 11 items, on 2
occasions (including the sample I emailed).

I've disabled the Dashboard module temporarily. But I intend to
revisit it, as we would like to run the Dash for the caching info/
controls. I'll keep you posted if I make any discoveries. If there is
anything you would like me to test, then let me know.

Is it possible to disable the Exception Viewer functionality in a way
that persists, without modifying the Dashboard code?

Thanks.

Joe

[Peter J. Farrell]

Joe,

I've add the ability to disable the exception logger in the 1.1.0 BER.  However, 1.1.0 is compatible with Mach-II 1.8+ so you'd have to upgrade to Mach-II Simplicity and us the Dashboard BER or modify the dashboard XML configuration to explicitly disable the logger (it's a simple boolean flag).  You can grab the a BER zip from mach-ii.com on the "code" page on the right hand side -- look for Dashboard BER.

At the moment, I'm a bit stumped regarding your issue -- either it's a Railo issue (possible because they've fixed a few issues in Railo 3 pertaining to Mach-II) or something else entirely.  I suspect Railo because I have the dashboard running on Adobe 8 and Open BD on several high traffic site and it's fine.  I suspect that Railo is holding on to a reference to something (possibly due to arrays here -- I know they handle them differently).  Might be worth to ping the Railo list -- basically we're storing an array of structs (each struct has a big array in it) in the application scope and then deleting the last elements after the array reaches 10 elements.

I believe the reason why you see 11 elements sometimes is just due to race condition, but that shouldn't be a problem.  I was asking how may element you see to make sure there weren't 1000 elements in there.  I may write a sized array caching strategy to solve this (if warranted -- need to talk to the team here on this one).

Best,
.Peter

Joe Roberts said the following on 11/03/2009 11:03 AM:

[Joe Roberts]

Thanks Peter,

I've been through the Dashboard code, and I'm also stumped. However,
since I have a setup where I can produce the problem, I plan to do
some tests to try to identify the exact cause, but this will have to
wait a few days at least.

I don't think that there is any more you can offer at this stage.
Thanks very much for your time and input, I'll keep you informed of
anything I discover.

Joe

[Peter J. Farrell]

Ok, thanks for giving me a heads up on where you going with this Joe.  I mostly think this is a Railo issue since it appears to be working just fine on our heavy traffic systems (on Adobe CF and/or Open BD).  Let us know if there is anything else we can do -- I'd be happy to fix an issue if I had had a clue to what is going on.

.pjf

Joe Roberts said the following on 11/04/2009 07:08 AM: