CF9 Enterprise, Memory Leak\Resource Issue

[Scott Thornton]

Hi,

 

( In case anyone is still subscribed to cfaussie )

 

I have recently moved some code from cf6 to cf9 enterprise and a particular process consumes all of the resources and hangs the CF server. The exact same code on CF9 standard works okay ( although different windows server version ).

 

The code is processing HL7 messages ( health messaging, admission, discharge messages etc ) and basically it works in a continuous cfloop, selecting the top 1 message from the database, and calls a bunch of extra pages to process the messages.

 

 

My errors look like the following, from the log file:

 

10/12 14:58:20 error ROOT CAUSE:

java.lang.OutOfMemoryError: GC overhead limit exceeded

                at java.util.IdentityHashMap.init(IdentityHashMap.java:244)

                at java.util.IdentityHashMap.<init>(IdentityHashMap.java:190)

                at coldfusion.monitor.memory.MemoryTrackerProxy.registerMemoryTracker(MemoryTrackerProxy.java:182)

                at coldfusion.monitor.memory.MemoryTracker.registerWithObject(MemoryTracker.java:222)

                at coldfusion.monitor.memory.MemoryTracker.onAddObject(MemoryTracker.java:174)

                at coldfusion.monitor.memory.MemoryTrackerProxy.onReplaceValue(MemoryTrackerProxy.java:598)

                at coldfusion.monitor.memory.MemoryTrackerProxy.onReplace(MemoryTrackerProxy.java:551)

                at coldfusion.runtime.Variable.setInternal(Variable.java:143)

                at coldfusion.runtime.Variable.set(Variable.java:71)

                at coldfusion.runtime.LocalScope.bindInternal(LocalScope.java:362)

                at cfsbbp03_pims_hl72ecfm1750835129$funcHL7LISTGETAT.runFunction(D:\WWWRoot\HNEPlatypus2\Batch\sbbp03_pims_hl7.cfm:86)

                at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472)

                at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:368)

                at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:55)

                at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321)

                at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:220)

                at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2582)

                at cfsbbp03_pims_hl72ecfm1750835129._factor7(D:\WWWRoot\HNEPlatypus2\Batch\sbbp03_pims_hl7.cfm:403)

                at cfsbbp03_pims_hl72ecfm1750835129._factor16(D:\WWWRoot\HNEPlatypus2\Batch\sbbp03_pims_hl7.cfm:319)

                at cfsbbp03_pims_hl72ecfm1750835129._factor17(D:\WWWRoot\HNEPlatypus2\Batch\sbbp03_pims_hl7.cfm:317)

                at cfsbbp03_pims_hl72ecfm1750835129._factor24(D:\WWWRoot\HNEPlatypus2\Batch\sbbp03_pims_hl7.cfm:180)

                at cfsbbp03_pims_hl72ecfm1750835129._factor26(D:\WWWRoot\HNEPlatypus2\Batch\sbbp03_pims_hl7.cfm:1)

                at cfsbbp03_pims_hl72ecfm1750835129.runPage(D:\WWWRoot\HNEPlatypus2\Batch\sbbp03_pims_hl7.cfm:1)

                at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

                at coldfusion.filter.CFVariablesScopeFilter.invoke(CFVariablesScopeFilter.java:56)

                at coldfusion.tagext.lang.ModuleTag.doStartTag(ModuleTag.java:271)

                at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)

                at cfsb_hl7_table_processor2ecfm1950658070._factor6(D:\WWWRoot\HNEPlatypus2\Batch\sb_hl7_table_processor.cfm:552)

                at cfsb_hl7_table_processor2ecfm1950658070._factor8(D:\WWWRoot\HNEPlatypus2\Batch\sb_hl7_table_processor.cfm:518)

                at cfsb_hl7_table_processor2ecfm1950658070._factor13(D:\WWWRoot\HNEPlatypus2\Batch\sb_hl7_table_processor.cfm:439)

                at cfsb_hl7_table_processor2ecfm1950658070._factor14(D:\WWWRoot\HNEPlatypus2\Batch\sb_hl7_table_processor.cfm:1)

                at cfsb_hl7_table_processor2ecfm1950658070.runPage(D:\WWWRoot\HNEPlatypus2\Batch\sb_hl7_table_processor.cfm:1)

 

I have moved the process back to my old server ( cf6 ), but the server admin team would prefer that the server is removed sooner rather than later.

 

I am contemplating putting another website on the cf9 server and running it from there.

 

There are few online forums with posts regarding similar problems, which mostly refer to var scoping variables in functions, or are looking at object creation.

 

I am seeking specific advice regarding what to do in this instance. Any gotcha, quick fixes, things to double check?

 

 

Unless explicitly attributed, the opinions expressed in this email are those of the author only and do not represent the official view of Hunter New England Local Health District nor the New South Wales Government..

---

[Simon Haddon]

Without seeing the code my first thought is that you are opening a file stream or file handler without closing it within the loop.  make sure that if there are any open files that they are closed.

Cheers,

Simon

--
You received this message because you are subscribed to the Google Groups "cfaussie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cfaussie+unsubscribe@googlegroups.com.
To post to this group, send email to cfau...@googlegroups.com.
Visit this group at https://groups.google.com/group/cfaussie.
For more options, visit https://groups.google.com/d/optout.

--

Kind regards
Simon Haddon

[charlie arehart]

Scott, my response may surprise you, but hear me out.

The error indicates that you have the CF Enterprise Server Monitor’s “start memory tracking” enabled (see the references to coldfusion.monitor.memory.MemoryTracker). That memory tracking feature has been known to cause problems—and tragically so, because people turn the feature on to understand problems, not realizing that it often causes more (and worse) problems.

And to be clear, this was not something you could have had enabled on CF6, because it came out in CF8. And it may not be that YOU turned this feature on, but perhaps someone else had. And they may even have done it before you moved your code to the server, and perhaps your code is more prone to causing problems with the Memory tracking feature. (Generally the issue is that the more and deeper the objects in memory, the more impact the memory tracking in the CFSM has.)

If you turn this off, and things work find, great (I’ve seen it happen before). But if the process in question still causes issue, we can keep oing.

First, you say that it “consumes all resources”. What mechanism are you using to confirm that? Do you mean that the memory and CPU for CF, as reported in Task Mgr, goes very high? If so, then you want to find out WHY that app does that. And that sounds like where you are, but let’s have you first make sure the CFSM is not the issue.

If the issue is not resolved by turning off the memory tracker, then you will want to use SOME mechanism to understand what is going on (either about the request or about CF, as configured). There are other aspects of that CFSM which could help (some needing the “start monitoring” feature enabled, but some even with no “start” features enabled) to find why the request is running slow and using resources. Better still may be if you had  either of the 3^rd party CF monitors, FusionReactor or SeeFusion.

But let’s start with hearing what you find about the above.

/charlie

--

You received this message because you are subscribed to the Google Groups "cfaussie" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cfaussie+u...@googlegroups.com.

[Scott Thornton]

Hi,

 

Simon:

There are a few places where CFFILE ( with an append) is used to log some things, but nothing that I could call a file stream or file handler.

 

Charlie:

I believe I turned on the memory tracking after the first half-dozen or so server crashes. I’ve turned it off subsequently. ( I have also read of it causing problems).

When the server stops responding to requests, they all have in common a large amount of Memory usage. CPU usage is quite low. I believe that the monitor is not running presently, memory usage is at 3,382MB.

I have used fusion-reactor.com previously on my old servers, and I had hoped I could avoid the cost of current fusion-reactor versions. I have obtained a quote for the new version and will get it installed soon ( hopefully ). I might be able to see if the demo version will work for me.

--

To unsubscribe from this group and stop receiving emails from it, send an email to cfaussie+u...@googlegroups.com.

To post to this group, send email to cfau...@googlegroups.com.
Visit this group at https://groups.google.com/group/cfaussie.
For more options, visit https://groups.google.com/d/optout.

--

Kind regards
Simon Haddon

--

You received this message because you are subscribed to the Google Groups "cfaussie" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cfaussie+u...@googlegroups.com.

To post to this group, send email to cfau...@googlegroups.com.
Visit this group at https://groups.google.com/group/cfaussie.
For more options, visit https://groups.google.com/d/optout.

Unless explicitly attributed, the opinions expressed in this email are those of the author only and do not represent the official view of Hunter New England Local Health District nor the New South Wales Government..

---

[charlie arehart]

Fair enough (that you feel the problem was there before you enabled the CFSM memory tracking). I’ll just say that there may be more to the problem than meets the eye based on the simple task mgr tracking you are doing.

And as for FusionReactor, you don’t need to buy it to try it. There is the free 14-day trial (or demo version, as you called it), assuming you had not already run that trial on the box.

And I’ll add that if you really want to resolve the problem, perhaps in less than an hour, I could help you, including installing and using FR (or again, perhaps the CFSM and its other features, along with the CF logs) to find out what’s going on with your server and that request (which may still be a configuration problem). No need for you to give me remote access. We’d use screensharing. For more, see the consulting page at carehart.org

No, it’s not a free or inexpensive solution, but if the problem is important enough, it may be a small price to pay compared to the time you have spent and would spend. And I have a satisfaction guarantee, so you won’t pay for time you don’t find valuable.  Of course, one challenge is the timezone difference. It’s just after midnight as I write this, and I will be going to bed literally after hitting send.

But I’ll understand if you may prefer to go it your own first. And certainly if you ask questions here I hope to be able to help. But these sort of challenges often prove difficult to spot from the outside with only what you share via email. Sometimes there’s no substitute for having experienced troubleshooting eyes on the variety of diagnostics which may pinpoint the problem. It’s what I do with customers every day, but I try to help in online resources like this where I can.

/charlie

[Paul Swan]

Hi Scott,

I actually work for HNE in the AppDev team so I might be able to help you out by looking at your code.  You can call me on ext. 14193

If I can't help you it would certainly be worth taking Charlie up on his offer as I've worked with him in the past and he's one of the best in the business.

Cheers

Paul

--

You received this message because you are subscribed to the Google Groups "cfaussie" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cfaussie+unsubscribe@googlegroups.com.

To post to this group, send email to cfau...@googlegroups.com.
Visit this group at https://groups.google.com/group/cfaussie.
For more options, visit https://groups.google.com/d/optout.

--

Regards

Paul Swan

[charlie arehart]

Thanks for the kind regards, Paul, and of course for your offering to help Scott directly by looking at his code.

That said, I will say again that with situations like Scott describes, my experience is that the best approach is to use diagnostics to help spot the problem, as the issue may be about configuration rather than coding. But either way, hope his problem gets solved soon.

/charlie