classloader deadlock?

[Chris]

I'm looking through stack traces (mmmmmm ... stack traces!) and am
wondering if I'm seeing some sort of deadlock, or if that's just the
way things work.

In the threads below, it appears they are all waiting on the same
thing ... all of which they have locked. and all of which they are
waiting on.

Are these threads correctly waiting to load class files, or is
something else going on?

[If it's related, our ColdFusion Template cache size=1024, and there
are many many more .cfm and .cfc files on the server. We use 25
simultaneous requests, and there were fewer than that running when
FusionReactor generated the stack traces]

thanks for any ideas,
Chris

I see these two entries in every thread:
- waiting on <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
- locked <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)

"jrpp-810" prio=5 tid=0x04201af0 nid=0x4ac in Object.wait()
[0x5a0af000..0x5a0afd94]
at java.lang.Object.wait(Native Method)
- waiting on <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.AbstractCache.fetch(AbstractCache.java:46)
- locked <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at coldfusion.util.SoftCache.get(SoftCache.java:81)
at
coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:
422)
at
coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:
704)
at
coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:
685)
at
coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:
616)
at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:341)
at coldfusion.runtime.CfJspPage._emptyTag(CfJspPage.java:1916)
at cfApplication2ecfm1914301910.runPage(F:\pool\content\Scripts\cdrh
\cfdocs\cfMAUDE\Application.cfm:54)

"jrpp-808" prio=5 tid=0x04c05948 nid=0x152c in Object.wait()
[0x598af000..0x598afd94]
at java.lang.Object.wait(Native Method)
- waiting on <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.AbstractCache.fetch(AbstractCache.java:46)
- locked <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at coldfusion.util.SoftCache.get(SoftCache.java:81)
at
coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:
422)
at
coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:
704)
at
coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:
685)
at
coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:
616)
at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:341)
at coldfusion.runtime.CfJspPage._emptyTag(CfJspPage.java:1916)
at cfApplication2ecfm1914301910.runPage(F:\pool\content\Scripts\cdrh
\cfdocs\cfMAUDE\Application.cfm:54)

"jrpp-804" prio=5 tid=0x04c998a0 nid=0x134c in Object.wait()
[0x51cff000..0x51cffd94]
at java.lang.Object.wait(Native Method)
- waiting on <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.AbstractCache.fetch(AbstractCache.java:46)
- locked <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at coldfusion.util.SoftCache.get(SoftCache.java:81)
at
coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:
422)
at
coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:
704)
at
coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:
685)
at
coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:
616)
at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:341)
at coldfusion.runtime.CfJspPage._emptyTag(CfJspPage.java:1916)
at cfApplication2ecfm1914301910.runPage(F:\pool\content\Scripts\cdrh
\cfdocs\cfMAUDE\Application.cfm:54)

"jrpp-803" prio=5 tid=0x52cf5758 nid=0x18e0 in Object.wait()
[0x516ff000..0x516ffd94]
at java.lang.Object.wait(Native Method)
- waiting on <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.AbstractCache.fetch(AbstractCache.java:46)
- locked <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at coldfusion.util.SoftCache.get(SoftCache.java:81)
at
coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:
422)
at
coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:
704)
at
coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:
685)
at
coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:
616)
at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:341)
at coldfusion.runtime.CfJspPage._emptyTag(CfJspPage.java:1916)
at cfApplication2ecfm453599847.runPage(F:\pool\content\Scripts\cdrh
\cfdocs\cfPMN\Application.cfm:35)

"jrpp-785" prio=5 tid=0x04cc2960 nid=0x12e0 in Object.wait()
[0x59faf000..0x59fafd94]
at java.lang.Object.wait(Native Method)
- waiting on <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.AbstractCache.fetch(AbstractCache.java:46)
- locked <0x15a456a8> (a coldfusion.util.AbstractCache$Lock)
at coldfusion.util.SoftCache.get(SoftCache.java:81)
at
coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:
428)
at
coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:
704)
at
coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:
685)
at
coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:
616)
at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:341)
at coldfusion.runtime.CfJspPage._emptyTag(CfJspPage.java:1916)
at cfApplication2ecfm453599847.runPage(F:\pool\content\Scripts\cdrh
\cfdocs\cfPMN\Application.cfm:35)

[Darren Pywell]

Hi Chris,

Take a look at this Adobe KB article and see if what you think; could
be your case:

http://kb2.adobe.com/cps/865/cpsid_86589.html

Hope that helps,
Darren

[Chris]

Thanks Darren. That's CF8, a little different from our CF7.02 (I
forgot to note that earlier). In that Adobe technote it's not clear
what to do with that issue, except load the CF8 fix.

This comment of theirs doesn't really help, either:
"Here is an example stack trace of a thread that is encountering a
deadlock and would be listed in multiple consecutive thread dumps.
This type of stack trace can occur normally and doesn't necessarily
indicate an issue with ColdFusion."
I guess they mean the issue is with JRun not ColdFusion?

We'll keep on digging :-)

Regards,
Chris

[Darren Pywell]

Hi Chris,

I think that this is all related to CF but what they mean with the
comment is that you have to see that state persist. I don't think that
they were meaning to imply it's not CF related although it does kinda
come across that way when you read it :-)

I think that the issue you are seeing though is basically the same
issue, although you may not be seeing a deadlock, perhaps only poor
performance.

Do you see that state remain once it's occurred or does it every
clear. What I mean by that is are exactly the same threads (thread
names) hanging forever in the same place or do they eventually clear
and simply get replaced with other threads loading other files?

We have seen a performance issue once where someone had the files
located on a share and had a lot locks like this. The reason was the
amount of time that the Template class loader was taking.

Hope that helps,
Darren

[David Stockton]

Hello Chris,

I'd also point out that we offer a CF consulting service where we can assist you in detail with your issue(s).

Details are available through our dedicated consulting brand http://www.cfconsultant.com/ - consulting from the makers of FusionReactor.

Best regards,
David Stockton
Fusion Support Team

--
You received this message because you are subscribed to the Google Groups "FusionReactor" group.
To post to this group, send email to fusion...@googlegroups.com.
To unsubscribe from this group, send email to fusionreacto...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fusionreactor?hl=en.

[charlie arehart]

Chris (besides what Darren helpfully said in reply), since you're on 7 and
this is related to class loading, what version of the JVM are you using?
When you view the "system information" page in the CF admin (the link is at
the top), what does it report? CF 7 came with a variation of 1.4. CF8 and
8.01 came with variations of 1.6, and both variations had a classloading
bug--in the jvm, which was fixed in JVM 1.6.0_10.

If you may have changed CF to point to a different JVM than that which it
comes with, perhaps you're pointing at one of those old 1.6 JVM versions. A
bit of a wild guess, but worth at least double-checking.

/charlie

[Chris]

Hi all, thanks for your comments.

Darren -- we'll track the threads to see how they change in
consecutive stack traces.

David -- I've put in a request to management for consulting support,
but they're not very keen. I haven't quite got to the point yet where
it's worth it out of pocket :-)

Charlie -- we run JVM 1.4.2_15. The thread dumps header is: Full
thread dump Java HotSpot(TM) Server VM (1.4.2_15-b02 mixed mode).
We're in the process of getting our customer up to ColdFusion9, so
we'll be on JVM 1.5 soon. It's tempting to try newer JVMs, but our
management prefers staying with Adobe recommendations.

I've been checking out tools:
SeeFusion's SeeStack has helped some.
IBMs Thread & Monitor Dump Analyzer -- does not work on the Sun JVM.
Samurai - his website does not appear to be responding now.
Thread Dump Analyzer on Java.net - helpful, though I'm still trying
to figure out what I'm seeing :-)

What tools to others use?

best,
Chris

[charlie arehart]

Yep, that's fine. I was only raising the concern if you had in fact updated
your CF7 jvm to point at 1.6 (and any early build of it). But you have not,
so it would seem the bug in those earlier 1.6 builds is not affecting you.
(But then again, it's always possible that the bug *was* there in 1.4 but
Sun did not address it, since 1.4 was likely out of support by the time 1.6
came out and was fixes, in its update _10.)

That said, if any of you are on CF 8 or 8.01, which does come by default
with the older 1.6 JVM updates that had the bug, not that Adobe has in fact
authorized upgrading it. More at:

http://www.carehart.org/blog/client/index.cfm/2011/10/28/CF911-Have-you-upda
ted-your-ColdFusion-JVM-to-24-yet-Important-security-fix-for-CF-89

Finally, Chris, you say "We're in the process of getting our customer up to

ColdFusion9, so we'll be on JVM 1.5 soon."

Actually, no, you'd be on 1.6. Both CF 8 and 9 come with 1.6 (successively
higher updates of it). While CF 7 came with 1.4, the timing of CF8's release
meant that there was never a version of CF that came with java 1.5. Just
offering that clarification if it may be helpful.

Looking forward to hearing how your problem resolves with the info Darren
has shared.

/charlie

PS As for "other tools", it depends on what you want to do. Do you mean for
looking at stack dumps? I find them straightforward enough to just read by
hand (once you've worked with them a few times, that becomes easier.)
SeeStack is a good tool to start with, too. I mentioned it as well as an old
Adobe technote on reading stack traces and thread dumps in a blog entry that
may interest you:

http://www.carehart.org/blog/client/index.cfm/2009/6/24/easier_thread_dumps

> -----Original Message-----
> From: fusion...@googlegroups.com
> [mailto:fusion...@googlegroups.com] On Behalf Of Chris
> Sent: Friday, December 16, 2011 6:18 PM
> To: fusion...@googlegroups.com
> Subject: Re: [fusionreactor] Re: classloader deadlock?
>