FW: FusionReactor Group: Re: memory usage

[Snake]

Does this have the FR techs stumped too ?

Looks to me like the problems is
- waiting on <0x1ad4835> (a
edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node)

But I have no idea what relates to.


Russ

-----Original Message-----
From: Snake [mailto:snake...@snakepit.net]
Sent: 12 March 2009 15:53
To: 'fusion...@googlegroups.com'
Subject: RE: FusionReactor Group: Re: memory usage

Ok here is another example. All the hung requests seem to look like this
when it happens, and it regularly seems to be caused by me using the
cfadmin.

"jrpp-631" prio=5 in Object.wait()

java.lang.Object.wait(Object.java:???)[Native Method]
- waiting on <0x1ad4835> (a
edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node)
java.lang.Object.wait(Object.java:443)
edu.emory.mathcs.backport.java.util.concurrent.TimeUnit.timedWait(TimeUnit.j
ava:301)
edu.emory.mathcs.backport.java.util.concurrent.helpers.WaitQueue$WaitNode.do
TimedWait(WaitQueue.java:79)
edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync.attempt(Se
maphore.java:346)
edu.emory.mathcs.backport.java.util.concurrent.Semaphore.tryAcquire(Semaphor
e.java:571)
coldfusion.CfmServlet.lock(CfmServlet.java:255)
coldfusion.CfmServlet.service(CfmServlet.java:174)
coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
com.intergral.fusionreactor.filter.FusionReactorFilter.B(null:???)
com.intergral.fusionreactor.filter.FusionReactorFilter.A(null:???)
com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(null:???)
jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletF
ilter.java:42)
coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
jrun.servlet.FilterChain.service(FilterChain.java:101)
jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:
320)
jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428
)
jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:26
6)
jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of Darren Pywell [FusionDebug Team]
Sent: 09 March 2009 15:38
To: FusionReactor
Subject: FusionReactor Group: Re: memory usage


Hi Russ,

The issue here is typically related to the Request Management Features
of CF8 Enterprise. It's possible in the Enterprise Edition to set
Request limits and Queue timeouts on the Request Tuning Screen of the
CF Administrator.

I think that problem is that the engine is looking to find a "slot" to
process the request in but there are not slots available. It then
looks to the queue to remove any requests that have exceeded the Queue
Timeout limit and if it doesn't find any then it puts the request into
a kind of "wait" state until either it times out, or a slot becomes
available. So if the requests are hanging here then you may need to
either lower the "Timeout requests waiting in queue after [nnnn]
seconds", or increase the number of requests that can be run (both of
which can also cause problems!)

All this may be pointing to an issue in the application that requests
are taking too long or it may be that the server is starved of
resource. Take a stack trace with FusionReactor's All Threads and see
what the other threads are hanging on.

Hope that helps,
Darren




On Mar 8, 8:35 pm, "Snake" <snake.li...@snakepit.net> wrote:
> Does FusionReactor have a way to breakdown memory usage and find out what
> requests and what variable scopes are consuming how much memory, like the
> CF8 server monitor does?
>
> Russ

[Darren Pywell [FusionDebug Team]]

Hi Russ,

Did you see my post on this?

http://groups.google.com/group/fusionreactor/msg/d57cef46b843207d

Cheers,
Darren

[Snake]

Hi Darren,

But the problem occurs regardless of how many requests are currently
running. So if there are zero requests running, then every single request
that starts will then have this issue, they will all sit there and
eventually timeout, all having the exact same stack trace.
If I increase the number of requests then it doesn't help, as it just means
more requests sitting there timing out. If I increase the timeout, they just
sit there longer.
So from what you are saying, CF is queuing the requests even though it
doesn't need to as there are slots available.

As I said, I have noticed that this specifically seems to happen if I modify
sandboxes in the cfadmin. Sometimes it will eventually right itself,
sometimes I will have to restart CF. I have added an exclusion filter for
the cfadmin pages so that they do not timeout, but it doesn't work and they
are also being killed by fusion Reactor.

The rule I added it

Regex: /cfide/administrator/(.*)

-
Russ

[charlie arehart]

Just a follow up from this discussion of the last couple of weeks. Russ has
solved his problem where he suffered terrible slowdowns server-wide whenever
making changes in the CF Enterprise sandbox settings. I got online with him
remotely and helped sort things out.

We ultimately determined that it was fixed by an update of his JVM (he'd
been running the default Java 1.6.0_04 that came with CF 8) to the newer
1.6.0_12. Some may know that earlier releases of 1.6 had class loading
issues, fixed as of the 0_10 release.

Anyway, he outlines his observations and the steps to solve the problem in
his blog entry:

http://russ.michaels.me.uk/index.cfm/2009/3/19/ColdFusion-8-performance-Issu
es-when-using-Java-6

For those who don't know, Intergral (makers of FusionReactor) offer such
troubleshooting consulting services. More at
http://www.fusion-reactor.com/support/services/ColdFusionConsulting.cfm.
Though I'm independent (not an Intergral employee), you'll see there that
I'm one of the folks who does help provide those consulting services.

/charlie

[Darren Pywell [FusionDebug Team]]

Charlie + Russ... nice one!

The request throttle where we see the requests backing up is
implemented in the JRun engine but the class loader issue only affects
CF; my guess would be that the JRun thread pool is waiting for a slot
to become free which takes a long time because of the class loader
thus giving the impression that nothing is happening on the box so the
box "hangs" because JRun can't process any CF requests?

Many thanks for sharing the findings with everyone. Hope this helps
others out there!

Darren


On Mar 23, 5:19 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> Just a follow up from this discussion of the last couple of weeks. Russ has
> solved his problem where he suffered terrible slowdowns server-wide whenever
> making changes in the CF Enterprise sandbox settings. I got online with him
> remotely and helped sort things out.
>
> We ultimately determined that it was fixed by an update of his JVM (he'd
> been running the default Java 1.6.0_04 that came with CF 8) to the newer
> 1.6.0_12. Some may know that earlier releases of 1.6 had class loading
> issues, fixed as of the 0_10 release.
>
> Anyway, he outlines his observations and the steps to solve the problem in
> his blog entry:
>

> http://russ.michaels.me.uk/index.cfm/2009/3/19/ColdFusion-8-performan...

> es-when-using-Java-6  
>
> For those who don't know, Intergral (makers of FusionReactor) offer such

> troubleshooting consulting services. More athttp://www.fusion-reactor.com/support/services/ColdFusionConsulting.cfm.

> Russ- Hide quoted text -
>
> - Show quoted text -

[Snake]

Just had this problem again, all stack traces are the same. Can't be the
same problem as last time as I believe we resolved that by using a new JVM
and stopping the slow class loading problem.



Russ

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of Darren Pywell [FusionDebug Team]

Sent: 17 March 2009 19:53
To: FusionReactor

[charlie arehart]

Russ, I see there was no discussion of your issue from last week. I'll
assume you still have the issue, so I have some thoughts/suggestions that I
don't think have been brought up before.

It's rather detailed, but perhaps some of the info will help others who find
this in the future, perhaps even while researching other problems.

(I also notice that while your very first message in the thread back in
March had to do with memory, this one instead has to do with the blocking on
edu.emory.mathcs.backport.java.util.concurrent.Semaphore, so I have changed
the subject accordingly.)

In a previous message of the thread Darren suggested it could be related to
CF's thread/request management. As further evidence, I see that the Java
docs for that class do indicate that it's indeed often used for thread
management:

http://backport-jsr166.sourceforge.net/doc/api/edu/emory/mathcs/backport/jav
a/util/concurrent/Semaphore.html

And further searching shows that others (on other Java EE servers) have had
similar problems, and they have also been seemingly related to thread pool
management (see http://www.icefaces.org/JForum/posts/list/6554.page). Others
have even reported the problems with CF
(http://www.jonhartmann.com/index.cfm/2009/2/5/ColdFusion-Instance-Issue and
http://forums.adobe.com/message/43707?tstart=0), with no resolution--other
than making sure that the three Start buttons in the CF Server Monitoring
tool are disabled. If you are on CF8/9 Enterprise, that's worth checking. To
be clear (for all readers), it does NOT matter whether you have the CF8/9
Server Monitor interface "open". If you hit those "start" buttons, CF then
starts collecting its info whether you have the interface open or not.

I'll assume, Russ, that's not your problem. As far as the thread pool issues
are concerned, though, I think it would help to explore the thread pool
management. Since you deal with lots of servers (if I recall right), can you
clarify/repeat the details of this server in question: what version of CF
you're on (6, 7, 8, 9), what edition (Standard or Enterprise), what
deployment (if Enterprise: Server or Multiserver), and what J2EE server (if
not JRun). The reason I ask all these things is that there can be issues
related to the setting of threads in the CF Admin relative to what they end
up being set under the covers in the J2EE server.

In that regard, can you report as well what your settings are for the CF
Admin Request Tuning page (or Settings, on CF7/6) for simultaneous (request)
threads, max jrun running/queued threads, and if on Enterprise, can you
confirm that the max jrun running thread count is indeed greater than the
sum of the max number thread settings above it?

Then can you also report your settings for the thread-related XML entries
for the JRunProxyService, as found in the jrun.xml file (either in
\[coldfusion]\runtime\servers\coldfusion\SERVER-INF, or in
\JRun4\servers\[instance]\SERVER-INF in a Multiserver deployment). The
parent XML entry is <service class="jrun.servlet.jrpp.JRunProxyService"
name="ProxyService">, and you'll see the various sub-entries related to
threads, like activeHandlerThreads. It just may help to eyeball those to see
if anything is unusual. I realize you may have already done it on your own.
Just offering more eyes to look it over here.

Of course, this is going beyond what FR is about, so again it's just to try
to help. But here's a way that you can actually use FR to help with this
issue of yours.

I'd propose that when your stack traces (and those of others reporting this)
identify threads that are "waiting" on a semaphore lock, it would be helpful
to find which other requests are "holding" that lock. The requests you're
seeing may be merely "victims". As is often the case, you want to find out
who the perpetrator is. You said, "all stack traces are the same", but it's
entirely possible that nearly all were the same, but some one was different,
and it may not have stood out.

Indeed, you may have "walked right past" the perpetrator because of what you
were looking for (again, no offense intended. Just offering this as much for
others as for you.) It would be easy to look only at threads that showed
that semaphore class as the first in the stack trace, viewing them as "the
problem". If my guess is correct, the perpetrator request (or requests)
would not likely show this semaphore class as the first in its stack trace,
but rather it would be further down the trace, where it would have obtained
the lock and then proceeded to do some other operations (which would be
reflected as running at the top of the stack trace).

And I'll note as well that the thread "holding" the lock may also not be a
CF "request" thread but could be some other thread, so you'll want to do a
thread dump (Resources>List All Threads>Stack Trace All).

Sadly, it's also possible that the stack traces may NOT show the "locking"
request having any reference to the semaphore at all. The stack trace shows
only what classes were called as a result of the current operation (whether
that's a CF tag/function in a CFML template, or some other operation under
the cover). I'm just theorizing here, but it could be that a given thread
had obtained the semaphore lock in a previous operation of the request, and
is still holding it across the request, so that at the time you did the
stack trace it shows the request doing other operations but has no reference
to the semaphore in its list of current methods.

If that's the case, then I would argue it's probably very difficult to spot
who's "holding" the lock that the rest are waiting on. I've long wished that
CF had some lock contention reporting, whether for CFLOCK, or for
transactions, or for things like this, and so on. Since such lock-oriented
operations obtain and release locks at their core, they could report on
their doing this, to help diagnose such issues. Even if there were issues
that made it only something that should be turned on for rare occasions, I
could see it helping solve lots of knotty problems.

This may be one of those, but it's possible that some of the simply things
proposed above will help. Let us know when you've digested and can respond
to this, Russ. And kudos to all who had the patience to get to the bottom of
this long message! :-)

/charlie

> -----Original Message-----
> From: fusion...@googlegroups.com
> [mailto:fusion...@googlegroups.com] On Behalf Of Snake
> Sent: Thursday, October 15, 2009 9:42 AM
> To: fusion...@googlegroups.com

[charlie arehart]

Russ, did you ever resolve this issue? There was no more discussion on this
thread after my lengthy reply below. It may be that people were just too
busy to read it. But bottom line, are you still seeing the issue?

[Russ Michaels]

I haven't had it for a while Charlie, but i'll re-open this next time I do.,

 

thanks again

--

You received this message because you are subscribed to the Google Groups "FusionReactor" group.
To post to this group, send email to fusion...@googlegroups.com.
To unsubscribe from this group, send email to fusionreacto...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fusionreactor?hl=.

--
---
Russ Michaels
I.T. Consultant

*Disclaimer:- nothing I say via email  is meant to offend, aggravate, annoy, insult, belittle or in any way imply anything toward any particular person unless expressly stated to the contrary. Mis-interpret this email at your own risk.