Force Killing of Long-Running Requests

[sjwoo]

Lately, I'm having to force kill a thread like this once a day.

-----

Hello, this is the FusionReactor Crash Protection agent at servercom,
instance: coldfusion.cfmx7.server.

http://server:8088/fusionreactor/fhtml.cfm?mode=main

The following alert was generated at Mon May 04 20:30:31 EDT 2009.

[Timeout/Aborted] Crash Protection - Request Run Time Alert: Actual
value: 60673ms, threshold value: 60000ms, current request: 2009-05-04
17:05:56.711 1241471156711 66235 jrpp-140 10.50.214.66 POST
http://server/default.cfm 60673 36 504896 182552 209536 26983
action=SearchResults 200 0

This request ran for approximately 12274844ms, and has been aborted.

-----

The server is not left in an inconsistent state afterwards, but it
bugs me. Is this the IE bug that is mentioned in the KB entry? Is
there any way I can prevent these from happening? I'm using FR2.0.
Thanks...

[charlie arehart]

@sungjwoo, there are a few things to consider here.

First, you can't really tell if the long-running request is of the nature
described in the technote unless you look to see what the request is doing
at a time when it seems hung. You can do that either manually, clicking the
"stack trace" icon in the running requests page, or it would be sent to you
automatically by email if you enabled email notification (n the
FusionReactor>Settings page).

BTW, you could also change the crash protection action to "notify" rather
than "abort and notify", to give you more time to consider what the request
is doing. What's important is to see the stack trace over a couple of
minutes to see if the request is "stuck" doing the same thing.

It may be simply that the request isn't even "stuck" but is just taking a
long time to do things: it could be involved in database interactions, it
could be doing a CFHTTP to a remote server. In either case, such a request
can't be "killed" anyway until it returns from that action.

In fact, your info below suggests this is what's happened:

[Timeout/Aborted] Crash Protection - Request Run Time Alert: Actual
value: 60673ms, threshold value: 60000ms, current request: 2009-05-04

This indicates the CP alert trigger time to do the kill was 60 seconds

(60000ms) but then later it says:

This request ran for approximately 12 274 844ms, and has been
aborted.

So it did take 200 minutes (or 3 hrs, 20 minutes) for the tag that it was
hung up on to finish. Then when the request came back to CF (and FR saw it),
it killed it before it could even got to the next tag.

/charlie

[sjwoo]

Hi Charlie,

Thanks so much for replying to my query. Here are my responses:

> First, you can't really tell if the long-running request is of the nature
> described in the technote unless you look to see what the request is doing
> at a time when it seems hung. You can do that either manually, clicking the
> "stack trace" icon in the running requests page, or it would be sent to you
> automatically by email if you enabled email notification (n the
> FusionReactor>Settings page).

I actually do have the stack traces sent to me. It's always what's
described in the technote -- java.net.SocketOutputStream.socketWrite0
(Native Method). So since it's a native socket call, it just hangs
forever.

> It may be simply that the request isn't even "stuck" but is just taking a
> long time to do things: it could be involved in database interactions, it
> could be doing a CFHTTP to a remote server. In either case, such a request
> can't be "killed" anyway until it returns from that action.
>
> In fact, your info below suggests this is what's happened:
>
>         [Timeout/Aborted] Crash Protection - Request Run Time Alert: Actual
> value: 60673ms, threshold value: 60000ms, current request: 2009-05-04
>
> This indicates the CP alert trigger time to do the kill was 60 seconds
> (60000ms) but then later it says:
>
>         This request ran for approximately 12 274 844ms, and has been
> aborted.
>
> So it did take 200 minutes (or 3 hrs, 20 minutes) for the tag that it was
> hung up on to finish. Then when the request came back to CF (and FR saw it),
> it killed it before it could even got to the next tag.

I did the hard thread kill -- that's why the request ran for 12274844
ms. If I hadn't killed it, it would probably still be running now.
These requests never come back -- they just keep on going. This
request was nothing more than a database-driven page -- almost always,
it returns within 300ms. It's just that sometimes, it never gets
released. I'm running CFMX7.0.2 and SQL2005. I suppose it's possible
that it's SQLserver that keeps this socket open, but I don't see why
it would do that, either.

- Sung

[sjwoo]

Hi Charlie,

It just happened again. Here's the message and the stack trace, with
the name of the server removed. The message was generated when I hard
killed the thread.

-----

Hello, this is the FusionReactor Crash Protection agent at [server],
instance: coldfusion.cfmx7.[server]

http://[server]:8088/fusionreactor/fhtml.cfm?mode=main

The following alert was generated at Wed May 13 10:49:02 EDT 2009.

[Timeout/Aborted] Crash Protection - Request Run Time Alert: Actual

value: 90829ms, threshold value: 90000ms, current request: 2009-05-13
08:56:28.798 1242219388798 5363 jrpp-47 10.16.41.189 GET http://[server]/default.cfm
90829 12 1036864 132407 1036864 904456 method=[fusebox_circuit] 200 0

This request ran for approximately 6753887ms, and has been aborted.

Running Requests:

2009-05-13 10:49:02.685 1242219388798 5363 jrpp-47 10.16.41.189 GET
http://[server]/default.cfm 6753887 19 1040064 206735 1040064 833328
method=[fusebox_circuit] 200 0



No new alerts will be generated until Wed May 13 10:50:02 EDT 2009

Alert complete.

FusionReactor



JVM Stack Trace
---------------

Thread ID: Reference Handler
Priority: 10
Hashcode: 11492077

"Reference Handler" daemon prio=10 tid=0x0075d760 nid=0x78c in
Object.wait() [0x58a2f000..0x58a2fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x171d40e0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:429)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
- locked <0x171d40e0> (a java.lang.ref.Reference$Lock)

-------------------------------------------------------------------------------

Thread ID: Finalizer
Priority: 8
Hashcode: 8038929

"Finalizer" daemon prio=9 tid=0x0075eb38 nid=0x68c in Object.wait()
[0x58b2f000..0x58b2fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x1721ddf8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x1721ddf8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

-------------------------------------------------------------------------------

Thread ID: Signal Dispatcher
Priority: 10
Hashcode: 20138897

"Signal Dispatcher" daemon prio=10 tid=0x58451af0 nid=0xc68 runnable
[0x00000000..0x00000000]

-------------------------------------------------------------------------------

Thread ID: AdapterThread
Priority: 10
Hashcode: 10893017

Thread not found or already finished.
-------------------------------------------------------------------------------

Thread ID: CompilerThread0
Priority: 10
Hashcode: 5969993

Thread not found or already finished.
-------------------------------------------------------------------------------

Thread ID: CompilerThread1
Priority: 10
Hashcode: 9923343

Thread not found or already finished.
-------------------------------------------------------------------------------

Thread ID: RMI TCP Accept-0
Priority: 5
Hashcode: 16112494

"RMI TCP Accept-0" daemon prio=5 tid=0x5930c250 nid=0x3fc runnable
[0x5a55f000..0x5a55fd8c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x1721d650> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: RMI Reaper
Priority: 5
Hashcode: 18426321

"RMI Reaper" prio=5 tid=0x59a39248 nid=0xd90 in Object.wait()
[0x5a75f000..0x5a75fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x1721ddd8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x1721ddd8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: GC Daemon
Priority: 2
Hashcode: 17332666

"GC Daemon" daemon prio=2 tid=0x5970a4c0 nid=0xc70 in Object.wait()
[0x5a85f000..0x5a85fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x1721ddd0> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x1721ddd0> (a sun.misc.GC$LatencyLock)

-------------------------------------------------------------------------------

Thread ID: RMI RenewClean-[10.19.91.37:3662]
Priority: 5
Hashcode: 24105223

"RMI RenewClean-[10.19.91.37:3662]" daemon prio=5 tid=0x58543fd8
nid=0x79c in Object.wait() [0x5b06f000..0x5b06fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17342f50> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x17342f50> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run
(DGCClient.java:500)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: RMI LeaseChecker
Priority: 5
Hashcode: 272232

"RMI LeaseChecker" daemon prio=5 tid=0x597a6680 nid=0x7a0 waiting on
condition [0x5b16f000..0x5b16fd8c]
at java.lang.Thread.sleep(Native Method)
at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: RMI TCP Accept-1099
Priority: 5
Hashcode: 12002049

"RMI TCP Accept-1099" daemon prio=5 tid=0x5d883148 nid=0xaf4 runnable
[0x62b3f000..0x62b3fd8c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x1765cbf0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: Thread-2
Priority: 5
Hashcode: 22760015

"Thread-2" daemon prio=5 tid=0x59a390e0 nid=0xaac in Object.wait()
[0x5a65f000..0x5a65fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x171d0560> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:429)
at java.util.TimerThread.mainLoop(Timer.java:403)
- locked <0x171d0560> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)

-------------------------------------------------------------------------------

Thread ID: jms-fifo-1
Priority: 5
Hashcode: 7054048

"jms-fifo-1" prio=5 tid=0x591e4e28 nid=0xc20 in Object.wait()
[0x5a95f000..0x5a95fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176023a8> (a jrun.jms.core.threads.FIFOQueue)
at java.lang.Object.wait(Object.java:429)
at jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:
45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
- locked <0x176023a8> (a jrun.jms.core.threads.FIFOQueue)
at jrun.jms.core.threads.ThreadPoolWorker.runWorker
(ThreadPoolWorker.java:91)
at jrun.jms.core.threads.ThreadPoolWorker$1.run(ThreadPoolWorker.java:
49)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: jms-fifo-2
Priority: 5
Hashcode: 9582575

"jms-fifo-2" prio=5 tid=0x591e47e0 nid=0x40c in Object.wait()
[0x5aa5f000..0x5aa5fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17602310> (a jrun.jms.core.threads.FIFOQueue)
at java.lang.Object.wait(Object.java:429)
at jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:
45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
- locked <0x17602310> (a jrun.jms.core.threads.FIFOQueue)
at jrun.jms.core.threads.ThreadPoolWorker.runWorker
(ThreadPoolWorker.java:91)
at jrun.jms.core.threads.ThreadPoolWorker$1.run(ThreadPoolWorker.java:
49)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: jms-fifo-3
Priority: 5
Hashcode: 26622608

"jms-fifo-3" prio=5 tid=0x59692e40 nid=0x744 in Object.wait()
[0x5ab5f000..0x5ab5fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176020b0> (a jrun.jms.core.threads.FIFOQueue)
at java.lang.Object.wait(Object.java:429)
at jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:
45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
- locked <0x176020b0> (a jrun.jms.core.threads.FIFOQueue)
at jrun.jms.core.threads.ThreadPoolWorker.runWorker
(ThreadPoolWorker.java:91)
at jrun.jms.core.threads.ThreadPoolWorker$1.run(ThreadPoolWorker.java:
49)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: jms-fifo-4
Priority: 5
Hashcode: 31385242

"jms-fifo-4" prio=5 tid=0x597a7db0 nid=0x710 in Object.wait()
[0x5ac5f000..0x5ac5fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17602110> (a jrun.jms.core.threads.FIFOQueue)
at java.lang.Object.wait(Object.java:429)
at jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:
45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
- locked <0x17602110> (a jrun.jms.core.threads.FIFOQueue)
at jrun.jms.core.threads.ThreadPoolWorker.runWorker
(ThreadPoolWorker.java:91)
at jrun.jms.core.threads.ThreadPoolWorker$1.run(ThreadPoolWorker.java:
49)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: jms-fifo-5
Priority: 5
Hashcode: 16099115

"jms-fifo-5" prio=5 tid=0x5966fd20 nid=0x628 in Object.wait()
[0x5ad5f000..0x5ad5fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17602030> (a jrun.jms.core.threads.FIFOQueue)
at java.lang.Object.wait(Object.java:429)
at jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:
45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
- locked <0x17602030> (a jrun.jms.core.threads.FIFOQueue)
at jrun.jms.core.threads.ThreadPoolWorker.runWorker
(ThreadPoolWorker.java:91)
at jrun.jms.core.threads.ThreadPoolWorker$1.run(ThreadPoolWorker.java:
49)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: Thread-4
Priority: 5
Hashcode: 28445704

"Thread-4" prio=5 tid=0x5975ee48 nid=0x880 in Object.wait()
[0x5e0cf000..0x5e0cfd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17544b48> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:429)
at jrun.jms.core.messagemgr.Scheduler.next(Scheduler.java:298)
- locked <0x17544b48> (a java.util.LinkedList)
at jrun.jms.core.messagemgr.Scheduler.run(Scheduler.java:233)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: mipc-1
Priority: 5
Hashcode: 3409668

"mipc-1" prio=5 tid=0x59911410 nid=0x844 runnable
[0x5e1df000..0x5e1dfd8c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x172cdfa0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at jrun.jms.core.kernel.mipc.MultiplexConnectionServer.run
(MultiplexConnectionServer.java:154)

-------------------------------------------------------------------------------

Thread ID: Thread-5
Priority: 5
Hashcode: 28884132

"Thread-5" prio=5 tid=0x5d3df298 nid=0x84c in Object.wait()
[0x5e5df000..0x5e5dfd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17554608> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:429)
at java.util.TimerThread.mainLoop(Timer.java:403)
- locked <0x17554608> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)

-------------------------------------------------------------------------------

Thread ID: Thread-6
Priority: 5
Hashcode: 23649247

"Thread-6" prio=5 tid=0x59739b98 nid=0x8a0 in Object.wait()
[0x5e6df000..0x5e6dfd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x175545a8> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:429)
at java.util.TimerThread.mainLoop(Timer.java:403)
- locked <0x175545a8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)

-------------------------------------------------------------------------------

Thread ID: Thread-7
Priority: 5
Hashcode: 20661129

"Thread-7" daemon prio=5 tid=0x5d4daa38 nid=0x864 in Object.wait()
[0x5e7df000..0x5e7dfd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17343780> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0x17343780> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)

-------------------------------------------------------------------------------

Thread ID: Thread-8
Priority: 5
Hashcode: 30345796

"Thread-8" prio=5 tid=0x5d414bb8 nid=0x86c in Object.wait()
[0x5ea0f000..0x5ea0fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17343738> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0x17343738> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)

-------------------------------------------------------------------------------

Thread ID: FusionReactor Web Server
Priority: 5
Hashcode: 27990832

"FusionReactor Web Server" prio=5 tid=0x5d4d5a58 nid=0x87c runnable
[0x5eb0f000..0x5eb0fd8c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x173435f0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at com.intergral.fusionreactor.N.C$_A.run(Unknown Source)

-------------------------------------------------------------------------------

Thread ID: FusionReactor Request Monitor
Priority: 5
Hashcode: 8790658

"FusionReactor Request Monitor" prio=5 tid=0x5d48cd70 nid=0x88c
runnable [0x5ec0f000..0x5ec0fd8c]
at java.util.regex.Pattern.sequence(Pattern.java:1623)
at java.util.regex.Pattern.expr(Pattern.java:1558)
at java.util.regex.Pattern.compile(Pattern.java:1291)
at java.util.regex.Pattern.<init>(Pattern.java:1047)
at java.util.regex.Pattern.compile(Pattern.java:808)
at com.intergral.fusionreactor.B.W.B(Unknown Source)
at com.intergral.fusionreactor.K.A.A(Unknown Source)
at com.intergral.fusionreactor.K.A.A(Unknown Source)
at com.intergral.fusionreactor.K.A.A(Unknown Source)
at com.intergral.fusionreactor.L.C.A(Unknown Source)
at com.intergral.fusionreactor.L.C.run(Unknown Source)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 4437584

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 24527206

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 12563505

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 30347216

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 18297876

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 6040673

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 23442499

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 1942958

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 6842239

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: obj-skimmer
Priority: 5
Hashcode: 6474826

"obj-skimmer" daemon prio=5 tid=0x59663bc0 nid=0xca0 in Object.wait()
[0x5fa6f000..0x5fa6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at coldfusion.server.j2ee.pool.PoolSkimmerThread.run
(PoolSkimmerThread.java:47)
- locked <0x17682a60> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: Thread-13
Priority: 5
Hashcode: 5378181

"Thread-13" prio=5 tid=0x5d9b8c80 nid=0x26c runnable
[0x61dbf000..0x61dbfd8c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x176829b0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at coldfusion.server.jrun4.metrics.CfstatServer.run(CfstatServer.java:
77)

-------------------------------------------------------------------------------

Thread ID: Thread-14
Priority: 5
Hashcode: 29017701

"Thread-14" daemon prio=5 tid=0x5d951d08 nid=0xcd8 in Object.wait()
[0x6203f000..0x6203fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17682688> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:429)
at java.util.TimerThread.mainLoop(Timer.java:403)
- locked <0x17682688> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)

-------------------------------------------------------------------------------

Thread ID: Thread-15
Priority: 5
Hashcode: 20399901

"Thread-15" prio=5 tid=0x5dba8288 nid=0xca4 in Object.wait()
[0x6213f000..0x6213fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: Thread-16
Priority: 5
Hashcode: 1963221

"Thread-16" prio=5 tid=0x5d6b9ba8 nid=0xc9c in Object.wait()
[0x6223f000..0x6223fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: Thread-17
Priority: 5
Hashcode: 11850807

"Thread-17" prio=5 tid=0x5dba8c48 nid=0x298 in Object.wait()
[0x6233f000..0x6233fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: Thread-18
Priority: 5
Hashcode: 30963360

"Thread-18" prio=5 tid=0x5dbdce70 nid=0x300 in Object.wait()
[0x6243f000..0x6243fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: Thread-19
Priority: 5
Hashcode: 5520580

"Thread-19" prio=5 tid=0x5dbace38 nid=0x288 in Object.wait()
[0x6253f000..0x6253fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: Thread-20
Priority: 5
Hashcode: 6391238

"Thread-20" prio=5 tid=0x60449e30 nid=0xcb0 in Object.wait()
[0x6263f000..0x6263fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: Thread-21
Priority: 5
Hashcode: 9814990

"Thread-21" prio=5 tid=0x59882e30 nid=0xcd0 in Object.wait()
[0x6273f000..0x6273fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: Thread-22
Priority: 5
Hashcode: 6951658

"Thread-22" prio=5 tid=0x5d691e18 nid=0xce4 in Object.wait()
[0x6283f000..0x6283fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: Thread-23
Priority: 5
Hashcode: 33543257

"Thread-23" prio=5 tid=0x5d7e0e18 nid=0x248 in Object.wait()
[0x6293f000..0x6293fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: Thread-24
Priority: 5
Hashcode: 13524911

"Thread-24" prio=5 tid=0x5daf4da0 nid=0xcac in Object.wait()
[0x62a3f000..0x62a3fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at java.lang.Object.wait(Object.java:429)
at coldfusion.util.SimpleThreadPool.getAssignment
(SimpleThreadPool.java:116)
- locked <0x176822b0> (a coldfusion.util.SimpleThreadPool)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:211)

-------------------------------------------------------------------------------

Thread ID: MulticastListener
Priority: 5
Hashcode: 15461904

"MulticastListener" prio=5 tid=0x5da8ede8 nid=0xcf4 runnable
[0x62e3f000..0x62e3fd8c]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
- locked <0x176539e0> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0x17653960> (a java.net.DatagramPacket)
- locked <0x17653930> (a java.net.MulticastSocket)
at coldfusion.license.MulticastListener.run(MulticastListener.java:
101)

-------------------------------------------------------------------------------

Thread ID: DestroyJavaVM
Priority: 5
Hashcode: 22706331

"DestroyJavaVM" prio=5 tid=0x00660670 nid=0x2c0 waiting on condition
[0x00000000..0x0011f884]

-------------------------------------------------------------------------------

Thread ID: FusionReactor Web Server (Server Thread Pool Member
Thread-88)
Priority: 5
Hashcode: 10757774

"FusionReactor Web Server (Server Thread Pool Member Thread-88)"
prio=5 tid=0x5923d898 nid=0x270 in Object.wait()
[0x0396f000..0x0396fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x161309b0> (a com.intergral.fusionreactor.L.B.B$_B)
at java.lang.Object.wait(Object.java:429)
at com.intergral.fusionreactor.L.B.B$_B.run(Unknown Source)
- locked <0x161309b0> (a com.intergral.fusionreactor.L.B.B$_B)

-------------------------------------------------------------------------------

Thread ID: FusionReactor Web Server (Server Thread Pool Member
Thread-89)
Priority: 5
Hashcode: 14634388

"FusionReactor Web Server (Server Thread Pool Member Thread-89)"
prio=5 tid=0x60a33930 nid=0xc10 in Object.wait()
[0x03a6f000..0x03a6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x16130a00> (a com.intergral.fusionreactor.L.B.B$_B)
at java.lang.Object.wait(Object.java:429)
at com.intergral.fusionreactor.L.B.B$_B.run(Unknown Source)
- locked <0x16130a00> (a com.intergral.fusionreactor.L.B.B$_B)

-------------------------------------------------------------------------------

Thread ID: Thread-90
Priority: 5
Hashcode: 6186326

"Thread-90" prio=5 tid=0x59698560 nid=0x800 waiting on condition
[0x03c6f000..0x03c6fd8c]
at
com.intergral.fusionreactor.trace.NativeManager.nativeRequestStackDump
(Native Method)
at com.intergral.fusionreactor.trace.NativeManager.requestStackDump
(Unknown Source)
- locked <0x50b82150> (a java.lang.Class)
at com.intergral.fusionreactor.B.W.B(Unknown Source)
at com.intergral.fusionreactor.K.A.A(Unknown Source)
at com.intergral.fusionreactor.K.C.run(Unknown Source)

-------------------------------------------------------------------------------

Thread ID: scheduler-0
Priority: 5
Hashcode: 21954470

"scheduler-0" prio=5 tid=0x5d6a1e30 nid=0x884 in Object.wait()
[0x5f06f000..0x5f06fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:133)
- locked <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:
128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:67)

-------------------------------------------------------------------------------

Thread ID: scheduler-1
Priority: 5
Hashcode: 24258946

"scheduler-1" prio=5 tid=0x59700270 nid=0x838 in Object.wait()
[0x6303f000..0x6303fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:133)
- locked <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:
128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:67)

-------------------------------------------------------------------------------

Thread ID: scheduler-2
Priority: 5
Hashcode: 208682

"scheduler-2" prio=5 tid=0x5da32458 nid=0xad8 in Object.wait()
[0x6373f000..0x6373fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:133)
- locked <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:
128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:67)

-------------------------------------------------------------------------------

Thread ID: scheduler-3
Priority: 5
Hashcode: 17665588

"scheduler-3" prio=5 tid=0x596d7130 nid=0xdc0 in Object.wait()
[0x68c7f000..0x68c7fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:133)
- locked <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:
128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:67)

-------------------------------------------------------------------------------

Thread ID: scheduler-4
Priority: 5
Hashcode: 15931984

"scheduler-4" prio=5 tid=0x59284d98 nid=0x7f8 in Object.wait()
[0x5be6f000..0x5be6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-5
Priority: 5
Hashcode: 4493892

"scheduler-5" prio=5 tid=0x592cf208 nid=0x7fc in Object.wait()
[0x5bf6f000..0x5bf6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-6
Priority: 5
Hashcode: 31382696

"scheduler-6" prio=5 tid=0x592cfb80 nid=0x804 in Object.wait()
[0x5c06f000..0x5c06fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-7
Priority: 5
Hashcode: 6865026

"scheduler-7" prio=5 tid=0x592d05c0 nid=0x80c in Object.wait()
[0x5c16f000..0x5c16fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-8
Priority: 5
Hashcode: 20357445

"scheduler-8" prio=5 tid=0x592d1090 nid=0x810 in Object.wait()
[0x5c26f000..0x5c26fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-9
Priority: 5
Hashcode: 26603258

"scheduler-9" prio=5 tid=0x592d1b60 nid=0x814 in Object.wait()
[0x5c36f000..0x5c36fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-10
Priority: 5
Hashcode: 8151014

"scheduler-10" prio=5 tid=0x592d1e20 nid=0x81c in Object.wait()
[0x5c46f000..0x5c46fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-11
Priority: 5
Hashcode: 24904351

"scheduler-11" prio=5 tid=0x592cc100 nid=0x818 in Object.wait()
[0x5c56f000..0x5c56fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-12
Priority: 5
Hashcode: 9454501

"scheduler-12" prio=5 tid=0x592ccbd0 nid=0x820 in Object.wait()
[0x5c66f000..0x5c66fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-13
Priority: 5
Hashcode: 21568375

"scheduler-13" prio=5 tid=0x592cd6a0 nid=0x780 in Object.wait()
[0x5c76f000..0x5c76fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-14
Priority: 5
Hashcode: 11163028

"scheduler-14" prio=5 tid=0x592ce170 nid=0x2d0 in Object.wait()
[0x5c86f000..0x5c86fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-15
Priority: 5
Hashcode: 14186835

"scheduler-15" prio=5 tid=0x59484010 nid=0x73c in Object.wait()
[0x5c96f000..0x5c96fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-16
Priority: 5
Hashcode: 354244

"scheduler-16" prio=5 tid=0x59484a88 nid=0x784 in Object.wait()
[0x5ca6f000..0x5ca6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-17
Priority: 5
Hashcode: 14645497

"scheduler-17" prio=5 tid=0x59485558 nid=0x824 in Object.wait()
[0x5cb6f000..0x5cb6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-18
Priority: 5
Hashcode: 25112887

"scheduler-18" prio=5 tid=0x59486028 nid=0x828 in Object.wait()
[0x5cc6f000..0x5cc6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-19
Priority: 5
Hashcode: 24655238

"scheduler-19" prio=5 tid=0x59486af8 nid=0x82c in Object.wait()
[0x5cd6f000..0x5cd6fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-20
Priority: 5
Hashcode: 16908692

"scheduler-20" prio=5 tid=0x5d1039e0 nid=0x890 in Object.wait()
[0x5e3df000..0x5e3dfd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: scheduler-21
Priority: 5
Hashcode: 11913785

"scheduler-21" prio=5 tid=0x59cb8818 nid=0x898 in Object.wait()
[0x5e4df000..0x5e4dfd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17205010> (a java.lang.Object)
at jrunx.scheduler.SchedulerService.createRunnable
(SchedulerService.java:181)
- locked <0x17205010> (a java.lang.Object)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: Thread-38
Priority: 5
Hashcode: 15701449

"Thread-38" prio=5 tid=0x6006a520 nid=0x434 waiting on condition
[0x6867f000..0x6867fd8c]
at java.lang.Thread.sleep(Native Method)
at coldfusion.runtime.FormCache.cleanup(FormCache.java:148)
at coldfusion.runtime.FormCache$1.run(FormCache.java:32)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: jrpp-59
Priority: 5
Hashcode: 13949316

"jrpp-59" prio=5 tid=0x5dd54618 nid=0x23c runnable
[0x079ef000..0x079efd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1a8d0540> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-62
Priority: 5
Hashcode: 21934429

"jrpp-62" prio=5 tid=0x5d9b0888 nid=0xd60 runnable
[0x07cef000..0x07cefd8c]
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at jrun.servlet.jrpp.JrppOutputStream$SpillStream.write
(JrppOutputStream.java:182)
at jrun.servlet.io.MetricsOutputStream.write(MetricsOutputStream.java:
75)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:
66)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)
- locked <0x1be83820> (a jrun.servlet.jrpp.JrppBufferedOutputStream)
at jrun.servlet.jrpp.JrppOutputStream.flush(JrppOutputStream.java:45)
at com.intergral.fusionreactor.filter.A.E.flush(Unknown Source)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:
410)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
- locked <0x10251e70> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
at java.io.PrintWriter.flush(PrintWriter.java:120)
- locked <0x10251e70> (a java.io.OutputStreamWriter)
at coldfusion.runtime.NeoJspWriter.flush(NeoJspWriter.java:210)
at coldfusion.runtime.NeoPageContext.flushOutput(NeoPageContext.java:
1626)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:42)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:
22)
at coldfusion.filter.RequestThrottleFilter.invoke
(RequestThrottleFilter.java:115)
at coldfusion.CfmServlet.service(CfmServlet.java:107)
at coldfusion.bootstrap.BootstrapServlet.service
(BootstrapServlet.java:78)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at com.intergral.fusionreactor.filter.FusionReactorFilter.B(Unknown
Source)
at com.intergral.fusionreactor.filter.FusionReactorFilter.A(Unknown
Source)
at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter
(Unknown Source)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke
(JRunRequestDispatcher.java:257)
at jrun.servlet.ServletEngineService.dispatch
(ServletEngineService.java:541)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable
(JRunProxyService.java:204)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable
(ThreadPool.java:318)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable
(ThreadPool.java:426)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable
(ThreadPool.java:264)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

-------------------------------------------------------------------------------

Thread ID: jrpp-68
Priority: 5
Hashcode: 24817648

"jrpp-68" prio=5 tid=0x616e0410 nid=0x994 runnable
[0x03b6f000..0x03b6fd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1b097938> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-71
Priority: 5
Hashcode: 5592579

"jrpp-71" prio=5 tid=0x0074ee68 nid=0x6e4 runnable
[0x03e6f000..0x03e6fd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1bbae300> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-72
Priority: 5
Hashcode: 28279852

"jrpp-72" prio=5 tid=0x0074ed00 nid=0x56c runnable
[0x03f6f000..0x03f6fd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1be1fdd0> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-74
Priority: 5
Hashcode: 3630010

"jrpp-74" prio=5 tid=0x59356408 nid=0x868 runnable
[0x0426f000..0x0426fd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1be219f8> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-75
Priority: 5
Hashcode: 2761327

"jrpp-75" prio=5 tid=0x616e0da8 nid=0xac runnable
[0x0436f000..0x0436fd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1c359f70> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-76
Priority: 5
Hashcode: 14960715

"jrpp-76" prio=5 tid=0x591d2620 nid=0xce8 runnable
[0x046df000..0x046dfd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1c376540> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-77
Priority: 5
Hashcode: 13732486

"jrpp-77" prio=5 tid=0x60aed220 nid=0xc98 runnable
[0x047df000..0x047dfd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1c3c74e0> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-78
Priority: 5
Hashcode: 29496988

"jrpp-78" prio=5 tid=0x5dc14ea0 nid=0xdd4 runnable
[0x048df000..0x048dfd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1c42a720> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-79
Priority: 5
Hashcode: 8152452

"jrpp-79" prio=5 tid=0x5d82b4d0 nid=0x28c runnable
[0x049df000..0x049dfd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1c425250> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-80
Priority: 5
Hashcode: 33398564

"jrpp-80" prio=5 tid=0x60b47dd0 nid=0x274 runnable
[0x04bdf000..0x04bdfd8c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x1c41d450> (a jrun.servlet.io.ReusableBufferedInputStream)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:578)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:570)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:588)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at jrun.servlet.jrpp.JRunProxyService.swapRunnable
(JRunProxyService.java:144)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.swapRunnable
(ThreadPool.java:290)
at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable
(ThreadPool.java:408)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.swapRunnable
(ThreadPool.java:250)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

-------------------------------------------------------------------------------

Thread ID: jrpp-81
Priority: 5
Hashcode: 29938811

"jrpp-81" prio=5 tid=0x58492e08 nid=0x2d8 runnable
[0x04cdf000..0x04cdfd8c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x173a8790> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
356)
at jrun.servlet.jrpp.JRunProxyService.accept(JRunProxyService.java:
104)
at jrun.servlet.jrpp.JRunProxyService.createRunnable
(JRunProxyService.java:120)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: jndi-0
Priority: 5
Hashcode: 32899900

"jndi-0" prio=5 tid=0x5862f028 nid=0x7a4 waiting for monitor entry
[0x5b26f000..0x5b26fd8c]
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:351)
- waiting to lock <0x172d61b8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
356)
at jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:
115)
at jrun.naming.NamingNetworkService.createRunnable
(NamingNetworkService.java:141)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: jndi-1
Priority: 5
Hashcode: 12417174

"jndi-1" prio=5 tid=0x5862f190 nid=0x7a8 waiting for monitor entry
[0x5b36f000..0x5b36fd8c]
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:351)
- waiting to lock <0x172d61b8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
356)
at jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:
115)
at jrun.naming.NamingNetworkService.createRunnable
(NamingNetworkService.java:141)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: jndi-2
Priority: 5
Hashcode: 20444825

"jndi-2" prio=5 tid=0x59a2dbe0 nid=0x7b0 runnable
[0x5b46f000..0x5b46fd8c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x172d61b8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
356)
at jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:
115)
at jrun.naming.NamingNetworkService.createRunnable
(NamingNetworkService.java:141)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: jndi-3
Priority: 5
Hashcode: 18798133

"jndi-3" prio=5 tid=0x59a2dd48 nid=0x7ac waiting for monitor entry
[0x5b56f000..0x5b56fd8c]
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:351)
- waiting to lock <0x172d61b8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
356)
at jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:
115)
at jrun.naming.NamingNetworkService.createRunnable
(NamingNetworkService.java:141)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: jndi-4
Priority: 5
Hashcode: 25248925

"jndi-4" prio=5 tid=0x59250210 nid=0x7b4 waiting for monitor entry
[0x5b66f000..0x5b66fd8c]
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:351)
- waiting to lock <0x172d61b8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
356)
at jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:
115)
at jrun.naming.NamingNetworkService.createRunnable
(NamingNetworkService.java:141)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable
(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable
(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable
(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

-------------------------------------------------------------------------------

Thread ID: tyrex.util.daemonMaster
Priority: 10
Hashcode: 29336913

"tyrex.util.daemonMaster" daemon prio=10 tid=0x59251398 nid=0x7b8 in
Object.wait() [0x5b76f000..0x5b76fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17342c28> (a jrunx.tyrex.services.DaemonMaster)
at jrunx.tyrex.services.DaemonMaster.run(DaemonMaster.java:269)
- locked <0x17342c28> (a jrunx.tyrex.services.DaemonMaster)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: Transaction Domain DefaultDomain
Priority: 5
Hashcode: 26811108

"Transaction Domain DefaultDomain" prio=5 tid=0x592cec98 nid=0x7cc in
Object.wait() [0x5b86f000..0x5b86fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x174c3850> (a
jrunx.tyrex.tm.impl.TransactionDomainImpl)
at java.lang.Object.wait(Object.java:429)
at jrunx.tyrex.tm.impl.TransactionDomainImpl.run
(TransactionDomainImpl.java:1300)
- locked <0x174c3850> (a jrunx.tyrex.tm.impl.TransactionDomainImpl)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: Transaction Domain PrivateJmsDomain
Priority: 5
Hashcode: 11826876

"Transaction Domain PrivateJmsDomain" prio=5 tid=0x596920f8 nid=0x7d0
in Object.wait() [0x5b96f000..0x5b96fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x174c38b8> (a
jrunx.tyrex.tm.impl.TransactionDomainImpl)
at java.lang.Object.wait(Object.java:429)
at jrunx.tyrex.tm.impl.TransactionDomainImpl.run
(TransactionDomainImpl.java:1300)
- locked <0x174c38b8> (a jrunx.tyrex.tm.impl.TransactionDomainImpl)
at java.lang.Thread.run(Thread.java:534)

-------------------------------------------------------------------------------

Thread ID: clock
Priority: 10
Hashcode: 27144547

"clock" daemon prio=10 tid=0x592c07f0 nid=0x83c waiting on condition
[0x5ce6f000..0x5ce6fd8c]
at java.lang.Thread.sleep(Native Method)
at jrunx.tyrex.services.Clock.run(Clock.java:202)

-------------------------------------------------------------------------------

Thread ID: scheduler-0
Priority: 5
Hashcode: 12355986

"scheduler-0" prio=5 tid=0x5d6a1e30 nid=0x884 in Object.wait()
[0x5f06f000..0x5f06fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:133)
- locked <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:
128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:67)

-------------------------------------------------------------------------------

Thread ID: scheduler-1
Priority: 5
Hashcode: 32246026

"scheduler-1" prio=5 tid=0x59700270 nid=0x838 in Object.wait()
[0x6303f000..0x6303fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:133)
- locked <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:
128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:67)

-------------------------------------------------------------------------------

Thread ID: scheduler-2
Priority: 5
Hashcode: 21562117

"scheduler-2" prio=5 tid=0x5da32458 nid=0xad8 in Object.wait()
[0x6373f000..0x6373fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:133)
- locked <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:
128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:67)

-------------------------------------------------------------------------------

Thread ID: SocketTimeout
Priority: 10
Hashcode: 6650219

"SocketTimeout" daemon prio=10 tid=0x5dd4c640 nid=0x4f4 waiting on
condition [0x6393f000..0x6393fd8c]
at java.lang.Thread.sleep(Native Method)
at HTTPClient.SocketTimeout.run(StreamDemultiplexor.java:917)

-------------------------------------------------------------------------------

Thread ID: scheduler-3
Priority: 5
Hashcode: 419368

"scheduler-3" prio=5 tid=0x596d7130 nid=0xdc0 in Object.wait()
[0x68c7f000..0x68c7fd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:133)
- locked <0x17625818> (a java.lang.Object)
at coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:
128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:67)

-------------------------------------------------------------------------------




On May 4, 9:01 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> 17:05:56.711 1241471156711 66235 jrpp-140 10.50.214.66 POSThttp://server/default.cfm60673 36 504896 182552 209536 26983

[charlie arehart]

Sung, please consider changing the CP to do a notify rather than abort. By
killing the request, when you get this email, it says "the request has been
aborted" so that you don't see it (jrpp-47) in the thread dump, so you can't
tell what it was "hung" on.

I realize you may fear that if you don't kill the requests, they'll hang
around forever. I don't know if that's true, though. Honestly, when the
abort takes place, if FR can't abort it because it's in a native method
call, it won't abort it until it comes out of that native method. You set
the CP to take place when the request exceeds 90 seconds, but note that the

message below says:

"This request ran for approximately 6753887ms, and has been aborted."

So it wasn't until the request ran for 6000+ seconds (1 hour 52 minutes)
that it finally came out of the native method and returned to CF (maybe it
was a long query, maybe it was a CFHTTP call, or a we service invocation).
When it did, that's when FR/the JVM was able to kill the request. But at
that point, I bet it would have run to completion, so you only stopped it
perhaps seconds or milliseconds before it would have finished on its own. In
that case, did the CP abort really "help". I'll argue not.

It's why I'm not a fan of doing a CP abort (as a first step to problem
solving). It *masks* getting good diagnostics that would help you know what
the real problem is.

But if you really don't want to do that, here's another thought: did you get
another email from the CP process that fired at the 90 seconds after the
request started? That would have also had a stack trace (for this jrpp-47
thread, running for the request FR has identified as #5363), and it's
possible that at the point it was already stuck in whatever was holding it
up. Better still, if you got other emails after that (they fire by default
every couple of minutes), then if you compare those emails, do they show it
stuck (for the same FR request number, 5363 on that jrpp-47 thread) at the
same method on each email? If so, that's your smoking gun.

[sjwoo]

Hi Charlie,

I don't see a reason why I would change the CP to do a notify instead
of an abort, because the thread stays around even when I have it set
to abort. Another one today:

---

Hello, this is the FusionReactor Crash Protection agent at

[server].nj.adp.com, instance: coldfusion.cfmx7.[server].

http://[server]:8088/fusionreactor/fhtml.cfm?mode=main

The following alert was generated at Mon May 18 18:44:22 EDT 2009.

[Timeout/Aborted] Crash Protection - Request Run Time Alert: Actual

value: 90657ms, threshold value: 90000ms, current request: 2009-05-18
12:26:28.779 1242663988779 32135 jrpp-84 10.33.1.129 POST http://[server]/page.cfm
90657 15 1038336 156776 1038336 881559 200 0

This request ran for approximately 22673232ms, and has been aborted.

---

The reason why this request has been aborted is because I am going
into FR and killing it hard (not soft kill, but hard kill via thread
kill). I am the one doing this; this request would not free itself
unless I did the hard kill. Right after I do the hard kill, I get
this message, so I am the one generating these emails.

I only get one email message. I've changed the setting to notify, but
like I said, it makes no difference.

- Sung

On May 13, 12:56 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

[charlie arehart]

Sung, I'll still assert that it's not quite "you" that's "generating these
emails", at least in that it's the CP notification that's sending you the
email (according to the email, which is saying it exceeded the run time
alert).

It is odd that it says it caught it running too long at 90000ms, but then
ran for 22673232m. Maybe someone from Intergral can explain why that's so.
Perhaps there's something I'm not understanding about the alert process.

Indeed, if you DO have the CP set to abort, then I'd wonder why THAT didn't
terminate the request but your hard kill did.

Anyway, I would still wonder what you see in the stack trace for this
request when you see it's hanging, before you try to do the kill. Have you
shared that here? You really want to get to the bottom of what it is that's
hanging it up, and that will help.

/charlie

[sjwoo]

Hi Charlie,

My hard killing of the thread is creating these emails. Let me
explain exactly what happens:

1) I go on FR
2) I see a transaction that's taking way too long
3) I attempt a soft kill
4) when soft kill doesn't work, I do a hard thread kill
5) I receive the FR Crash Alert email

I do not receive an email until I hard kill the thread.

I have it set to notify, so with the next one, I'll copy and paste the
stack trace from FR. Thanks!

On May 18, 7:31 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> 12:26:28.779 1242663988779 32135 jrpp-84 10.33.1.129 POSThttp://[server]/page.cfm

> 90657 15 1038336 156776 1038336 881559  200 0
>
> This request ran for approximately 22673232ms, and has been aborted.
>
> ---
>
> The reason why this request has been aborted is because I am going
> into FR and killing it hard (not soft kill, but hard kill via thread
> kill).  I am the one doing this; this request would not free itself
> unless I did the hard kill.  Right after I do the hard kill, I get
> this message, so I am the one generating these emails.
>
> I only get one email message.  I've changed the setting to notify, but
> like I said, it makes no difference.
>

> - Sung- Hide quoted text -
>
> - Show quoted text -

[charlie arehart]

Sung, I understand. I was just saying that your kill doesn't "cause" the
email. That's not a feature of FR, that I know of. Instead, it's the CP that
is sending the email.

Can anyone from Intergral step in here to discuss where this confusion is
coming in? Why would the CP alert fire an email when he does a manual hard
kill? Is it that the CP alert is somehow waiting for the thread to be
finished? I wouldn't have thought so, as I thought CP alert emails were
triggered by the events they monitor (not ever by manual killing of
threads).

Anyway, Sung, the more important point is would you please do a stack trace
BEFORE killing a thread.

I see you saying that you've also got the CP notify on, and that therefore
should also send you a stack trace, but those come at a random point in
time, when some event is detected (too little memory, too many requests, a
request crossing the long-request time). That will also give a stack trace
for all running threads, which is a lot more info.

Since you know of some one request that is running long, and you do now go
in and manually kill it, I'm proposing instead that you simply do the stack
trace on it (click the little magnifying glass to the left of the request in
the running requests page), and let us know what you see to be the state
that the seeming hung thread is in, before you kill it. Better still,
refresh the stack trace and confirm that it's in the same state after the
refresh?

All I'm getting at is that killing the thread doesn't solve the root cause
of the problem. You need to get to the bottom of that, unless you want to
always have to manually kill the thread. I'm trying to help you get to that
root cause. The stack trace (taken before the kill) will help immensely.

BTW, I don't mind at all elaborating on this discussion, as many others also
will benefit from learning (or being reminded) how to use these techniques
to solve such problems. Just sharing my experience as a fellow user.

/charlie

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of sjwoo
Sent: Tuesday, May 19, 2009 9:39 AM
To: FusionReactor
Subject: FusionReactor Group: Re: Force Killing of Long-Running Requests

Hi Charlie,

My hard killing of the thread is creating these emails. Let me

[sjwoo]

Hi Charlie,

Since I changed the option from abort and notify to just notify, I've
had less issues. In fact, since my last message, I haven't had any --
until today. So before I killed the thread, I copied the stack
trace. Here it is -- let me know what you think.

Thread Stack TraceTrace Time: 16:00:41.740 03-Jun-2009
Request ID: 59979
Script Name: http://server/page.cfm
Started: 15:58:18.892 03-Jun-2009
Exec Time: 142848ms
Memory Used: (24%)255,843KB
Memory Free: 781,596KB
Thread ID: jrpp-147
Priority: 5
Hashcode: 30728566
"jrpp-147" prio=5 tid=0x5d547410 nid=0xad4 runnable
[0x04acf000..0x04acfd8c]

at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at jrun.servlet.jrpp.JrppOutputStream$SpillStream.write
(JrppOutputStream.java:182)
at jrun.servlet.io.MetricsOutputStream.write(MetricsOutputStream.java:
75)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:
66)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)

- locked <0x1f4208f8> (a jrun.servlet.jrpp.JrppBufferedOutputStream)

at jrun.servlet.jrpp.JrppOutputStream.flush(JrppOutputStream.java:45)
at com.intergral.fusionreactor.filter.A.E.flush(Unknown Source)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:
410)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)

- locked <0x20a27978> (a java.io.OutputStreamWriter)

at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
at java.io.PrintWriter.flush(PrintWriter.java:120)

- locked <0x20a27978> (a java.io.OutputStreamWriter)

On May 19, 12:50 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:

> stack trace from FR.  Thanks!- Hide quoted text -

[charlie arehart]

Ok, well that shows that you are in the socketwrite0 state, ultimately going
through the JrppOutputStream class, and all that is the crux of a fix that
was made in 3.01, where there's an optional (experimental) feature that you
can enable to try to have FR stop these particular hung threads, which are
due to a bug in IE (yes, in the browser). The threads can't be stopped by
FR's kill, which is why it wasn't solving the problem.

So what release of FR are you running? If still 2 (or a point release like
2.04), you will need to buy an upgrade to 3.01 (not expensive, starting at
199 for Standard, and with lots of other beneficial improvements in both 3
and 3.01, as discussed on the site). If you're on 3.0, you can update to
3.01 free.

Info at the FR site on the update is at:

http://www.fusion-reactor.com/fr/whatsnew.cfm (what was new in 3)
http://www.fusion-reactor.com/support/kb/FRS-214.cfm (what was new in 3.01,
but just bullet points)
http://www.fusion-reactor.com/fr/upgrade.cfm (upgrade pricing from 2)
http://www.fusion-reactor.com/fr/featurefocus/ (more on things new in both 3
and 3.01)

I've also blogged in the past about the updates, also:

http://www.carehart.org/blog/client/index.cfm/2008/1/21/fusionreactor3_relea
sed
http://www.carehart.org/blog/client/index.cfm/2008/12/7/fusionreactor301_rel
eased_some_gems

As for applying the new feature, it's not documented in the release notes or
anything. Fortunately, it was discussed by Darren (of Intergral) in reply to
a thread here late last year. Here's the specific message in the thread with
the steps to follow:

http://groups.google.com/group/fusionreactor/msg/2e57c5fbab5305b1

Thanks, too, for sharing the news that changing the "abort and notify" to
just "notify" helped. I've always suspected that doing the abort was
potentially more trouble than it was worth, and that more often it's more
important to get to the bottom of the problem, rather than have FR spend
cycles trying to abort. Plus, there are simply some threads that FR cannot
abort. Maybe one of the engineers can speak up on whether such issues can
lead to more problems (in attempting an auto kill) than may be worth, and
whether there are either guidelines we can follow, or possible improvements
that could be considered in FR.

Anyway, let us know how things go with the above.

/charlie

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of sjwoo
Sent: Wednesday, June 03, 2009 4:06 PM
To: FusionReactor
Subject: FusionReactor Group: Re: Force Killing of Long-Running Requests

Hi Charlie,

Since I changed the option from abort and notify to just notify, I've

had less issues. In fact, since my last message, I haven't had any --
until today. So before I killed the thread, I copied the stack
trace. Here it is -- let me know what you think.

Thread Stack TraceTrace Time: 16:00:41.740 03-Jun-2009
Request ID: 59979
Script Name: http://server/page.cfm
Started: 15:58:18.892 03-Jun-2009
Exec Time: 142848ms
Memory Used: (24%)255,843KB
Memory Free: 781,596KB
Thread ID: jrpp-147
Priority: 5
Hashcode: 30728566
"jrpp-147" prio=5 tid=0x5d547410 nid=0xad4 runnable
[0x04acf000..0x04acfd8c]
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at jrun.servlet.jrpp.JrppOutputStream$SpillStream.write
(JrppOutputStream.java:182)

<snip>

[charlie arehart]

Sjwoo, did you ever resolve things (on this issue from a couple months ago)?
Anything to report to us, who were following the discussion here on the
list?

/charlie


> -----Original Message-----
> From: fusion...@googlegroups.com
> [mailto:fusion...@googlegroups.com] On Behalf Of charlie arehart
> Sent: Wednesday, June 03, 2009 5:57 PM
> To: fusion...@googlegroups.com
> Subject: FusionReactor Group: Re: Force Killing of Long-Running
> Requests
>
>

[Snake]

Hi,

We have had a few occurrences where CF is not responding and all sites are
down, but FR shows everything is fine on the server, although no requests
etc.
This server is running FR 2.0.4 and CF7, is this a known issue that would be
resolved by upgrading to FR 3?

Russ

[charlie arehart]

Hey Russ, an upgrade could be helpful for some reasons (more in a moment),
but here are some thoughts to consider first. I don't want to presume
anything, and maybe this info will also help others in similar situations in
the future.

You say there are "no requests". Do you mean just that none are showing up
in "running requests"?

Or do you also mean that if you look at "request history" there are none
that have run recently? It could be that some are getting in, which are not
seen in the running request displays as they run fast. That's worth
double-checking and would help narrow things down a bit.

And as you may know (again, I'm pointing this out for all here), the request
history shows the most recent finished requests at the top, and you can see
both the time they ended and a request id, which increments for each new
request. That id can be a handy way to eyeball things quickly: if you
refresh the request history, is that number incrementing?

If the request history shows some requests running, again that helps show
that it's not completely dead. Or they may show only coming in on other than
the external web server. If there are no requests at all that show having
come in for several minutes, that could be a still different problem. (As
another test, can you get in on the CF Admin? If that's also using CF's
available built-in web server, that too would be a different set of
threads.)

Rather than speculate on which it may be for you, can you tell us what you
see?

Also worth noting: if the sites that you (and customers) are visiting are
requested via (say) IIS, but you're opening the FR admin using FR's built-in
web server (via a port, like 8088), then it could suggest that CF's fine but
the problem is with your web server, or the connector between CF and the
webserver. It could also be a threading problem affecting only the external
web server (jrpp) threads.

I'd just double-check these things before concluding that an update of FR is
the answer, but there are certainly great benefits to the upgrade: a better
interface (with a really nice system metrics page), better logs, and more.

/charlie

> -----Original Message-----
> From: fusion...@googlegroups.com
> [mailto:fusion...@googlegroups.com] On Behalf Of Snake
> Sent: Friday, August 07, 2009 9:20 AM
> To: fusion...@googlegroups.com

[Snake]

The point of setting requests to abort is so that your server doesn't get
hung up with stuck or long running requests. If you set to notify only, and
problems occur when you are not able to deal with it, the hung requests will
queue up until you have used up all available requests, and then everything
will be queued indefinitely and CF will no longer serve up any pages.

It would be better to use the notify option only at times when you will be
available to investigate the hung requests and manually kill them if
required or if you are sure that they do not happen often enough or long
enough to use up all your requests before you notice.
I.e. the requests are long running but do eventually end, and you will still
have enough requests in the mean time to deal with the rest of the traffic.
Or they are requests which will not abort and have to be manually killed
anyway.

Russ

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of sjwoo
Sent: 18 May 2009 23:56
To: FusionReactor
Subject: FusionReactor Group: Re: Force Killing of Long-Running Requests

[Snake]

Actually perhaps this would make a good feature request for intergral, a
SCHEDULER for how the crash protection works, so we can set it to notify at
certain times and abort at others.

[charlie arehart]

Hey Russ, I just want to clarify for readers that your observation of when
to use/not use the "notify" feature of CP here is in reply to a note from
sjwoo back in May. If one reads more of the emails in that thread (again at
http://groups.google.com/group/fusionreactor/browse_thread/thread/b2f4377abb
71efdf/4e33098b53c39c98 ), they may come to a different conclusion. :-)

Indeed, I only press this because SJWoo wrote back a few messages later in
that thread (after accepting my explanation of why the notify feature may be
better in many cases) and he reported that "since I changed the option from

abort and notify to just notify, I've had less issues. In fact, since my

last message, I haven't had any." That's pretty compelling evidence to
counter what you've said.

Now, I do realize that what you're saying seems logical (use the abort
feature "so that your server doesn't get hung up with stuck or long running
request" and "use the notify option only at times when you will be available
to investigate the hung requests...") but as you note many requests can't be
terminated and the feature doesn't help then. In fact, it may cause the very
instability that is referred to in the docs and on screen about the abort
feature. More than that, I just find that having CF/FR abort requests is
potentially masking the root cause of problems.

These are just preferences, of course. You stated yours, and I wanted to
reiterate mine since it came up later in the thread. More importantly, sj
offered the evidence that changing to not use the abort feature really
helped his server a lot. It's just not clear if you saw that, and if others
may have missed that thread, I don't think the point should be lost. Just
trying to help.

/charlie

[Snake]

Sorry I didn't notice the date.
The situation may of course be different on a dedicated server than a shared
hosting server, where you have to consider hundreds of different sites, a
lot of which will simply have poorly written code that takes a long time to
run and will tie up threads. We don't get too many that cannot be killed
automatically, but we often have a pile-up of long running requests, usually
from the same site, which would use up all available requests if left
running, which results in all sites stop responding, and as you can imagine
the phone starts ringing pretty quick when customers notice.

[charlie arehart]

Ah, interesting observation. Good point about how things could be different
in a shared vs dedicated environment. Thanks very much for pointing that
out.

/charlie

> -----Original Message-----
> From: fusion...@googlegroups.com
> [mailto:fusion...@googlegroups.com] On Behalf Of Snake
> Sent: Friday, August 21, 2009 7:34 AM
> To: fusion...@googlegroups.com
> Subject: FusionReactor Group: Re: Force Killing of Long-Running
> Requests
>
>