most threads were in TIMED_WAITING

898 views

Skip to first unread message

Chris

unread,

May 28, 2014, 6:02:32 PM5/28/14

to fusion...@googlegroups.com

Hi, I hope I have enough figured out to ask a smart question:

Most threads were in TIMED_WAITING when I did a Stack Trace while a server had maximum threads and was becoming unresponsive.

The TIMED_WAITING threads were all waiting on the same thing, and all for a Native Method. I'd figure that was a datasource call but our Database folks say the databases were being very lightly used, with no issues. Could it be ColdFusion preparing queries to send to the database?

The pattern of a server increasing to max. threads, going unresponsive, then suddenly everything is back to normal seems like TIMED_WAITING threads are timing out.

Any thoughts?

thank you,

Chris

Here is a summary of the stack trace:

no. of threads:
13    TIMED_WAITING
7    RUNNABLE
3    BLOCKED

"jrpp-2183" Id=24882 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@edffa
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@edffa
    at java.lang.Object.wait(Object.java:443)

"jrpp-2193" Id=24970 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@19734f6
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@19734f6
    at java.lang.Object.wait(Object.java:443)

"jrpp-2199" Id=24977 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@17bfed4
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@17bfed4
    at java.lang.Object.wait(Object.java:443)

"jrpp-2201" Id=24991 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@15fc50c
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@15fc50c
    at java.lang.Object.wait(Object.java:443)

"jrpp-2200" Id=24990 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@81bc76
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@81bc76
    at java.lang.Object.wait(Object.java:443)

"jrpp-2190" Id=24941 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@e104a1
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@e104a1
    at java.lang.Object.wait(Object.java:443)

"jrpp-2202" Id=24992 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@9b897e
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@9b897e
    at java.lang.Object.wait(Object.java:443)

"jrpp-2209" Id=25000 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@198ad54
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@198ad54
    at java.lang.Object.wait(Object.java:443)

"jrpp-2208" Id=24999 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@9c37f7
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@9c37f7
    at java.lang.Object.wait(Object.java:443)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2197" Id=24974 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@11a2198
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@11a2198
    at java.lang.Object.wait(Object.java:443)

"jrpp-2206" Id=24996 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@1e5316
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@1e5316
    at java.lang.Object.wait(Object.java:443)

"jrpp-2203" Id=24993 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@1d42e08
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@1d42e08
    at java.lang.Object.wait(Object.java:443)

"jrpp-2204" Id=24994 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@e0e101
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@e0e101
    at java.lang.Object.wait(Object.java:443)

"jrpp-2210" Id=25001 RUNNABLE (in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)

"jrpp-2185" Id=24926 BLOCKED on java.security.SecureRandom@7c485d owned by "jrpp-2181" Id=24880
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - blocked on java.security.SecureRandom@7c485d
    at java.security.SecureRandom.next(SecureRandom.java:455)

"jrpp-2207" Id=24997 RUNNABLE
    at com.rsa.cryptoj.c.pV.clone(Unknown Source)

"jrpp-2205" Id=24995 BLOCKED on java.security.SecureRandom@7c485d owned by "jrpp-2207" Id=24997
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - blocked on java.security.SecureRandom@7c485d
    at java.security.SecureRandom.next(SecureRandom.java:455)

"jrpp-2198" Id=24976 RUNNABLE (in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)

"jrpp-2189" Id=24940 RUNNABLE (in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)

"jrpp-2167" Id=24755 RUNNABLE (in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)

"jrpp-2181" Id=24880 BLOCKED on java.security.SecureRandom@7c485d owned by "jrpp-2207" Id=24997
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - blocked on java.security.SecureRandom@7c485d
    at java.security.SecureRandom.next(SecureRandom.java:455)

"jrpp-2184" Id=24889 RUNNABLE
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:171)

"jrpp-2194" Id=24971 RUNNABLE (in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)

Charlie Arehart

unread,

May 29, 2014, 1:10:05 AM5/29/14

to fusion...@googlegroups.com

Chris, you seem to have shown only the first couple of lines of each stack trace. Was it that that was all there was? (I’d think not, but if so, where did you get them from, in FR?)

If there was more, please do share it. You often need to see down 10-15 lines of a stack trace to really see what “line of code” (if any) that the stack trace indicates the request was running.

Also, is this in fact the stack traces from running requests (shown in the request activity page)? Or was it something from the resources>thread dump? That’s not as useful, as it’s ALL threads in the JVM, not only those really running requests.

Hope that helps.

/charlie

--
You received this message because you are subscribed to the Google Groups "FusionReactor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fusionreacto...@googlegroups.com.
To post to this group, send email to fusion...@googlegroups.com.
Visit this group at http://groups.google.com/group/fusionreactor.
For more options, visit https://groups.google.com/d/optout.

Chris

unread,

May 29, 2014, 8:25:16 AM5/29/14

to fusion...@googlegroups.com

Hi Charlie,

These are from FR, using Stack Trace All from the Requests > Activity page (while the server reached max threads, the memory usage flattened out (ineffective GC), and the server began being unresponsive).

Below is the Plain output that I excerpted from in my previous email. (I didn't want to dump a lot of stuff to the group).

TIMED_WAITING - the threads all look the same. I don't see any application files in the details.

RUNNABLE threads show our application .cfc and .cfm files. Most of these apps are known heavy users of our servers. Also, is it correct to say that these TIMED_WAITING threads were spawned from events in these RUNNABLE threads?

BLOCKED threads -- two are waiting on a RUNNABLE thread (jrpp-2207), and the third BLOCKED thread is waiting on one of the other BLOCKED threads ... so I'm thinking these are not the issues we're looking for.

Thanks for the help!

Chris

"jrpp-2183" Id=24882 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@edffa
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@edffa
    at java.lang.Object.wait(Object.java:443)

    at edu.emory.mathcs.backport.java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:301)
    at edu.emory.mathcs.backport.java.util.concurrent.helpers.WaitQueue$WaitNode.doTimedWait(WaitQueue.java:79)
    at edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync.attempt(Semaphore.java:346)
    at edu.emory.mathcs.backport.java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:571)
    at coldfusion.CfmServlet.lock(CfmServlet.java:281)
    at coldfusion.CfmServlet.service(CfmServlet.java:200)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2193" Id=24970 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@19734f6
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@19734f6
    at java.lang.Object.wait(Object.java:443)

"jrpp-2199" Id=24977 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@17bfed4
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@17bfed4
    at java.lang.Object.wait(Object.java:443)

"jrpp-2201" Id=24991 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@15fc50c
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@15fc50c
    at java.lang.Object.wait(Object.java:443)

"jrpp-2200" Id=24990 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@81bc76
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@81bc76
    at java.lang.Object.wait(Object.java:443)

"jrpp-2190" Id=24941 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@e104a1
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@e104a1
    at java.lang.Object.wait(Object.java:443)

"jrpp-2202" Id=24992 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@9b897e
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@9b897e
    at java.lang.Object.wait(Object.java:443)

"jrpp-2209" Id=25000 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@198ad54
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@198ad54
    at java.lang.Object.wait(Object.java:443)

"jrpp-2208" Id=24999 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@9c37f7
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@9c37f7
    at java.lang.Object.wait(Object.java:443)

    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2197" Id=24974 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@11a2198
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@11a2198
    at java.lang.Object.wait(Object.java:443)

"jrpp-2206" Id=24996 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@1e5316
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@1e5316
    at java.lang.Object.wait(Object.java:443)

"jrpp-2203" Id=24993 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@1d42e08
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@1d42e08
    at java.lang.Object.wait(Object.java:443)

"jrpp-2204" Id=24994 TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@e0e101
    at java.lang.Object.wait(Native Method)
    - waiting on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node@e0e101
    at java.lang.Object.wait(Object.java:443)

"jrpp-2210" Id=25001 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)

    at macromedia.jdbc.sqlserver.dda.w(Unknown Source)
    at macromedia.jdbc.sqlserver.dda.o(Unknown Source)
    at macromedia.jdbc.sqlserver.tds.ddq.a(Unknown Source)
    at macromedia.jdbc.sqlserver.tds.ddo.e(Unknown Source)
    at macromedia.jdbc.sqlserver.tds.ddn.a(Unknown Source)
    at macromedia.jdbc.sqlserver.ddj.h(Unknown Source)
    at macromedia.jdbc.sqlserverbase.ddde.v(Unknown Source)
    at macromedia.jdbc.sqlserverbase.ddde.r(Unknown Source)
    at macromedia.jdbc.sqlserverbase.ddb9.execute(Unknown Source)
    at macromedia.jdbc.sqlserverbase.ddcc.execute(Unknown Source)
    at com.intergral.fusionreactor.jdbc.PreparedStatementSurrogate.execute(PreparedStatementSurrogate.java:189)
    at coldfusion.server.j2ee.sql.JRunPreparedStatement.execute(JRunPreparedStatement.java:91)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1436)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1183)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1114)
    at coldfusion.sql.SqlImpl.execute(SqlImpl.java:341)
    at coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:915)
    at coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:590)
    at cfqry_DrugSearch2ecfm149264730.runPage(F:\Pool\Content\scripts\cder\drugsatfda\qry_DrugSearch.cfm:186)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
    at cffbx_Switch2ecfm1848778397._factor2(F:\Pool\Content\scripts\cder\drugsatfda\fbx_Switch.cfm:131)
    at cffbx_Switch2ecfm1848778397._factor104(F:\Pool\Content\scripts\cder\drugsatfda\fbx_Switch.cfm:97)
    at cffbx_Switch2ecfm1848778397._factor105(F:\Pool\Content\scripts\cder\drugsatfda\fbx_Switch.cfm:47)
    at cffbx_Switch2ecfm1848778397.runPage(F:\Pool\Content\scripts\cder\drugsatfda\fbx_Switch.cfm:1)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
    at cffbx_fusebox30_CF50_ErkiEsken2ecfm619765951._factor3(F:\Pool\Content\scripts\cder\drugsatfda\fbx_fusebox30_CF50_ErkiEsken.cfm:249)
    at cffbx_fusebox30_CF50_ErkiEsken2ecfm619765951.runPage(F:\Pool\Content\scripts\cder\drugsatfda\fbx_fusebox30_CF50_ErkiEsken.cfm:1)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
    at cfindex2ecfm433156480.runPage(F:\Pool\Content\scripts\cder\drugsatfda\index.cfm:1)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:399)
    at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126)
    at coldfusion.CfmServlet.service(CfmServlet.java:201)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2185" Id=24926 BLOCKED on java.security.SecureRandom@7c485d owned by "jrpp-2181" Id=24880
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - blocked on java.security.SecureRandom@7c485d
    at java.security.SecureRandom.next(SecureRandom.java:455)

    at java.util.Random.nextInt(Random.java:253)
    at coldfusion.util.UUIDUtils.createUUID(UUIDUtils.java:116)
    at coldfusion.runtime.ClientScopeServiceImpl.GetCFTOKEN(ClientScopeServiceImpl.java:850)
    at coldfusion.runtime.AppHelper.getClientIDs(AppHelper.java:273)
    at coldfusion.runtime.AppHelper.ValidateClientAndSession(AppHelper.java:902)
    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:309)
    at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion.CfmServlet.service(CfmServlet.java:201)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2207" Id=24997 RUNNABLE
at com.rsa.cryptoj.c.pV.clone(Unknown Source)

    at com.rsa.cryptoj.c.oV.a(Unknown Source)
    at com.rsa.cryptoj.c.pV.a(Unknown Source)
    at com.rsa.cryptoj.c.oy.a(Unknown Source)
    at com.rsa.cryptoj.c.oy.c(Unknown Source)
    at com.rsa.cryptoj.c.oy.a(Unknown Source)
    at com.rsa.cryptoj.c.fA.c(Unknown Source)
    at com.rsa.cryptoj.c.oy.c(Unknown Source)
    at com.rsa.cryptoj.c.d.engineNextBytes(Unknown Source)
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    at java.security.SecureRandom.next(SecureRandom.java:455)
    at java.util.Random.nextInt(Random.java:253)
    at coldfusion.util.UUIDUtils.createUUID(UUIDUtils.java:116)
    at coldfusion.runtime.ClientScopeServiceImpl.GetCFTOKEN(ClientScopeServiceImpl.java:850)
    at coldfusion.runtime.AppHelper.getClientIDs(AppHelper.java:273)
    at coldfusion.runtime.AppHelper.ValidateClientAndSession(AppHelper.java:902)
    at coldfusion.tagext.lang.ApplicationTag.doStartTag(ApplicationTag.java:308)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
    at cfApplication2ecfm154106626.runPage(F:\Pool\Content\scripts\cder\drugsatfda\Application.cfm:31)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
    at coldfusion.filter.CfincludeFilter.include(CfincludeFilter.java:33)
    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:297)
    at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion.CfmServlet.service(CfmServlet.java:201)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2205" Id=24995 BLOCKED on java.security.SecureRandom@7c485d owned by "jrpp-2207" Id=24997
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - blocked on java.security.SecureRandom@7c485d
    at java.security.SecureRandom.next(SecureRandom.java:455)

    at java.util.Random.nextInt(Random.java:253)
    at coldfusion.util.UUIDUtils.createUUID(UUIDUtils.java:116)
    at coldfusion.runtime.ClientScopeServiceImpl.GetCFTOKEN(ClientScopeServiceImpl.java:850)
    at coldfusion.runtime.AppHelper.getClientIDs(AppHelper.java:273)
    at coldfusion.runtime.AppHelper.ValidateClientAndSession(AppHelper.java:902)
    at coldfusion.tagext.lang.ApplicationTag.doStartTag(ApplicationTag.java:308)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
    at cfApplication2ecfm154106626.runPage(F:\Pool\Content\scripts\cder\drugsatfda\Application.cfm:31)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
    at coldfusion.filter.CfincludeFilter.include(CfincludeFilter.java:33)
    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:297)
    at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126)
    at coldfusion.CfmServlet.service(CfmServlet.java:201)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2198" Id=24976 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)

    at macromedia.jdbc.sqlserver.dda.w(Unknown Source)
    at macromedia.jdbc.sqlserver.dda.o(Unknown Source)
    at macromedia.jdbc.sqlserver.tds.ddq.a(Unknown Source)
    at macromedia.jdbc.sqlserver.tds.ddo.e(Unknown Source)
    at macromedia.jdbc.sqlserver.tds.ddn.a(Unknown Source)
    at macromedia.jdbc.sqlserver.ddj.h(Unknown Source)
    at macromedia.jdbc.sqlserverbase.ddde.v(Unknown Source)
    at macromedia.jdbc.sqlserverbase.ddde.r(Unknown Source)
    at macromedia.jdbc.sqlserverbase.ddb9.execute(Unknown Source)
    at macromedia.jdbc.sqlserverbase.ddcc.execute(Unknown Source)
    at com.intergral.fusionreactor.jdbc.PreparedStatementSurrogate.execute(PreparedStatementSurrogate.java:189)
    at coldfusion.server.j2ee.sql.JRunPreparedStatement.execute(JRunPreparedStatement.java:91)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1436)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1183)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1114)
    at coldfusion.sql.SqlImpl.execute(SqlImpl.java:341)
    at coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:915)
    at coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:590)
    at cfcdrhnew2drss2ecfm613580387.runPage(F:\Pool\Content\scripts\cdrh\cfdocs\cfTopic\cdrhnew-rss.cfm:30)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:399)
    at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion.CfmServlet.service(CfmServlet.java:201)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2189" Id=24940 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)

    at macromedia.jdbc.oracle.net8.ddb.b(Unknown Source)
    at macromedia.jdbc.oracle.net8.ddb.w(Unknown Source)
    at macromedia.jdbc.oracle.net8.ddb.u(Unknown Source)
    at macromedia.jdbc.oracle.net8.ddb.f(Unknown Source)
    at macromedia.jdbc.oracle.net8.ddf.b(Unknown Source)
    at macromedia.jdbc.oracle.ddk.b(Unknown Source)
    at macromedia.jdbc.oracle.ddk.a(Unknown Source)
    at macromedia.jdbc.oracle.ddk.p(Unknown Source)
    at macromedia.jdbc.oraclebase.dddf.v(Unknown Source)
    at macromedia.jdbc.oraclebase.dddf.r(Unknown Source)
    at macromedia.jdbc.oraclebase.dddf.execute(Unknown Source)
    at com.intergral.fusionreactor.jdbc.StatementSurrogate.execute(StatementSurrogate.java:63)
    at coldfusion.server.j2ee.sql.JRunStatement.execute(JRunStatement.java:348)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1420)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1183)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1114)
    at coldfusion.sql.SqlImpl.execute(SqlImpl.java:341)
    at coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:915)
    at coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:590)
    at cfenforce_rpt2ecfc2061839150$funcINIT.runFunction(F:\Pool\Content\scripts\Enforcement\enforce_rpt.cfc:28)
    at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472)
    at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)
    at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:368)
    at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:55)
    at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321)
    at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:220)
    at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:491)
    at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:337)
    at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2360)
    at cfERHeader2ecfm1484869519.runPage(F:\Pool\Content\scripts\Enforcement\ERHeader.cfm:3)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
    at cfenforce_rpt2dProduct2dTabs2ecfm38249561.runPage(F:\Pool\Content\scripts\Enforcement\enforce_rpt-Product-Tabs.cfm:2)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:399)
    at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion.CfmServlet.service(CfmServlet.java:201)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2167" Id=24755 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)

"jrpp-2181" Id=24880 BLOCKED on java.security.SecureRandom@7c485d owned by "jrpp-2207" Id=24997
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - blocked on java.security.SecureRandom@7c485d
    at java.security.SecureRandom.next(SecureRandom.java:455)

    at java.util.Random.nextInt(Random.java:253)
    at coldfusion.util.UUIDUtils.createUUID(UUIDUtils.java:123)
    at coldfusion.runtime.ClientScopeServiceImpl.GetCFTOKEN(ClientScopeServiceImpl.java:850)
    at coldfusion.runtime.AppHelper.getClientIDs(AppHelper.java:273)
    at coldfusion.runtime.AppHelper.ValidateClientAndSession(AppHelper.java:902)
    at coldfusion.tagext.lang.ApplicationTag.doStartTag(ApplicationTag.java:308)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
    at cfApplication2ecfm154106626.runPage(F:\Pool\Content\scripts\cder\drugsatfda\Application.cfm:31)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
    at coldfusion.filter.CfincludeFilter.include(CfincludeFilter.java:33)
    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:297)
    at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126)
    at coldfusion.CfmServlet.service(CfmServlet.java:201)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2184" Id=24889 RUNNABLE
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:171)

    at macromedia.jdbc.oraclebase.ddt.a(Unknown Source)
    at macromedia.jdbc.oraclebase.ddbh.a(Unknown Source)
    at macromedia.jdbc.oraclebase.ddch.getMetaData(Unknown Source)
    at com.intergral.fusionreactor.jdbc.ResultSetSurrogate.getMetaData(ResultSetSurrogate.java:549)
    at coldfusion.sql.QueryTable.populate(QueryTable.java:370)
    at coldfusion.sql.QueryTable.populate(QueryTable.java:245)
    at coldfusion.sql.Executive.getRowSet(Executive.java:641)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1439)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1183)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1114)
    at coldfusion.sql.SqlImpl.execute(SqlImpl.java:341)
    at coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:915)
    at coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:590)
    at cfenforce_rpt2ecfc2061839150$funcINIT.runFunction(F:\Pool\Content\scripts\Enforcement\enforce_rpt.cfc:28)
    at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472)
    at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)
    at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:368)
    at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:55)
    at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321)
    at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:220)
    at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:491)
    at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:337)
    at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2360)
    at cfERHeader2ecfm1484869519.runPage(F:\Pool\Content\scripts\Enforcement\ERHeader.cfm:3)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2722)
    at cfenforce_rpt2dProduct2dTabs2ecfm38249561.runPage(F:\Pool\Content\scripts\Enforcement\enforce_rpt-Product-Tabs.cfm:2)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:399)
    at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion.CfmServlet.service(CfmServlet.java:201)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"jrpp-2194" Id=24971 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)

    at macromedia.jdbc.oracle.net8.ddb.b(Unknown Source)
    at macromedia.jdbc.oracle.net8.ddb.w(Unknown Source)
    at macromedia.jdbc.oracle.net8.ddb.u(Unknown Source)
    at macromedia.jdbc.oracle.net8.ddb.f(Unknown Source)
    at macromedia.jdbc.oracle.net8.ddf.b(Unknown Source)
    at macromedia.jdbc.oracle.ddk.b(Unknown Source)
    at macromedia.jdbc.oracle.ddk.a(Unknown Source)
    at macromedia.jdbc.oracle.ddk.p(Unknown Source)
    at macromedia.jdbc.oraclebase.dddf.v(Unknown Source)
    at macromedia.jdbc.oraclebase.dddf.r(Unknown Source)
    at macromedia.jdbc.oraclebase.ddb_.execute(Unknown Source)
    at com.intergral.fusionreactor.jdbc.PreparedStatementSurrogate.execute(PreparedStatementSurrogate.java:189)
    at coldfusion.server.j2ee.sql.JRunPreparedStatement.execute(JRunPreparedStatement.java:91)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1436)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1183)
    at coldfusion.sql.Executive.executeQuery(Executive.java:1114)
    at coldfusion.sql.SqlImpl.execute(SqlImpl.java:341)
    at coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:915)
    at coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:590)
    at cfwlDataAccess2ecfc2097304160$funcGETSEARCHRESULTSRECCOUNT.runFunction(F:\Pool\Content\scripts\WarningLetters\wlDataAccess.cfc:74)
    at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472)
    at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:405)
    at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:368)
    at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:55)
    at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321)
    at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:517)
    at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:496)
    at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:355)
    at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2301)
    at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:382)
    at cfwlSearchResult2ecfm1238252515._factor1(F:\Pool\Content\scripts\WarningLetters\wlSearchResult.cfm:80)
    at cfwlSearchResult2ecfm1238252515._factor5(F:\Pool\Content\scripts\WarningLetters\wlSearchResult.cfm:70)
    at cfwlSearchResult2ecfm1238252515.runPage(F:\Pool\Content\scripts\WarningLetters\wlSearchResult.cfm:1)
    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
    at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:399)
    at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion.CfmServlet.service(CfmServlet.java:201)
    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
    at sun.reflect.GeneratedMethodAccessor1333.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:96)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:437)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:311)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:472)
    at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
    at sun.reflect.GeneratedMethodAccessor1329.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:78)
    at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
    at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
    at jrun.servlet.FilterChain.doFilter(FilterChain.java)
    at jrun.servlet.FilterChain.service(FilterChain.java:101)
    at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
    at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
    at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

David Stockton

unread,

May 29, 2014, 9:11:18 AM5/29/14

to fusion...@googlegroups.com

Hi Chris,

A couple of notes about your stack....

The "TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node" threads are the requests that CF has QUEUED because you've hit your "max simultaneous request count" limit.

The "RUNNABLE at .... socketRead0" are all waiting on information back from the DB server - not necessarily a bad thing, depends if a) they were slow queries b) they are expected to be quick, etc

The "BLOCKED on java.security.SecureRandom" are threads waiting on the synchronized UUID creation. I'd imagine you've checked the "Use UUID for CFTOKEN" box in the CFadmin - this is recommended from a security perspective (because it makes it less guessable) but is bad for performance because it requires synchronization and also takes up server CPU resources and perhaps more importantly, entropy. I see from the file paths you're on a Windows server - from memory Windows uses file activity to provide entropy/seeding for random numbers; it could also just be CPU starved.

The "RUNNABLE at com.rsa.cryptoj.c.pV.clone" is the thread that the BLOCKED ones are waiting on for it to create a new UUID (see notes above).

A few hints...

- You could increase the max simultaneous request count - if the new requests coming in don't require a new session, they could process without being blocked.

- Check you're not getting too many bots hitting you and creating sessions that aren't necessary - you could prevent a session being created based on user agent or other similar strategies to minimize needless creation of new sessions - Charlie and others have great blog posts on this.

- A bit more in-depth but it could simply be that you're getting a lot of new user traffic and you don't have the CPU/entropy resources to handle it... there are hardware devices that can provide random numbers/entropy.

Best regards,

David Stockton

Charlie Arehart

unread,

May 29, 2014, 10:25:37 AM5/29/14

to fusion...@googlegroups.com

Let me add to David’s great reply by adding a more generic point on all this: it’s not generally as valuable to do that “strack trace all” on that page if you want to know why certain threads are slow.

Instead, you’d generally be better off stack tracing them individually, so you can know what stack trace goes with what request.

With the below, you (and we) have no idea which requests are slow or may be fast—remember: a stack trace if just a point in time display of what the JVM says a given thread is doing at that moment. It conveys no info on how long it had been doing that one thing, let alone the duration of the request itself.

So from the FR running requests interface, you can easily see the duration of each request, and even sort them by longest-running, and then then pick a slow one to stack trace it, and see what it says it’s doing (and refresh it, to see if it’s still doing that seconds later). Then you can compare other long-running requests. That is more effective than trying to make sense of just a thread dump of all running request threads (or worse, the dump of all threads in the resource>all threads page, though there are indeed times when you may want to use that to see what threads OTHER than those for CF requests are doing, including cfthread threads.)

Finally, if one gets the FR CP alert email, that gives the dual benefit of a) providing this stack trace of request threads by email (when you’re not looking at FR, and when there’s some problem state like too many requests running) and b) it lists at the top of the email the running requests and their duration, so you can easily do the same sort of process above (find a long running request, find its request id on the left of the running request list, go to the request details at the bottom of the email to find that request, get its thread id, and find its one stack trace in the email.)

Of course, again, it’s far easier just to stack trace a single long-running request in the FR interface, if you can be (or get) online when there is one to investigate.

Hope that helps (you or others reading along).

/charlie

Chris

unread,

May 30, 2014, 4:24:51 PM5/30/14

to fusion...@googlegroups.com

Wow! Thank you David and Charlie. That really helps.

Yes, our CF "simultaneous template requests" (.cfm files) was at the default 10. We did not realize that was the effective limit, not the 25 "running JRun threads" we have set.

David,

If CF hits the max. limit for other requests (cfc, Web Service, Flash Remoting), does it give this same response
"TIMED_WAITING on edu.emory.mathcs.backport.java.util.concurrent.Semaphore$FairSync$Node" ?

We have enabled "Use UUID for cftoken " for security. If we enable "Use J2EE session variables" will that enable JSESSIONID and is that faster than UUID?

We are deploying code to kill sessions caused by blacklisted crawlers. Thanks for the reminder!

Also, can you point me to any online resources for items such as the cause of the TIMED_WAITING (above)?

many thanks,
Chris

David - we have support with our FusionReactor licenses. Let me know if you'd like to treat this as billable time :-)