[framework-one] Application Timeout

[Dan Vega]

I am seeing some really odd behavior from an application and I just
can not seem to hunt down the culprit. I have this application that
works great once it gets going. Its just the strangest thing ever
though. When I first hit the url in the morning it takes 2 minutes and
times out. I can hit it another 4 times and it does the same thing. I
take a look at the logs and this is what I see.

ColdFusion9\runtime\logs\coldfusion-out.log
05/05 08:43:56 Error [jrpp-712] - The request has exceeded the
allowable time limit Tag: cfoutput The specific sequence of files
included or processed is: D:\websites\sterisdealer\index.cfm, line:
1067

ColdFusion9\logs\exception.log
"Error","jrpp-712","05/05/10","08:46:42","myapp","The request has
exceeded the allowable time limit Tag: cfoutput The specific sequence
of files included or processed is: D:\web\\myapp\index.cfm, line: 1067
"
coldfusion.runtime.RequestTimedOutException: The request has exceeded
the allowable time limit Tag: cfoutput
at coldfusion.tagext.io.OutputTag.doStartTag(OutputTag.java:149)
at cfdump2ecfm565982208._factor54(E:\cf9_final\cfusion\wwwroot\WEB-INF
\cftags\dump.cfm:119)
at cfdump2ecfm565982208.runPage(E:\cf9_final\cfusion\wwwroot\WEB-INF
\cftags\dump.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
at
coldfusion.filter.CFVariablesScopeFilter.invoke(CFVariablesScopeFilter.java:
56)
at coldfusion.tagext.lang.ModuleTag.doStartTag(ModuleTag.java:271)
at cfframework2ecfc336524780$funcDUMPEXCEPTION.runFunction(D:\web
\wwwroot\fw1\org\corfield\framework.cfc:1067)
at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472)
at coldfusion.runtime.UDFMethod
$ArgumentCollectionFilter.invoke(UDFMethod.java:368)
at
coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:
55)
at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321)
at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:220)
at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2582)
at cfframework2ecfc336524780$funcFAILURE.runFunction(D:\web\wwwroot
\fw1\org\corfield\framework.cfc:728)
at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472)
at coldfusion.runtime.UDFMethod
$ArgumentCollectionFilter.invoke(UDFMethod.java:368)
at
coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:
55)
at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321)
at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:220)
at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2582)
at cfframework2ecfc336524780$funcONERROR.runFunction(D:\web\\wwwroot
\fw1\org\corfield\framework.cfc:278)
at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472)
at coldfusion.runtime.UDFMethod
$ArgumentCollectionFilter.invoke(UDFMethod.java:368)
at
coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:
55)
at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321)
at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:220)
at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:490)
at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:336)
at coldfusion.runtime.AppEventInvoker.invoke(AppEventInvoker.java:88)
at coldfusion.runtime.AppEventInvoker.onError(AppEventInvoker.java:
427)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:
396)
at
coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:
48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:
40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:87)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
at
coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:
28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:
22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:53)
at coldfusion.CfmServlet.service(CfmServlet.java:200)
at
coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:
89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at
coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:
42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:
46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at
jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:
286)
at
jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:
543)
at
jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:
203)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)


When I asked Sean about this his suggestion was to rename the
frameworks onError method so I could see the actual error. I did this
but still got something very much the same. Here is the kicker, in my
setupApplication I am loading coldspring

public void function setupApplication() {
var params = {};

local.cs = new
lib.coldspring.beans.defaultXmlBeanFactory({},params);
cs.loadBeans(getDirectoryFromPath(getCurrentTemplatePath()) & "conf
\beans.xml");
setBeanFactory(cs);

}

If I comment out the 3 lines from above the app fires right up. Of
course the app won't work because none of my beans are in place. If I
uncomment those 3 lines and reload it, the app works as expected. I am
not sure if this is just a coincidence or what. I am not putting this
on fw/1 or coldspring. I am just really confused on hunting down this
bug and was hoping one of you smart people could help point me in the
right direction.

Thanks for taking the time out of your day to look at my problem!

--
You received this message because you are subscribed to the Google Groups "framework-one" group.
To post to this group, send email to framew...@googlegroups.com.
To unsubscribe from this group, send email to framework-on...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/framework-one?hl=en.

[Ryan Cogswell]

On Wed, May 5, 2010 at 8:19 AM, Dan Vega <dan...@gmail.com> wrote:
> I am seeing some really odd behavior from an application and I just
> can not seem to hunt down the culprit. I have this application that
> works great once it gets going. Its just the strangest thing ever
> though. When I first hit the url in the morning it takes 2 minutes and
> times out. I can hit it another 4 times and it does the same thing. I
> take a look at the logs and this is what I see.
>

While it's hung (before it times out), bring up the CF administrator
and create a snapshot (dump of the current stack traces from all
threads). That will hopefully give us better information for getting
at the root of the problem.

Ryan

[Shane Pitts]

I experience the exact same thing, but I wonder how different our setups are..  I'm using FW1 with ColdSpring and Reactor ORM.  I wasn't having any issues at all, until Sean opened my eyes to the AOP proxies in ColdSpring, and I transitioned all my beans to that.  What I deduced was that it was just having a really hard time creating all the proxies, writing all the remote files, and firing everything up in the allotted time.. 

The worst case scenario, I have Reactor in "development" mode, have reloadOnEveryRequest=true, and have * in all the method names for the Proxies on about 12 beans...  I can just about kiss startup goodbye.

But, if I comment out the proxy bean calls that are lazy-init=false in ColdSpring, starts up in no time.. Then if I set Reactor to 'production'.. it's all compiled and running before I can blink once..

It's a little painful during development, when I am constantly making changes, adding to the reactor xml, etc..  but I have a testing server that I run a fully compiled/cached/production version, and it's amazingly quick.

shane

[Dan Vega]

This is just weird. I don't think its a coldspring issue now either. I
am sorry for using this forum for my issues guys but while I am here I
might as well share what I have found. My other really big sight on
the server has the same issue. Times out after awhile after the server
restarts during the application startup. If I go to reload it seems to
come up fine and then the only way to reproduce this issue is to boot
the service. Here is a snapshot from when the server was hanging.

Snapshot generated on : Wed May 05 09:39:52 EDT 2010 Caused by :
User request
Profiling : Enabled
Memory Tracking : Enabled

*Total Running requests - 2
*Total Queued requests - 0
Template Running requests - 1
Template Queued requests - 0
Flash Remoting Running requests - 1
Flash Remoting Queued requests - 0
CFC method Running requests - 0
CFC method Queued requests - 0
Web Service Running requests - 0
Web Service Queued requests - 0

JVM Memory Stats
Free Memory : 34838352 bytes
Used Memory : 58943664 bytes

Server Scope Memory Used - 0 bytes
[Variable Type : Variable Name : Size]
STRUCT : coldfusion : 2992 bytes
STRUCT : os : 200 bytes

Application Scope Memory Used - 0 bytes
[Application Name : Memory Used]
cfadmin : 0 bytes
[Variable Type : Variable Name : Size]
cfadmin : applicationname : 56 bytes
portal : 0 bytes
[Variable Type : Variable Name : Size]
portal : applicationname : 64 bytes
dotcom : 0 bytes
[Variable Type : Variable Name : Size]
dotcom : applicationname : 56 bytes
false : debug : 48 bytes

Session Scope Memory Used - 0 bytes
[Session Id : Application Name :Session Size]
cfadmin_15523_66406941 : cfadmin : 0 bytes
[Variable Type : Variable Name : Size]
CFID=15523&CFTOKEN=66406941 : urltoken : 96 bytes
CFADMIN_15523_66406941 : sessionid : 80 bytes
66406941 : cftoken : 56 bytes
15523 : cfid : 48 bytes

Throttle Stats [Throttle Queue Size : Throttle Memory Used]
0 : 0.0 megabytes

Query Cache Stats -
[Hit Ratio : Size : Count]
0.0 : 0 bytes : 0

Cached Queries -
[Query Name : DSN : Size : TIme Executed : Time Taken : Line No :
Function Name : Template Path]

DB Pool Stats -
[DSN :Open Conn Count : Total Conn Count : Max Con Count : Avg Open
Conn Count : Avg Total Conn Count]
test2 : 0 : 0 : Unlimited : 0 : 0
** 50 more that look just like that **


Thread jrpp-4 request type - TEMPLATE REQUEST
*Template Path - D:\websites\portal\index.cfm
*Request Parameters - {}
*Request Method - GET
*Client IP address - ********
*Thread elapsed time - 7750 milliseconds
*Application Name - portal
*request/local/variables scope variables
[Scope Type: Type : Name : Size]
-APPLICATION : STRUCT : ___IMPLICITARRYSTRUCTVAR0 : 320 bytes
-APPLICATION : STRUCT : ORMSETTINGS : 320 bytes
-APPLICATION : STRUCT : FRAMEWORK : 152 bytes
-APPLICATION : STRUCT : ___IMPLICITARRYSTRUCTVAR1 : 152 bytes
-APPLICATION : SIMPLE : EVENTHANDLER : 128 bytes
-APPLICATION : SIMPLE : /coldspring : 120 bytes
-APPLICATION : SIMPLE : NAME : 64 bytes
-APPLICATION : SIMPLE : DATASOURCE : 64 bytes
-APPLICATION : SIMPLE : DEFAULTSECTION : 56 bytes
-APPLICATION : SIMPLE : DEFAULTITEM : 48 bytes
-APPLICATION : SIMPLE : EVENTHANDLING : 48 bytes
-APPLICATION : SIMPLE : ERROR : 48 bytes
-APPLICATION : SIMPLE : sessionmanagement : 48 bytes
-APPLICATION : SIMPLE : CFCLOCATION : 48 bytes
-APPLICATION : SIMPLE : FLUSHATREQUESTEND : 48 bytes
-APPLICATION : SIMPLE : DBCREATE : 48 bytes
-APPLICATION : SIMPLE : ORMENABLED : 48 bytes
-APPLICATION : SIMPLE : applicationtimeout : 16 bytes
-APPLICATION : SIMPLE : sessiontimeout : 16 bytes
-APPLICATION : STRUCT : MAPPINGS : 0 bytes
*CF stack -

*Currently executing queries -



Java stack trace -
Full thread dump Java 1.6.0_14

"obj-skimmer" prio=5 tid=76 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"jrpp-4" prio=5 tid=119 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
macromedia.jdbc.oracle.net8.OracleDataProvider.getArrayOfBytesFromSocket(Unknown
Source)
at
macromedia.jdbc.oracle.net8.OracleDataProvider.readFirstPacketInBuffer(Unknown
Source)
at
macromedia.jdbc.oracle.net8.OracleDataProvider.readPacket(Unknown
Source)
at macromedia.jdbc.oracle.net8.OracleDataProvider.receive(Unknown
Source)
at
macromedia.jdbc.oracle.net8.OracleNet8NSPTDAPacket.sendRequest(Unknown
Source)
at macromedia.jdbc.oracle.OracleImplStatement.fetchNext(Unknown
Source)
at macromedia.jdbc.oracle.OracleImplStatement.fetchNext2(Unknown
Source)
at macromedia.jdbc.oracle.OracleImplStatement.execute(Unknown
Source)
at macromedia.jdbc.oraclebase.BaseStatement.commonExecute(Unknown
Source)
at
macromedia.jdbc.oraclebase.BaseStatement.executeQueryInternal(Unknown
Source)
at
macromedia.jdbc.oraclebase.BaseStatement.executeDatabaseMetaDataQuery(Unknown
Source)
at
macromedia.jdbc.oraclebase.BaseDatabaseMetaData.getColumns(Unknown
Source)
at coldfusion.sql.TableMetaData.initColumns(TableMetaData.java:
133)
at coldfusion.sql.TableMetaData.<init>(TableMetaData.java:52)
at coldfusion.sql.DBMetaData.getTableMetaData(DBMetaData.java:93)
at
coldfusion.orm.mapping.CFPropertyBinder.readCFCData(CFPropertyBinder.java:
319)
at
coldfusion.orm.mapping.CFPropertyBinder.buildCFCPersistenceMetadata(CFPropertyBinder.java:
187)
at
coldfusion.orm.mapping.CFPropertyBinder.getCFCPersistenceMetadata(CFPropertyBinder.java:
167)
at
coldfusion.orm.hibernate.HibernateConfiguration.generateHbmDocFromCFCs(HibernateConfiguration.java:
759)
at
coldfusion.orm.hibernate.HibernateConfiguration.buildConfiguration(HibernateConfiguration.java:
511)
at
coldfusion.orm.hibernate.HibernateConfiguration.initHibernateConfiguration(HibernateConfiguration.java:
180)
at
coldfusion.orm.hibernate.HibernateConfiguration.<init>(HibernateConfiguration.java:
141)
at
coldfusion.orm.hibernate.ConfigurationManager.initConfiguration(ConfigurationManager.java:
69)
at
coldfusion.orm.hibernate.HibernateProvider.InitializeORMForApplication(HibernateProvider.java:
182)
at
coldfusion.orm.hibernate.HibernateProvider.beforeApplicationStart(HibernateProvider.java:
85)
at
coldfusion.filter.ApplicationFilter.fireBeforeAppStartEvent(ApplicationFilter.java:
475)
at
coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:221)

at
coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:
48)
at
coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:87)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:
70)
at

coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:
74)

"obj-skimmer" prio=5 tid=91 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"tyrex.util.daemonMaster" prio=10 tid=27 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at jrunx.tyrex.services.DaemonMaster.run(DaemonMaster.java:269)
at java.lang.Thread.run(Thread.java:619)

"scheduler-10" prio=5 tid=40 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"org.apache.commons.vfs.cache.SoftRefFilesCache$SoftRefReleaseThread"
prio=5 tid=96 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at org.apache.commons.vfs.cache.SoftRefFilesCache
$SoftRefReleaseThread.run(SoftRefFilesCache.java:79)

"worker #2" prio=5 tid=106 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.jnbridge.jnbcore.server.b.c.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)

"Thread-17" prio=5 tid=102 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
coldfusion.util.SimpleThreadPool.getAssignment(SimpleThreadPool.java:
113)
at coldfusion.util.SimpleWorkerThread.run(SimpleThreadPool.java:
207)

"Finalizer" prio=8 tid=3 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:
159)

"scheduler-5" prio=5 tid=35 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"jndi-5" prio=5 tid=113 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at jrunx.scheduler.JSemaphore.acquire(JSemaphore.java:74)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
349)
at
jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:115)
at
jrun.naming.NamingNetworkService.createRunnable(NamingNetworkService.java:
141)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"scheduler-2" prio=5 tid=110 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:
132)
at
coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:68)

"Thread-13" prio=5 tid=97 RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
coldfusion.server.jrun4.metrics.CfstatServer.run(CfstatServer.java:74)

"obj-skimmer" prio=5 tid=87 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"Attach Listener" prio=5 tid=5 RUNNABLE

"obj-skimmer" prio=5 tid=85 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"worker #1" prio=5 tid=105 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.jnbridge.jnbcore.server.b.c.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)

"Timer-1" prio=5 tid=60 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)

"scheduler-4" prio=5 tid=34 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"scheduler-9" prio=5 tid=39 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"jrpp-3" prio=5 tid=118 RUNNABLE
at java.lang.Thread.getStackTrace(Thread.java:1436)
at
coldfusion.monitor.stack.J5StackTrace.getAllStackTraces(J5StackTrace.java:
39)
at
coldfusion.monitor.active.Snapshot.getJavaStackTrace(Snapshot.java:
583)
at
coldfusion.monitor.active.Snapshot.getSnapshotData(Snapshot.java:149)
at coldfusion.monitor.active.Snapshot.dumpSnapshot(Snapshot.java:
572)
at coldfusion.monitor.active.Snapshot.dumpSnapshot(Snapshot.java:
559)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at coldfusion.runtime.StructBean.invoke(StructBean.java:502)
at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2393)
at
cfservermonitoring2ecfc809089413$funcDUMPSNAPSHOT.runFunction(E:
\cf9_final\cfusion\wwwroot\CFIDE\adminapi\servermonitoring.cfc:1971)

at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472)
at coldfusion.runtime.UDFMethod

$ReturnTypeFilter.invoke(UDFMethod.java:405)

at coldfusion.runtime.UDFMethod
$ArgumentCollectionFilter.invoke(UDFMethod.java:368)
at
coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:
55)
at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:
321)
at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:220)
at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:
490)
at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:

436)
at
coldfusion.flash.filter.CFCInvokeFilter.invokeMethod(CFCInvokeFilter.java:
308)
at
coldfusion.flash.filter.CFCInvokeFilter.invoke(CFCInvokeFilter.java:
198)
at
coldfusion.flash.filter.CFCInvokeFilter.invoke(CFCInvokeFilter.java:
76)
at
coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:363)
at
coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at
coldfusion.flash.filter.CFCInvokeDebugFilter.invoke(CFCInvokeDebugFilter.java:
55)
at
coldfusion.flash.filter.CFCInvokePathFilter.invoke(CFCInvokePathFilter.java:
70)
at
coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:
28)

at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at
coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at

coldfusion.flash.messaging.ColdFusionAdapter.invoke(ColdFusionAdapter.java:
298)
at
flex.messaging.services.RemotingService.serviceMessage(RemotingService.java:
183)
at
flex.messaging.MessageBroker.routeMessageToService(MessageBroker.java:
1503)
at
flex.messaging.endpoints.AbstractEndpoint.serviceMessage(AbstractEndpoint.java:
884)
at
flex.messaging.endpoints.amf.MessageBrokerFilter.invoke(MessageBrokerFilter.java:
121)
at
flex.messaging.endpoints.amf.LegacyFilter.invoke(LegacyFilter.java:
158)
at
flex.messaging.endpoints.amf.SessionFilter.invoke(SessionFilter.java:
44)
at
flex.messaging.endpoints.amf.BatchProcessFilter.invoke(BatchProcessFilter.java:
67)
at
flex.messaging.endpoints.amf.SerializationFilter.invoke(SerializationFilter.java:
146)
at
flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:
278)
at
coldfusion.flash.messaging.CFAMFEndPoint.service(CFAMFEndPoint.java:
200)
at
flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:
322)
at
coldfusion.flex.ColdFusionMessageBrokerServlet.service(ColdFusionMessageBrokerServlet.java:
51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

at
coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:
89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at

coldfusion.filter.FlashRequestControlFilter.doFilter(FlashRequestControlFilter.java:
71)

at
coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:
42)
at
jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:
286)
at
jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:
543)
at
jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:
203)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"scheduler-1" prio=5 tid=98 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:
132)
at
coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:68)

"jndi-4" prio=5 tid=26 BLOCKED
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:388)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
357)
at
jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:115)
at
jrun.naming.NamingNetworkService.createRunnable(NamingNetworkService.java:
141)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"GC Daemon" prio=2 tid=13 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at sun.misc.GC$Daemon.run(GC.java:100)

"RMI TCP Accept-0" prio=5 tid=11 RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.rmi.transport.tcp.TCPTransport
$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport
$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:619)

"scheduler-12" prio=5 tid=42 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"obj-skimmer" prio=5 tid=93 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"scheduler-3" prio=5 tid=33 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"jndi-3" prio=5 tid=25 BLOCKED
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:388)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
357)
at
jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:115)
at
jrun.naming.NamingNetworkService.createRunnable(NamingNetworkService.java:
141)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"scheduler-13" prio=5 tid=43 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"jms-fifo-1" prio=5 tid=14 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
at
jrun.jms.core.threads.ThreadPoolWorker.runWorker(ThreadPoolWorker.java:
91)
at jrun.jms.core.threads.ThreadPoolWorker
$1.run(ThreadPoolWorker.java:49)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=89 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"DestroyJavaVM" prio=5 tid=114 RUNNABLE

"jms-fifo-5" prio=5 tid=18 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
at
jrun.jms.core.threads.ThreadPoolWorker.runWorker(ThreadPoolWorker.java:
91)
at jrun.jms.core.threads.ThreadPoolWorker
$1.run(ThreadPoolWorker.java:49)
at java.lang.Thread.run(Thread.java:619)

"worker #0" prio=5 tid=104 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.jnbridge.jnbcore.server.b.c.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=94 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=73 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=75 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"jms-fifo-3" prio=5 tid=16 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
at
jrun.jms.core.threads.ThreadPoolWorker.runWorker(ThreadPoolWorker.java:
91)
at jrun.jms.core.threads.ThreadPoolWorker
$1.run(ThreadPoolWorker.java:49)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=74 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"worker #4" prio=5 tid=108 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.jnbridge.jnbcore.server.b.c.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)

"Thread-8" prio=5 tid=62 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:
95)

"scheduler-19" prio=5 tid=49 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"scheduler-0" prio=5 tid=65 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at coldfusion.scheduling.Scheduler.createRunnable(Scheduler.java:
132)
at
coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:68)

"jms-fifo-4" prio=5 tid=17 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
at
jrun.jms.core.threads.ThreadPoolWorker.runWorker(ThreadPoolWorker.java:
91)
at jrun.jms.core.threads.ThreadPoolWorker
$1.run(ThreadPoolWorker.java:49)
at java.lang.Thread.run(Thread.java:619)

"scheduler-6" prio=5 tid=36 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"scheduler-18" prio=5 tid=48 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"jndi-0" prio=5 tid=22 BLOCKED
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:388)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
357)
at
jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:115)
at
jrun.naming.NamingNetworkService.createRunnable(NamingNetworkService.java:
141)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"obj-skimmer" prio=5 tid=66 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"jndi-2" prio=5 tid=24 BLOCKED
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:388)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
357)
at
jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:115)
at
jrun.naming.NamingNetworkService.createRunnable(NamingNetworkService.java:
141)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"Java2D Disposer" prio=10 tid=109 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at sun.java2d.Disposer.run(Disposer.java:125)
at java.lang.Thread.run(Thread.java:619)

"RMI Reaper" prio=5 tid=12 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=81 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"scheduler-20" prio=5 tid=56 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"scheduler-14" prio=5 tid=44 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"obj-skimmer" prio=5 tid=86 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=69 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=67 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"worker #3" prio=5 tid=107 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.jnbridge.jnbcore.server.b.c.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=68 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=70 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=83 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=71 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"Signal Dispatcher" prio=9 tid=4 RUNNABLE

"Thread-4" prio=5 tid=54 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at jrun.jms.core.messagemgr.Scheduler.next(Scheduler.java:298)
at jrun.jms.core.messagemgr.Scheduler.run(Scheduler.java:233)
at java.lang.Thread.run(Thread.java:619)

"scheduler-7" prio=5 tid=37 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"scheduler-16" prio=5 tid=46 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"obj-skimmer" prio=5 tid=79 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"jrpp-0" prio=5 tid=57 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:
258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:
581)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:
573)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:
591)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at
jrun.servlet.jrpp.JRunProxyService.swapRunnable(JRunProxyService.java:
143)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.swapRunnable(ThreadPool.java:292)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.swapRunnable(ThreadPool.java:410)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.swapRunnable(ThreadPool.java:252)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

"mipc-1" prio=5 tid=55 RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
jrun.jms.core.kernel.mipc.MultiplexConnectionServer.run(MultiplexConnectionServer.java:
155)

"Transaction Domain DefaultDomain" prio=5 tid=28 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
jrunx.tyrex.tm.impl.TransactionDomainImpl.run(TransactionDomainImpl.java:
1300)
at java.lang.Thread.run(Thread.java:619)

"scheduler-17" prio=5 tid=47 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"obj-skimmer" prio=5 tid=72 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"jrpp-5" prio=5 tid=121 RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
357)
at
jrun.servlet.jrpp.JRunProxyService.accept(JRunProxyService.java:103)
at
jrun.servlet.jrpp.JRunProxyService.createRunnable(JRunProxyService.java:
119)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"obj-skimmer" prio=5 tid=88 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"scheduler-0" prio=5 tid=30 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"jms-fifo-2" prio=5 tid=15 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
jrun.jms.core.threads.FIFOQueue.queueHasElements(FIFOQueue.java:45)
at jrun.jms.core.threads.FIFOQueue.get(FIFOQueue.java:65)
at
jrun.jms.core.threads.ThreadPoolWorker.runWorker(ThreadPoolWorker.java:
91)
at jrun.jms.core.threads.ThreadPoolWorker
$1.run(ThreadPoolWorker.java:49)
at java.lang.Thread.run(Thread.java:619)

"jrpp-1" prio=5 tid=115 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:
258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:
581)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:
573)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:
591)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at
jrun.servlet.jrpp.JRunProxyService.swapRunnable(JRunProxyService.java:
143)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.swapRunnable(ThreadPool.java:292)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.swapRunnable(ThreadPool.java:410)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.swapRunnable(ThreadPool.java:252)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

"jndi-1" prio=5 tid=23 RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:
357)
at
jrun.naming.NamingNetworkService.accept(NamingNetworkService.java:115)
at
jrun.naming.NamingNetworkService.createRunnable(NamingNetworkService.java:
141)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"scheduler-15" prio=5 tid=45 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"Transaction Domain PrivateJmsDomain" prio=5 tid=29 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
jrunx.tyrex.tm.impl.TransactionDomainImpl.run(TransactionDomainImpl.java:
1300)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=92 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"RMI Scheduler(0)" prio=5 tid=20 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer
$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor
$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor
$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
947)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

"clock" prio=10 tid=51 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at jrunx.tyrex.services.Clock.run(Clock.java:202)

"scheduler-8" prio=5 tid=38 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"Timer-2" prio=5 tid=61 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)

"Reference Handler" prio=10 tid=2 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:
116)

"cfthread-0" prio=5 tid=120 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at coldfusion.util.GenericThreadPool
$ThreadPoolRunnableFactory.createRunnable(GenericThreadPool.java:177)
at
coldfusion.scheduling.ThreadPool.createRunnable(ThreadPool.java:128)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:68)

"scheduler-21" prio=5 tid=58 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"obj-skimmer" prio=5 tid=78 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=84 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=77 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=82 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"obj-skimmer" prio=5 tid=90 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"Timer-0" prio=5 tid=59 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)

"scheduler-2" prio=5 tid=32 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"obj-skimmer" prio=5 tid=80 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:
47)
at java.lang.Thread.run(Thread.java:619)

"ServerThread" prio=5 tid=103 RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at com.jnbridge.jnbcore.server.b.b.if(Unknown Source)
at com.jnbridge.jnbcore.JNBMain$a.run(Unknown Source)

"scheduler-11" prio=5 tid=41 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"scheduler-1" prio=5 tid=31 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at
jrunx.scheduler.SchedulerService.createRunnable(SchedulerService.java:
188)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.createRunnable(ThreadPool.java:287)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.createRunnable(ThreadPool.java:349)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.createRunnable(ThreadPool.java:241)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"jrpp-2" prio=5 tid=117 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:
258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:
581)
at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:
573)
at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:
591)
at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:
231)
at
jrun.servlet.jrpp.JRunProxyService.swapRunnable(JRunProxyService.java:
143)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.swapRunnable(ThreadPool.java:292)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.swapRunnable(ThreadPool.java:410)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.swapRunnable(ThreadPool.java:252)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

[Ryan Cogswell]

So the thread below is the only one of interest which is in the middle
of the Hhibernate ORM initialization. I guess the first thing I would
do is adjust your request timeout settings to something much higher
(e.g. 30 minutes instead of 2). Chances are you just have an
expensive application initialization. I would first figure out just
how long the first request takes if allowed to continue. I haven't
used Hibernate, so I'm not familiar with what all it may be trying to
do during its initialization and what factors might cause such a slow
startup time.

[Dan Vega]

Ryan you are the man!
Wow, so I rebooted the service (its the only way to replicate the
issue) but this time set

this.ormenabled = false

and it fired right up! So with that we can deduct that the problem is
when the application (any app) that uses Hibernate is first
initialized that it goes through some setup. That setup is taking
longer than the request timeout and therefor its causing a timeout.
Has anyone else come across this as well?

1.) With 4 applications on this server that use hibernate what are
optimal settings?
2.) Is this a good place for onServerStart() ? Never used it, but
could it essential initialize these applications? nks

Thanks for looking into this everyone!

[robert shires]

Do you specify the CFClocation for ORM? I know that this can speed up initialization by by reducing the number of directories CF will have to scan.

"Specifies the directory (or array of directories) that should be used by ColdFusion to search for persistent CFCs to generate the mapping. If cfclocation is set, ColdFusion looks at only the paths specified in it. If it is not set, ColdFusion looks at the application directory, its sub-directories, and its mapped directories to search for persistent CFCs."

I would think that with 4 app this might cause a slowdown.

Robert

[Dan Vega]

Yes - Always. In this particular application I only have one
directory. I wonder if I generated the hbmxml files and it no longer
had to do that if it would speed it up, I will give that a try.

Dan

> > framework-on...@googlegroups.com<framework-one%2Bunsubscribe@goog legroups.com>
> > .

> > > For more options, visit this group athttp://
> > groups.google.com/group/framework-one?hl=en.
>

> > --
> > You received this message because you are subscribed to the Google Groups
> > "framework-one" group.
> > To post to this group, send email to framew...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > framework-on...@googlegroups.com<framework-one%2Bunsubscribe@goog legroups.com>
> > .

[Sean Corfield]

On Wed, May 5, 2010 at 7:10 AM, Dan Vega <dan...@gmail.com> wrote:
> Snapshot generated on : Wed May 05 09:39:52 EDT 2010      Caused by :
> User request
> Profiling : Enabled
> Memory Tracking : Enabled

Really? Profiling and Memory Tracking Enabled? Check the Server
Monitor. If you have these settings enabled, performance will tank on
certain things.

This catches a lot of people out because they go into the Server
Monitor, enable the options to do some debugging and then quit the
Server Monitor without disabling these options... but they're sticky
options and they remain enabled *even after a server restart*... And,
for performance, they can be pretty bad on CFC-based apps.

Just something else to check.
--
Sean A Corfield -- (904) 302-SEAN
Railo Technologies, Inc. -- http://getrailo.com/
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

[Dan Vega]

No I saw the same symptoms on all 3 servers (dev/test/prod). I only
turned profiling on the dev server to take a snapshot.

On May 5, 5:45 pm, Sean Corfield <seancorfi...@gmail.com> wrote:

> On Wed, May 5, 2010 at 7:10 AM, Dan Vega <danv...@gmail.com> wrote:
> > Snapshot generated on : Wed May 05 09:39:52 EDT 2010      Caused by :
> > User request
> > Profiling : Enabled
> > Memory Tracking : Enabled
>
> Really? Profiling and Memory Tracking Enabled? Check the Server
> Monitor. If you have these settings enabled, performance will tank on
> certain things.
>
> This catches a lot of people out because they go into the Server
> Monitor, enable the options to do some debugging and then quit the
> Server Monitor without disabling these options... but they're sticky
> options and they remain enabled *even after a server restart*... And,
> for performance, they can be pretty bad on CFC-based apps.
>
> Just something else to check.
> --
> Sean A Corfield -- (904) 302-SEAN

> Railo Technologies, Inc. --http://getrailo.com/
> An Architect's View --http://corfield.org/

[Dan Vega]

Saw the same issues again this morning. Changing the request timeout
to 90 sec fixed the problem. I am not convinced that this is the right
solution though. There has to be a way to figure out what is causing
this slow start up.