CF8 runaway process, need help with thread dump
jl
We are running CF801 multi server, IIS7, Windows2008. About one or
twice a week a runaway process appears and the only way to get rid of
it is to restart the JRUN admin process. The process cannot be killed
from the monitor no matter how many times I click on the "!". I took a
snapshot while 4 of these runaways was in process. Below is the info
related to one of the processes. All 4 of the process dumps list the
same query and the same classes involved in the stack trace. This page
(search.cfm) does a couple of things 1) queries a search service
(Endeca) via http call to find items that are searched for 2) queries
our DB to get information about items to display. #1 uses http calls
to the search service and #2 uses jdbc to query the database so I see
2 possibilities where this may be stuck, either the http call or the
query. I do not think it is the query because the query does not show
up in the CF monitor active or slowest query screens.
I think it is related to the http call to the search service. As you
read up the stack trace from the bottom one will notice the call to
the search service
at com.endeca.navigation.HttpENEConnection.query(Unknown Source)
Reading further up the trace it appears to be reading the returned
information
at java.io.FilterInputStream.read(FilterInputStream.java:111)
at sun.net.www.protocol.http.HttpURLConnection
$HttpInputStream.read(HttpURLConnection.java:2254)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
The trace ends on a read.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
I am not familiar with reading stack traces and would like your
opinion on what you think the issue may be. After the reading of the
returned information is complete shouldn't there be a closing of the
input stream socket? Any ideas aer appreciated. Below is the dump from
one of these processes.
Thanks.
Joe
Thread jrpp-2111 request type - TEMPLATE REQUEST
*Template Path - search.cfm
*Request Parameters - {Ne=[Ljava.lang.String;@1a50e036, N=
[Ljava.lang.String;@33b006b9}
*Request Method - GET
*Client IP address - some_ip
*Thread elapsed time - 278645639 milliseconds
*Application Name - some_name
*CF stack -
search.cfm
at search.cfm
*Currently executing queries -
[IsCached : Name : Line No : DSN : Size: Start Time]
false : some_cfc: 78 : some_jdbc: 500 bytes : Mon Aug 17 08:40:58 EDT
2009
-SQL - select first 1 ....
from some_table
where num = some_ID_number
"jrpp-2111" prio=5 tid=1058181 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:
254)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at sun.net.www.MeteredStream.read(MeteredStream.java:116)
at java.io.FilterInputStream.read(FilterInputStream.java:111)
at sun.net.www.protocol.http.HttpURLConnection
$HttpInputStream.read(HttpURLConnection.java:2254)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:
256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at com.endeca.navigation.OptiInputBuffer.read(Unknown Source)
at com.endeca.navigation.OptiInputBuffer.readFully(Unknown
Source)
at com.endeca.navigation.OptiInputBuffer.readLine(Unknown Source)
at com.endeca.navigation.OptiBinaryInterp.FormatAttribute(Unknown
Source)
at com.endeca.navigation.OptiBinaryInterp.FormatBin(Unknown
Source)
at com.endeca.navigation.OptiBinaryInterp.FormatSeeAlso(Unknown
Source)
at com.endeca.navigation.OptiBinaryInterp.FormatSeeAlsos(Unknown
Source)
at com.endeca.navigation.OptiBinaryInterp.ParseBinaryNode(Unknown
Source)
at com.endeca.navigation.OptiBackend.getNavigation(Unknown
Source)
at com.endeca.navigation.HttpENEConnection.query(Unknown Source)
at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at coldfusion.runtime.java.JavaProxy.invoke(JavaProxy.java:87)
at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2272)
at cfsearch2ecfm1211332420._factor57(search.cfm:299)
at cfsearch2ecfm1211332420.runPage(search.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:
370)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:
65)
at coldfusion.filter.ApplicationFilter.invoke
(ApplicationFilter.java:273)
at coldfusion.filter.RequestMonitorFilter.invoke
(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke
(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:
70)
at coldfusion.filter.ClientScopePersistenceFilter.invoke
(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke
(DatasourceFilter.java:22)
at coldfusion.CfmServlet.service(CfmServlet.java:175)
at coldfusion.bootstrap.BootstrapServlet.service
(BootstrapServlet.java:89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter
(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter
(BootstrapFilter.java:46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:
42)
at jrun.servlet.JRunRequestDispatcher.invoke
(JRunRequestDispatcher.java:286)
at jrun.servlet.ServletEngineService.dispatch
(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable
(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable
(ThreadPool.java:428)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)