Hung threads in application server logs
ish_a...@yahoo.co.in
I am seeing the hung threads exception in my logs as below
[3/12/09 20:55:17:223 MST] 5dd92e49 ThreadMonitor W WSVR0605W: Thread "Servlet.Engine.Transports : 5" (1e13ae49) has been active for 4,873,079 milliseconds and may be hung. There are 1 threads in total in the server that may be hung.
[3/12/09 20:55:24:751 MST] 5dd92e49 ThreadMonitor W WSVR0605W: Thread "Servlet.Engine.Transports : 8" (3ae7ee48) has been active for 6,702,997 milliseconds and may be hung. There are 2 threads in total in the server that may be hung.
[3/12/09 20:55:25:499 MST] 5dd92e49 ThreadMonitor W WSVR0605W: Thread "Servlet.Engine.Transports : 0" (4b4d2e49) has been active for 4,875,358 milliseconds and may be hung. There are 3 threads in total in the server that may be hung.
[3/12/09 20:55:25:999 MST] 5dd92e49 ThreadMonitor W WSVR0605W: Thread "Servlet.Engine.Transports : 3" (3a152e49) has been active for 4,875,010 milliseconds and may be hung. There are 4 threads in total in the server that may be hung.
[3/12/09 20:55:08:171 MST] 3a152e49 TraceNLS u No message text associated with key HttpConnection.run:.java.lang.OutOfMemoryError in bundle com.ibm.ejs.resources.seriousMessages
[3/12/09 20:55:09:649 MST] 317dae49 TraceNLS u No message text associated with key Unable.to.read.and.handle.request:.null in bundle com.ibm.ejs.resources.seriousMessages
[3/12/09 20:55:15:537 MST] 1e13ae49 TraceNLS u No message text associated with key Unable.to.read.and.handle.request:.null in bundle com.ibm.ejs.resources.seriousMessages
[3/12/09 20:55:15:537 MST] 4b4d2e49 TraceNLS u No message text associated with key Unable.to.read.and.handle.request:.null in bundle com.ibm.ejs.resources.seriousMessages
[3/12/09 20:55:09:649 MST] 3ae7ee48 TraceNLS u No message text associated with key HttpConnection.run:.java.lang.OutOfMemoryError in bundle com.ibm.ejs.resources.seriousMessages
[3/12/09 20:55:09:649 MST] 1a6c6e49 TraceNLS u No message text associated with key HttpConnection.run:.java.lang.OutOfMemoryError in bundle com.ibm.ejs.resources.seriousMessages
[3/12/09 20:55:08:121 MST] 3a152e49 HttpConnectio E HttpConnection.run: java.lang.OutOfMemoryError
[3/12/09 20:55:08:133 MST] 317dae49 HttpConnectio E Unable to read and handle request: null
[3/12/09 20:55:27:828 MST] 3d9bee42 TraceNLS u No message text associated with key Unable.to.read.and.handle.request:.JVMXE006:OutOfMemoryError,.stAllocArray.for.executeJava.failed in bundle com.ibm.ejs.resources.seriousMessages
[3/12/09 20:55:08:133 MST] 1e13ae49 HttpConnectio E Unable to read and handle request: null
[3/12/09 20:55:15:537 MST] 4b4d2e49 HttpConnectio E Unable to read and handle request: null
*and i too have the following exception*
[3/12/09 17:39:06:608 MST] 363cae49 RoleBasedAuth E SECJ0306E: No received or invocation credential exist on the thread. The Role based authorization check will not have an accessId of the caller to check. The parameters are: access check method getState on resource Server and module Server. The stack trace is java.lang.Exception: dump thread stack for debugging
at com.ibm.ws.security.role.RoleBasedAuthorizerImpl.checkAccess(RoleBasedAuthorizerImpl.java(Compiled Code))
at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java(Compiled Code))
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java(Compiled Code))
at com.ibm.ws.management.connector.AdminServiceDelegator.invoke(AdminServiceDelegator.java(Compiled Code))
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.ibm.ws.management.connector.soap.SOAPConnector.invoke(SOAPConnector.java(Compiled Code))
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java(Compiled Code))
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java(Compiled Code))
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java(Compiled Code))
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java(Compiled Code))
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
.
[3/12/09 17:39:06:621 MST] 363cae49 RoleBasedAuth A SECJ0305I: Role based authorization check failed for security name , accessId no_cred_no_access_id while invoking method getState on resource Server and module Server.
*Eventually the Out of Memory Exception occurs and the application hangs. Are these two interrelated? What shall be done to remove this?We are running on AIX machine with Websphere5.1.*
*Thanks in Advance.*
Brian S Paskin
Regards,
Brian
walker.l2
> Hi, Your thread dump does not show the hung threads. You did not take it in time.
The OP hasn't provided a thread dump, just an extract from the
System.out / System.err logs
walker.l2
>
> What shall be done to remove this?
>
Take thread dump / JVM dump when you start seeing hung threads (before
the OOM Error occurs if possible - this may be difficult because
according to the log timestamps your OOM seems to occur before your
thread hangs). Your application is probably deadlocked somewhere and
starting up new threads faster than old ones finish (because the old
ones are deadlocked) until eventually you run out of memory.
> We are running on AIX machine with Websphere5.1.*
>
Then you are running on an unsupported version (5.1 went out of
support last September)
ish_a...@yahoo.co.in
I have attached the heap dumps.
Thanks
ish_a...@yahoo.co.in
Brian S Paskin
Regards,
Brian