Our weblogic server stops responding after some time. We have been seeing all
the possiblities. One thing we found is, its throwing lots and lots of Broken
Pipe exceptions. I just wanted to know, whether this exception may cause the app
server stops responding after a period of time.
I read from this newsgroup that these exceptions are coming because, user may
hit the stop or reload button before he gets the response.
We are using WLS4.5.1 and jdk 1.2.2. We always get the broken pipe exception when
we forward the httpRequest to another jsp/servlet.
I would like to know whether we can handle this exception and whether this would
cause my app server to stop responding.
I appreciate your help.
THanks
V Prakash
*********** java.io.IOException: Broken pipe
javax.servlet.ServletException: java.io.IOException: Broken pipe
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Compiled Code)
at java.lang.Throwable.<init>(Compiled Code)
at java.lang.Exception.<init>(Compiled Code)
at javax.servlet.ServletException.<init>(ServletException.java:68)
at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispat
cherImpl.java:111)
at com.rhii.at.controller.utilities.LOBUtilities.invokeJsp(LOBUtilities.
java)
at com.rhii.at.controller.utilities.LOBUtilities.invokeJsp(LOBUtilities.
java)
at com.rhii.at.controller.Home.service(Home.java)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:840)
at com.wily.introscope.agent.probe.servlet.ManagedHttpServlet.service(Ma
nagedHttpServlet.java:95)
Hmm. Not sure whether the broken pipe situation is related to your server hang
(although it could be). Are you able to take a thread dump when the server hangs
(http://www.weblogic.com/docs45/techsupport/threaddump.html)? If you can please post
the results here.
What OS version are you running? Also, are you running with the latest service pack
(sp 15)? If not please download and test with the latest service pack.
Thanks,
Michael
Prakash wrote:
--
Michael Young
Developer Relations Engineer
BEA Support
Thanks for your reply:
I am using sp13
Running Sun Solaris OS5.7
And, Here is the coredump (full). ( I could not post the full dump, so
deleted some from the middle.)
SIGQUIT
A SIGQUIT has been received. Do you want to:
[ 0 ] continue program
[ 1 ] check & print one deadlock
[ 2 ] check & print all deadlocks
[ 3 ] dump thread stacks
[ 4 ] dump lock registry
[ 5 ] heap inspection
[ 6 ] terminate program
Type number corresponding to selected action: Found 0 deadlock
Full thread dump:
"HighPriority TimeEventGenerator" (TID:0x1fcc03c,
sys_thread_t:0x1fcbf80, state:CW, thread_t: t@151,
threadID:0xd8231dc8, stack_bottom:0xd8232000, stack_size:0x20000)
prio=9
[1] weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:256)
[2] weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:143)
[3] java.lang.Thread.run(Thread.java:485)
-------------------
"ListenThread" (TID:0x1e57894, sys_thread_t:0x1e577d8, state:R,
thread_t: t@150, threadID:0xd8131dc8, stack_bottom:0xd8132000,
stack_size:0x20000) prio=5
[1] java.net.PlainSocketImpl.socketAccept(Native Method)
[2] java.net.PlainSocketImpl.accept(PlainSocketImpl.java:406)
[3] java.net.ServerSocket.implAccept(ServerSocket.java:237)
[4] java.net.ServerSocket.accept(ServerSocket.java:223)
[5] weblogic.t3.srvr.ListenThread.run(ListenThread.java:230)
-------------------
"Introscope Agent Reporter" (TID:0x120817c, sys_thread_t:0x12080c0,
state:CW, thread_t: t@149, threadID:0xd8171dc8,
stack_bottom:0xd8172000, stack_size:0x20000) prio=5
[1] java.lang.Object.wait(Object.java:424)
[2] com.wily.util.adt.FixedSizeRollingQueue.blockingDequeue(FixedSizeRollingQueue.java:66)
[3] com.wily.jip.JIPClient.sendNextQueuedBinding(JIPClient.java:1197)
[4] com.wily.jip.JIPClient$Reporter.run(JIPClient.java:1368)
[5] java.lang.Thread.run(Thread.java:485)
-------------------
"Introscope Agent Listener" (TID:0x11d759c, sys_thread_t:0x11d74e0,
state:R, thread_t: t@148, threadID:0xd81a1dc8,
stack_bottom:0xd81a2000, stack_size:0x20000) prio=5
[1] java.net.SocketInputStream.socketRead(Native Method)
[2] java.net.SocketInputStream.read(SocketInputStream.java:85)
[3] java.io.BufferedInputStream.fill(BufferedInputStream.java:171)
[4] java.io.BufferedInputStream.read(BufferedInputStream.java:207)
[5] java.io.FilterInputStream.read(FilterInputStream.java:73)
[6] com.wily.util.io.EncryptInputStream.read(EncryptInputStream.java:81)
[7] java.io.DataInputStream.readByte(DataInputStream.java:224)
[8] com.wily.jip.JIPClient.handleServerCommand(JIPClient.java:1025)
[9] com.wily.jip.JIPClient$Listener.run(JIPClient.java:1406)
[10] java.lang.Thread.run(Thread.java:485)
-------------------
"Introscope Agent Connector" (TID:0x11d7e04, sys_thread_t:0x11d7d48,
state:CW, thread_t: t@147, threadID:0xd81d1dc8,
stack_bottom:0xd81d2000, stack_size:0x20000) prio=5
[1] com.wily.jip.JIPClient.waitForStates(JIPClient.java:407)
[2] com.wily.jip.JIPClient.waitForStates(JIPClient.java:391)
[3] com.wily.jip.JIPClient$Connector.run(JIPClient.java:1315)
[4] java.lang.Thread.run(Thread.java:485)
-------------------
"null Agent" (TID:0x1119d2c, sys_thread_t:0x1119c70, state:CW,
thread_t: t@146, threadID:0xd8201dc8, stack_bottom:0xd8202000,
stack_size:0x20000) prio=5
[1] java.lang.Thread.sleep(Native Method)
[2] com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:538)
[3] java.lang.Thread.run(Thread.java:485)
-------------------
"NBExecuteThread-1" (TID:0xadd664, sys_thread_t:0xadd5a8, state:CW,
thread_t: t@142, threadID:0xd8b11dc8, stack_bottom:0xd8b12000,
stack_size:0x20000) prio=5
[1] java.lang.Object.wait(Object.java:424)
[2] weblogic.t3.srvr.ExecuteThread.waitForRequest(ExecuteThread.java:86)
[3] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"NBExecuteThread-0" (TID:0xadc494, sys_thread_t:0xadc3d8, state:CW,
thread_t: t@141, threadID:0xd8b41dc8, stack_bottom:0xd8b42000,
stack_size:0x20000) prio=5
[1] java.lang.Object.wait(Object.java:424)
[2] weblogic.t3.srvr.ExecuteThread.waitForRequest(ExecuteThread.java:86)
[3] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-127" (TID:0xadb0dc, sys_thread_t:0xadb020, state:CW,
thread_t: t@140, threadID:0xd8b71dc8, stack_bottom:0xd8b72000,
stack_size:0x20000) prio=5
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3] weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4] weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5] weblogic.cluster.replication.ReplicationManager_WLStub.create(ReplicationManager_WLStub.java:77)
[6] weblogic.cluster.replication.ReplicationManager.createSecondary(ReplicationManager.java:431)
[7] weblogic.cluster.replication.ReplicationManager.register(ReplicationManager.java:618)
[8] weblogic.servlet.internal.session.ReplicatedSession.<init>(ReplicatedSession.java:100)
[9] weblogic.servlet.internal.session.ReplicatedSessionContext.getNewSession(ReplicatedSessionContext.java:46)
[10] weblogic.servlet.internal.ServletRequestImpl.getNewSession(ServletRequestImpl.java:1094)
[11] weblogic.servlet.internal.ServletRequestImpl.getSession(ServletRequestImpl.java:988)
[12] com.rhii.at.controller.utilities.SessionDataBroker.getMutableProfile(SessionDataBroker.java)
[13] com.rhii.at.controller.utilities.LOBUtilities.setupLOBs(LOBUtilities.java)
[14] com.rhii.at.controller.Home.service(Home.java)
[15] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[16] com.wily.introscope.agent.probe.servlet.ManagedHttpServlet.service(ManagedHttpServlet.java:88)
[17] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[18] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[19] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:246)
[20] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[21] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[22] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-126" (TID:0xad9d24, sys_thread_t:0xad9c68, state:CW,
thread_t: t@139, threadID:0xd8ba1dc8, stack_bottom:0xd8ba2000,
stack_size:0x20000) prio=5
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3] weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4] weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5] weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:147)
[6] weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:698)
[7] weblogic.servlet.internal.session.ReplicatedSession.sync(ReplicatedSession.java:412)
[8] weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:52)
[9] weblogic.servlet.internal.ServletRequestImpl.syncSession(ServletRequestImpl.java:1087)
[10] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[11] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[12] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:246)
[13] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[14] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[15] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-125" (TID:0xad896c, sys_thread_t:0xad88b0, state:CW,
thread_t: t@138, threadID:0xd8bd1dc8, stack_bottom:0xd8bd2000,
stack_size:0x20000) prio=5
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3] weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4] weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5] weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:147)
[6] weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:698)
[7] weblogic.servlet.internal.session.ReplicatedSession.sync(ReplicatedSession.java:412)
[8] weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:52)
[9] weblogic.servlet.internal.ServletRequestImpl.syncSession(ServletRequestImpl.java:1087)
[10] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[11] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[12] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:246)
[13] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[14] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[15] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-124" (TID:0xad75b4, sys_thread_t:0xad74f8, state:CW,
thread_t: t@137, threadID:0xd8c01dc8, stack_bottom:0xd8c02000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-123" (TID:0xad63e4, sys_thread_t:0xad6328, state:CW,
thread_t: t@136, threadID:0xd8c31dc8, stack_bottom:0xd8c32000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-122" (TID:0xad500c, sys_thread_t:0xad4f50, state:CW,
thread_t: t@135, threadID:0xd8c61dc8, stack_bottom:0xd8c62000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-121" (TID:0xad3c54, sys_thread_t:0xad3b98, state:CW,
thread_t: t@134, threadID:0xd8c91dc8, stack_bottom:0xd8c92000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-120" (TID:0xad289c, sys_thread_t:0xad27e0, state:R,
thread_t: t@133, threadID:0xd8cc1dc8, stack_bottom:0xd8cc2000,
stack_size:0x20000) prio=5
[1] java.net.SocketOutputStream.socketWrite(Native Method)
[2] java.net.SocketOutputStream.write(SocketOutputStream.java:87)
[3] weblogic.servlet.internal.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:155)
[4] weblogic.servlet.internal.ServletOutputStreamImpl.finish(ServletOutputStreamImpl.java:268)
[5] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[6] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[7] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-119" (TID:0xad14e4, sys_thread_t:0xad1428, state:CW,
thread_t: t@132, threadID:0xd8cf1dc8, stack_bottom:0xd8cf2000,
stack_size:0x20000) prio=5
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3] weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4] weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5] weblogic.cluster.replication.ReplicationManager_WLStub.create(ReplicationManager_WLStub.java:77)
[6] weblogic.cluster.replication.ReplicationManager.createSecondary(ReplicationManager.java:431)
[7] weblogic.cluster.replication.ReplicationManager.register(ReplicationManager.java:618)
[8] weblogic.servlet.internal.session.ReplicatedSession.<init>(ReplicatedSession.java:100)
[9] weblogic.servlet.internal.session.ReplicatedSessionContext.getNewSession(ReplicatedSessionContext.java:46)
[10] weblogic.servlet.internal.ServletRequestImpl.getNewSession(ServletRequestImpl.java:1094)
[11] weblogic.servlet.internal.ServletRequestImpl.getSession(ServletRequestImpl.java:988)
[12] com.rhii.at.controller.utilities.SessionDataBroker.getMutableProfile(SessionDataBroker.java)
[13] com.rhii.at.controller.utilities.LOBUtilities.setupLOBs(LOBUtilities.java)
[14] com.rhii.at.controller.AdvancedSearch.service(AdvancedSearch.java)
[15] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[16] com.wily.introscope.agent.probe.servlet.ManagedHttpServlet.service(ManagedHttpServlet.java:88)
[17] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[18] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[19] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:246)
[20] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[21] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[22] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-118" (TID:0xad0314, sys_thread_t:0xad0258, state:CW,
thread_t: t@131, threadID:0xd8d21dc8, stack_bottom:0xd8d22000,
stack_size:0x20000) prio=5
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3] weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4] weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5] weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:147)
[6] weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:698)
[7] weblogic.servlet.internal.session.ReplicatedSession.sync(ReplicatedSession.java:412)
[8] weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:52)
[9] weblogic.servlet.internal.ServletRequestImpl.syncSession(ServletRequestImpl.java:1087)
[10] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[11] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[12] weblogic.servlet.JSPServlet.service(JSPServlet.java:78)
[13] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[14] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[15] weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:69)
[16] com.rhii.at.controller.utilities.LOBUtilities.invokeJsp(LOBUtilities.java)
[17] com.rhii.at.controller.SearchResults.service(SearchResults.java)
[18] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[19] com.wily.introscope.agent.probe.servlet.ManagedHttpServlet.service(ManagedHttpServlet.java:88)
[20] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[21] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[22] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:246)
[23] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[24] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[25] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-117" (TID:0xacef5c, sys_thread_t:0xaceea0, state:CW,
thread_t: t@130, threadID:0xd8d51dc8, stack_bottom:0xd8d52000,
stack_size:0x20000) prio=5
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3] weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4] weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5] weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:147)
[6] weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:698)
[7] weblogic.servlet.internal.session.ReplicatedSession.sync(ReplicatedSession.java:412)
[8] weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:52)
[9] weblogic.servlet.internal.ServletRequestImpl.syncSession(ServletRequestImpl.java:1087)
[10] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[11] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[12] weblogic.servlet.JSPServlet.service(JSPServlet.java:78)
[13] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[14] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[15] weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:69)
[16] com.rhii.at.controller.utilities.LOBUtilities.invokeJsp(LOBUtilities.java)
[17] com.rhii.at.controller.SkillInventory.service(SkillInventory.java)
[18] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[19] com.wily.introscope.agent.probe.servlet.ManagedHttpServlet.service(ManagedHttpServlet.java:88)
[20] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[21] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[22] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:246)
[23] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[24] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[25] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-116" (TID:0xacdba4, sys_thread_t:0xacdae8, state:CW,
thread_t: t@129, threadID:0xd8d81dc8, stack_bottom:0xd8d82000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
.
.
.
"ExecuteThread-8" (TID:0x9f7504, sys_thread_t:0x9f7448, state:CW,
thread_t: t@21, threadID:0xda1c1dc8, stack_bottom:0xda1c2000,
stack_size:0x20000) prio=5
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3] weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4] weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5] weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:147)
[6] weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:698)
[7] weblogic.servlet.internal.session.ReplicatedSession.sync(ReplicatedSession.java:412)
[8] weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:52)
[9] weblogic.servlet.internal.ServletRequestImpl.syncSession(ServletRequestImpl.java:1087)
[10] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[11] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[12] weblogic.servlet.JSPServlet.service(JSPServlet.java:78)
[13] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[14] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[15] weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:69)
[16] com.rhii.at.controller.utilities.LOBUtilities.invokeJsp(LOBUtilities.java)
[17] com.rhii.at.controller.SkillInventory.service(SkillInventory.java)
[18] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[19] com.wily.introscope.agent.probe.servlet.ManagedHttpServlet.service(ManagedHttpServlet.java:88)
[20] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[21] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[22] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:246)
[23] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[24] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[25] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-7" (TID:0x9c25d4, sys_thread_t:0x9c2518, state:CW,
thread_t: t@20, threadID:0xda1f1dc8, stack_bottom:0xda1f2000,
stack_size:0x20000) prio=5
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3] weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4] weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5] weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:147)
[6] weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:698)
[7] weblogic.servlet.internal.session.ReplicatedSession.sync(ReplicatedSession.java:412)
[8] weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:52)
[9] weblogic.servlet.internal.ServletRequestImpl.syncSession(ServletRequestImpl.java:1087)
[10] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[11] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[12] weblogic.servlet.JSPServlet.service(JSPServlet.java:78)
[13] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[14] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[15] weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:69)
[16] com.rhii.at.controller.utilities.LOBUtilities.invokeJsp(LOBUtilities.java)
[17] com.rhii.at.controller.Search.ControlRedirect(Search.java)
[18] com.rhii.at.controller.Search.service(Search.java)
[19] javax.servlet.http.HttpServlet.service(HttpServlet.java:835)
[20] com.wily.introscope.agent.probe.servlet.ManagedHttpServlet.service(ManagedHttpServlet.java:88)
[21] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[22] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[23] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:246)
[24] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[25] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[26] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-6" (TID:0xa05114, sys_thread_t:0xa05058, state:CW,
thread_t: t@19, threadID:0xda4f1dc8, stack_bottom:0xda4f2000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-5" (TID:0xa0a52c, sys_thread_t:0xa0a470, state:CW,
thread_t: t@18, threadID:0xda5c1dc8, stack_bottom:0xda5c2000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-4" (TID:0xa3a55c, sys_thread_t:0xa3a4a0, state:CW,
thread_t: t@17, threadID:0xda5f1dc8, stack_bottom:0xda5f2000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-3" (TID:0xa4515c, sys_thread_t:0xa450a0, state:R,
thread_t: t@16, threadID:0xdae61dc8, stack_bottom:0xdae62000,
stack_size:0x20000) prio=5
[1] java.net.SocketOutputStream.socketWrite(Native Method)
[2] java.net.SocketOutputStream.write(SocketOutputStream.java:87)
[3] weblogic.servlet.internal.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:155)
[4] weblogic.servlet.internal.ServletOutputStreamImpl.finish(ServletOutputStreamImpl.java:268)
[5] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[6] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[7] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-2" (TID:0x993944, sys_thread_t:0x993888, state:CW,
thread_t: t@15, threadID:0xdae91dc8, stack_bottom:0xdae92000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-1" (TID:0x9cd144, sys_thread_t:0x9cd088, state:CW,
thread_t: t@14, threadID:0xdaec1dc8, stack_bottom:0xdaec2000,
stack_size:0x20000) prio=5
[1] weblogic.common.internal.DynaQueue.getW(DynaQueue.java:219)
[2] weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:212)
[3] weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:19)
[4] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"ExecuteThread-0" (TID:0xa3aea4, sys_thread_t:0xa3ade8, state:CW,
thread_t: t@13, threadID:0xdaef1dc8, stack_bottom:0xdaef2000,
stack_size:0x20000) prio=5
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3] weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4] weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5] weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:147)
[6] weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:698)
[7] weblogic.servlet.internal.session.ReplicatedSession.sync(ReplicatedSession.java:412)
[8] weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:52)
[9] weblogic.servlet.internal.ServletRequestImpl.syncSession(ServletRequestImpl.java:1087)
[10] weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:84)
[11] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:261)
[12] weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:246)
[13] weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:369)
[14] weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:264)
[15] weblogic.t3.srvr.ExecuteThread.run(ExecuteThread.java:96)
-------------------
"TimeEventGenerator" (TID:0xa1f69c, sys_thread_t:0xa1f5e0, state:CW,
thread_t: t@12, threadID:0xdb0b1dc8, stack_bottom:0xdb0b2000,
stack_size:0x20000) prio=5
[1] weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:256)
[2] weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:143)
[3] java.lang.Thread.run(Thread.java:485)
-------------------
"SpinnerRandomSource" (TID:0xa07aa4, sys_thread_t:0xa079e8, state:CW,
thread_t: t@11, threadID:0xdbe61dc8, stack_bottom:0xdbe62000,
stack_size:0x20000) prio=5
[1] java.lang.Object.wait(Object.java:424)
[2] weblogic.security.SpinnerThread.stopSpinning(SpinnerRandomBitsSource.java:100)
[3] weblogic.security.SpinnerThread.run(SpinnerRandomBitsSource.java:114)
-------------------
"Finalizer" (TID:0x454604, sys_thread_t:0x454548, state:CW, thread_t:
t@6, threadID:0xfec51dc8, stack_bottom:0xfec52000, stack_size:0x20000)
prio=8
[1] java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:106)
[2] java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:128)
[3] java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:175)
-------------------
"Reference Handler" (TID:0x274ec, sys_thread_t:0x27430, state:CW,
thread_t: t@5, threadID:0xfec81dc8, stack_bottom:0xfec82000,
stack_size:0x20000) prio=10
[1] java.lang.Object.wait(Object.java:424)
[2] java.lang.ref.Reference$ReferenceHandler.run(Reference.java:107)
-------------------
"Signal dispatcher" (TID:0x42ddec, sys_thread_t:0x42dd30, state:MW,
thread_t: t@4, threadID:0xfecc1dc8, stack_bottom:0xfecc2000,
stack_size:0x20000) prio=10
-------------------
"main" (TID:0x3808c, sys_thread_t:0x37fd0, state:CW, thread_t: t@1,
threadID:0x25130, stack_bottom:0xffbf0000, stack_size:0x20000) prio=5
[1] java.lang.Object.wait(Object.java:424)
[2] weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:1697)
[3] java.lang.reflect.Method.invoke(Native Method)
[4] weblogic.Server.startServerDynamically(Server.java:103)
[5] weblogic.Server.main(Server.java:65)
[6] weblogic.Server.main(Server.java:55)
-------------------
Registered lock dump:
mutexes:
Heap lock locked
global roots lock locked
EE table lock locked
Lock registry unlocked
JNI global ref. lock unlocked
vtableLock unlocked
utf8 hash table locked
GC/utf8 hash table unlocked
string intern table locked
class loader constraint locked
GC/class loader constraint unlocked
Class unloading disable-lock unlocked
jniWeakRef list locked
GC/jniWeakRef list unlocked
constant pool resolution unlocked
depLock locked
GC/depLock unlocked
gc traps lock unlocked
zone lock locked
GC/zone lock unlocked
prestubs locked
GC/prestubs unlocked
codeCollectMut locked
GC/codeCollectMut unlocked
Code Access lock locked
GC/Code Access lock unlocked
Thread queue lock locked
Thread creation lock locked
recursive mutexes:
Global inconsistent lock 1 lock(s) by <GC-like-Thread>
BinClass lock 1 lock(s) by <GC-like-Thread>
JBE compiler lock unlocked
Thanks for the thread dump. I'll assume the parts you deleted were similar to the parts you left in.
I have a few thoughts from looking at the thread dump. Some of this is guesswork since I don't know your
application very well.
Looks like you are running 128 execute threads, and not using nativeIO. Most of your executeThreads are stuck in
the following stack trace:
[1] weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:32)
[2] weblogic.rjvm.ResponseImpl.getThrowable(ResponseImpl.java:45)
[3]weblogic.rmi.extensions.BasicResponse.getThrowable(BasicResponse.java:10)
[4]weblogic.rmi.extensions.BasicRequest.sendReceive(BasicRequest.java:41)
[5]weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:147)
[6]weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:698)
[7]weblogic.servlet.internal.session.ReplicatedSession.sync(ReplicatedSession.java:412)
[8]weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:52)
etc etc
So it looks like you are running in a cluster doing in-memory session replication. The server is stuck trying to
replicate session information. Is this correct - you are running in a cluster? Is your cluster working
correctly? Can you confirm that each cluster member is aware of the other (ie multicast is working properly).
What are you using as a proxy?
If you are not running in a cluster then you should search your startup scripts and weblogic.properties files for
the following cluster properties and either comment them out or delete them:
weblogic.cluster.enable
weblogic.cluster.multicastAddress
weblogic.cluster.name
weblogic.httpd.clustering.enable
weblogic.httpd.session.persistence
weblogic.httpd.session.persistentStoreType=replicated
If you are not running a cluster then this may be the cause of your hanging problems as well as the cause of your
broken pipe exceptions.
A few other suggestions:
1. Use nativeIO (weblogic.system.nativeIO.enable=true). This will free up execute threads for handling client
requests. This will reduce (but not eliminate) the problem you are having.
2. Be aware that WLS 4.5.x is very near the end of its life - BEA will not support this version as of June this
year. You may want to consider upgrading in the near future.
Let me know your thoughts on this.
Regards,
Michael
Prakash wrote:
--
Thanks for the reply. Here are some more answers:
Yes we are using clusters. We have 2 application servers running and
both are clustered. Just for your heads-up we are getting the
following exception in our log frequently. I am not sure this will
help you to analyse the issue:
Tue Mar 05 15:22:05 PST 2002:<E> <HTTP> Servlet request terminiated
with Error:
Tue Mar 05 15:22:05 PST 2002:<E> <HTTP>
weblogic.cluster.replication.NotFoundExc
eption: unregister unable to find object 858962234784479277
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Compiled Code)
at java.lang.Throwable.<init>(Compiled Code)
at java.lang.Exception.<init>(Compiled Code)
at weblogic.cluster.replication.NotFoundException.<init>(Compiled
Code)
at weblogic.cluster.replication.ReplicationManager.find(Compiled
Code)
at weblogic.cluster.replication.ReplicationManager.unregister(Compiled
C
ode)at weblogic.servlet.internal.session.ReplicatedSession.invalidate(Compil
ed Code)
at com.rhii.at.controller.JobBoardLogin.service(Compiled Code)
at javax.servlet.http.HttpServlet.service(Compiled Code)
at com.wily.introscope.agent.probe.servlet.ManagedHttpServlet.service(Co
mpiled Code)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(Compiled
Code
)
at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled
C
ode)
at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled
C
ode)
at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled
Code)
at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
at weblogic.t3.srvr.ExecuteThread.run(Compiled Code)
--------------- nested within: ------------------
weblogic.utils.NestedError: Can't unregister an ROID that does not
exist
- with nested exception:
[weblogic.cluster.replication.NotFoundException: unregister unable to
find objec
t 858962234784479277]at java.lang.Throwable.fillInStackTrace(Native
Method)
at java.lang.Throwable.fillInStackTrace(Compiled Code)
at java.lang.Throwable.<init>(Compiled Code)
at java.lang.Error.<init>(Error.java:50)
at weblogic.utils.NestedError.<init>(NestedError.java:23)
at weblogic.servlet.internal.session.ReplicatedSession.invalidate(Compil
ed Code)
at com.rhii.at.controller.JobBoardLogin.service(Compiled Code)
at javax.servlet.http.HttpServlet.service(Compiled Code)
at com.wily.introscope.agent.probe.servlet.ManagedHttpServlet.service(Co
mpiled Code)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(Compiled
Code
)
at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled
C
ode)
at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled
C
ode)
at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled
Code)
at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
at weblogic.t3.srvr.ExecuteThread.run(Compiled Code)
Another thing, yes we will be upgrading our system very soon. Talks
are going on.
I will change the nativeIO parameter and see the impact.
Can you confirm me about the above exception. Is that mean our
multicast clustering is not working properly? What should we do?
Upgrading to sp15 (4.1.5) will improve something?
Thanks
V Prakash
Yes, this exception confirms that you are having issues with multicast. The server
cannot replicate or remove replicated objects on the secondary server.
I suspect using nativeIO will help but not solve the problems - it may delay the
hang but it won't eliminate it. I don't know that upgrading to sp15 will help.
There is only 1 multicast related change since sp13 - there's a new property to help
specify a separate interface NIC for multicast. This isn't really a fix so I don't
know that this will help. In any case, the issue is that multicast doesn't appear
to be working, and that issue is external to WebLogic Server.
We do have a simple multicast test utility that you might try using. See
http://www.weblogic.com/docs45/techstart/utils.html#multicasttest for details on
using this. You will want to run this test from both of your cluster machines at
the same time (wls should not be running while you do this). Be forwarned that this
test is simple and will not reveal all multicast problems, but it will indicate
whether your multicast address is working at all.
The best way to remedy this is to get your sysadmin/IT folks to step in and help set
you up a working multicast address.
Hope this helps,
Michael
Prakash wrote:
--
Thanks a lot. I will do the test immediately and make sure the multicast is working
fine. In the mean time, I am in the process of upgrading to sp15 and use nativeIO
as well.
I will keep you updated on this after I am done with the above things. Thank you
again, for your help.
V Prakash
I have server1 with ip 172.17.12.209 and server2 with ip 172.17.12.210
on server1' s console I see
Wed Mar 06 23:26:43 PST 2002:<I> <Cluster> Adding server
7371664068035509141S172.17.12.210:[7001,700
1,443,443,7001,-1] to cluster view
and on server2's console I see
Wed Mar 06 23:26:49 PST 2002:<I> <Cluster> Adding server
-2838835315604947354S172.17.12.209:[7001,70
01,443,443,7001,-1] to cluster view
Are you seeing similar messages when both servers starts up??
--
Kumar
Would it be possible to post thread dumps from both servers preferably 3-5?
I wanted to see what the other server is doing when this guy hangs.
Actually this kind of problems are best trouble shooted if you can work with support
--
Kumar
idem, our server 4.5.1 (SP15) stops responding.
sometimes with Broken Pipe.
Sometimes lots SessionBean process Rollback.
Sometimes nothing in log.
Pls take a thread dump and post it here.
Thanks.
Michael
Nicolas Bert wrote:
--