Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

prod env problem!

9 views
Skip to first unread message

HARPREET SINGH

unread,
Feb 7, 2005, 6:48:38 PM2/7/05
to
Hi,
We have Weblogic 8.1 on solaris platform and have a critical application one managed server.This application/server is hangs up quite often ,like 4,5 times in a week.There are 400 execute threads and 3 reader threads.Most of the time the execute threads remain idle ,at the time of problem too, and reader thread are busy.
I get following Exceptions in log files at different time of problems:
####<Feb 7, 2005 8:22:30 AM CST> <Info> <EJB> <lcy002194> <production> <ExecuteThread: '389' for queue: 'weblogic.kernel.Default'> <<anonymous>> <> <BEA-010049> <EJB Exception in method: remove: java.rmi.NoSuchObjectException: Bean has been deleted..
java.rmi.NoSuchObjectException: Bean has been deleted.
at weblogic.ejb20.swap.DiskSwap.read(DiskSwap.java:172)
at weblogic.ejb20.manager.StatefulSessionManager.getBean(StatefulSessionManager.java:326)
at weblogic.ejb20.manager.StatefulSessionManager.preInvoke(StatefulSessionManager.java:396)
at weblogic.ejb20.manager.StatefulSessionManager.remove(StatefulSessionManager.java:936)
at weblogic.ejb20.manager.ReplicatedStatefulSessionManager.remove(ReplicatedStatefulSessionManager.java:196)
at weblogic.ejb20.internal.StatefulEJBObject.remove(StatefulEJBObject.java:92)
at com.granite.middle.session.ClientServiceEJB_u5heex_EOImpl.remove(ClientServiceEJB_u5heex_EOImpl.java:4642)
at com.granite.middle.session.ClientServiceEJB_u5heex_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:466)
at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:409)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:404)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
>
####<Feb 7, 2005 8:23:38 AM CST> <Info> <EJB> <lcy002194> <production> <ExecuteThread: '389' for queue: 'weblogic.kernel.Default'> <<anonymous>> <BEA1-14E3B0CB38DA> <BEA-010051> <EJB Exception occurred during invocation from home: com.granite.middle.worxng.session.Queue_68j9s_HomeImpl@1ec56c9 threw exception: java.lang.NullPointerException
java.lang.NullPointerException
at com.granite.middle.worxng.session.QueueBean.getViewPrivilegeQueueList(QueueBean.java:1056)
at com.granite.middle.worxng.session.Queue_68j9s_EOImpl.getViewPrivilegeQueueList(Queue_68j9s_EOImpl.java:208)
at com.granite.middle.worxng.session.Queue_68j9s_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:466)
at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:409)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:404)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java


####<Feb 7, 2005 4:50:20 PM CST> <Info> <EJB> <lcy002194> <production> <ExecuteThread: '390' for queue: 'weblogic.kernel.Default'> <<anonymous>> <BEA1-0371ACF1523A94FFBE2B> <BEA-010051> <EJB Exception occurred during invocation from home: com.granite.middle.session.PathEJB_9q5d2l_HomeImpl@f5c440 threw exception: weblogic.utils.NestedError: Unexpected Exception: - with nested exception:
[java.security.PrivilegedActionException: java.sql.SQLException: Transaction rolled back: Transaction timed out after 599 seconds
Xid=BEA1-0371ACF1523A94FFBE2B(23844485),Status=Active,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=599,seconds left=60,activeThread=Thread[ExecuteThread: '390' for queue: 'weblogic.kernel.Default',5,Thread Group for Queue: 'weblogic.kernel.Default'],XAServerResourceInfo[JMS_xngJMSFileStore]=(ServerResourceInfo[JMS_xngJMSFileStore]=(state=suspended,assigned=none),xar=JMS_xngJMSFileStore),XAServerResourceInfo[xngprod1_prod]=(ServerResourceInfo[xngprod1_prod]=(state=started,assigned=none),xar=xngprod1_prod),SCInfo[granite+production]=(state=active),local properties=({weblogic.jdbc.jta.xngprod1_prod=weblogic.jdbc.wrapper.TxInfo@d0c4da}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=production+10.174.2.194:7280+granite+t3+, XAResources={},NonXAResources={})],CoordinatorURL=production+10.174.2.194:7280+granite+t3+)]
java.security.PrivilegedActionException: java.sql.SQLException: Transaction rolled back: Transaction timed out after 599 seconds
Xid=BEA1-0371ACF1523A94FFBE2B(23844485),Status=Active,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=599,seconds left=60,activeThread=Thread[ExecuteThread: '390' for queue: 'weblogic.kernel.Default',5,Thread Group for Queue: 'weblogic.kernel.Default'],XAServerResourceInfo[JMS_xngJMSFileStore]=(ServerResourceInfo[JMS_xngJMSFileStore]=(state=suspended,assigned=none),xar=JMS_xngJMSFileStore),XAServerResourceInfo[xngprod1_prod]=(ServerResourceInfo[xngprod1_prod]=(state=started,assigned=none),xar=xngprod1_prod),SCInfo[granite+production]=(state=active),local properties=({weblogic.jdbc.jta.xngprod1_prod=weblogic.jdbc.wrapper.TxInfo@d0c4da}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=production+10.174.2.194:7280+granite+t3+, XAResources={},NonXAResources={})],CoordinatorURL=production+10.174.2.194:7280+granite+t3+)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:188)
at com.granite.middle.helper.jdbc.ConnectionJDBC.getNewConnection(ConnectionJDBC.java:843)
at com.granite.middle.helper.jdbc.ConnectionJDBC.getConnection(ConnectionJDBC.java:819)
at com.granite.middle.helper.jdbc.ProfilePrivilegeHelper.checkPrivilege(ProfilePrivilegeHelper.java:553)
at com.granite.middle.helper.jdbc.ProfilePrivilegeHelper.checkPrivilege(ProfilePrivilegeHelper.java:452)
at com.granite.middle.logic.PathLogic.getItem(PathLogic.java:429)
at com.granite.middle.session.PathBean.getItem(PathBean.java:834)
at com.granite.middle.session.PathBean.getItem(PathBean.java:809)
at com.granite.middle.session.PathEJB_9q5d2l_EOImpl.getItem(PathEJB_9q5d2l_EOImpl.java:2854)
at com.granite.bdl.protocol.IIOP.adapter.IIOPPathAdapter.load(IIOPPathAdapter.java:246)
at com.granite.bdl.protocol.IIOP.adapter.IIOPPathAdapter.loadTransport(IIOPPathAdapter.java:1700)
at com.granite.bdl.protocol.BDLAdapterImpl.load(BDLAdapterImpl.java:69)
at com.granite.bdl.protocol.PathImpl.load(PathImpl.java:4656)
at com.granite.bdl.protocol.PathImpl.loadLiveArchive(PathImpl.java:5004)
at com.granite.xng_events.DefaultXngEventSerializer.getEventElement(DefaultXngEventSerializer.java:442)
at com.granite.xng_events.DefaultXngEventSerializer.serialize(DefaultXngEventSerializer.java:92)
at com.granite.xng_events.XngEventMessageBean.onMessage(XngEventMessageBean.java:442)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:382)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:316)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:281)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2596)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2516)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
Caused by: java.sql.SQLException: Transaction rolled back: Transaction timed out after 599 seconds
Xid=BEA1-0371ACF1523A94FFBE2B(23844485),Status=Active,numRepliesOwedMe=0,numRepliesOwedOthers=0,second

Please provide me some ideas!
Thanks

Rob Woollen

unread,
Feb 7, 2005, 7:39:29 PM2/7/05
to
HARPREET SINGH wrote:
> Hi,
> We have Weblogic 8.1 on solaris platform and have a critical application one managed server.This application/server is hangs up quite often ,like 4,5 times in a week.There are 400 execute threads and 3 reader threads.Most of the time the execute threads remain idle ,at the time of problem too, and reader thread are busy.
> I get following Exceptions in log files at different time of problems:
> ####<Feb 7, 2005 8:22:30 AM CST> <Info> <EJB> <lcy002194> <production> <ExecuteThread: '389' for queue: 'weblogic.kernel.Default'> <<anonymous>> <> <BEA-010049> <EJB Exception in method: remove: java.rmi.NoSuchObjectException: Bean has been deleted..
> java.rmi.NoSuchObjectException: Bean has been deleted.
> at weblogic.ejb20.swap.DiskSwap.read(DiskSwap.java:172)
> at weblogic.ejb20.manager.StatefulSessionManager.getBean(StatefulSessionManager.java:326)
> at weblogic.ejb20.manager.StatefulSessionManager.preInvoke(StatefulSessionManager.java:396)
> at weblogic.ejb20.manager.StatefulSessionManager.remove(StatefulSessionManager.java:936)
> at weblogic.ejb20.manager.ReplicatedStatefulSessionManager.remove(ReplicatedStatefulSessionManager.java:196)
> at weblogic.ejb20.internal.StatefulEJBObject.remove(StatefulEJBObject.java:92)
> at com.granite.middle.session.ClientServiceEJB_u5heex_EOImpl.remove(ClientServiceEJB_u5heex_EOImpl.java:4642)
> at com.granite.middle.session.ClientServiceEJB_u5heex_EOImpl_WLSkel.invoke(Unknown Source)
> at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:466)
> at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
> at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:409)
> at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
> at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
> at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:404)
> at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
> at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
> at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

This is a remove of a stateful session bean which has already been
deleted. Should be harmless.

>
> ####<Feb 7, 2005 8:23:38 AM CST> <Info> <EJB> <lcy002194> <production> <ExecuteThread: '389' for queue: 'weblogic.kernel.Default'> <<anonymous>> <BEA1-14E3B0CB38DA> <BEA-010051> <EJB Exception occurred during invocation from home: com.granite.middle.worxng.session.Queue_68j9s_HomeImpl@1ec56c9 threw exception: java.lang.NullPointerException
> java.lang.NullPointerException
> at com.granite.middle.worxng.session.QueueBean.getViewPrivilegeQueueList(QueueBean.java:1056)
> at com.granite.middle.worxng.session.Queue_68j9s_EOImpl.getViewPrivilegeQueueList(Queue_68j9s_EOImpl.java:208)
> at com.granite.middle.worxng.session.Queue_68j9s_EOImpl_WLSkel.invoke(Unknown Source)
> at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:466)
> at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
> at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:409)
> at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
> at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
> at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:404)
> at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
> at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java
>
>

This is a NullPointerException in some com.granite.* code. That appears
to be a bug in that code.

This is a transaction which has timedout after 10 minutes. First off,
10 minutes is too long for your tx timeout. I would not recommend
increasing the timeout beyond the default (30 seconds).

I'd suggest monitoring the server and taking some thread dumps when you
notice the problems. I suspect something is hung.

-- Rob

0 new messages