weblogic 5.1 - Unrelated exceptions in application

瀏覽次數：1 次

跳到第一則未讀訊息

Haider Kazmi

未讀,

2004年2月10日晚上10:08:232004/2/10

收件者：

I have been trying to figure this problem for about an hour now. Apparently
it is only happening on heavy load, cannot reproduce it in our QA
environment

For last 2 weeks webloigc 5.1 sp9 on Sun Jdk 1.3, Solaris 5.8 (our live
platform) starts acting strange after being run for a couple of hours.
Interestingly no one has made changes to it, and the problem does not look
related to a software patch in anycase, might even be a JVM bug.

Basically weblogic starts throwing Exceptions all over the code, mostly they
are null pointer exceptions except where we check for a null and throw an
alternate exception. Its almost as if the reference table in the JVM decides
that everything is null.

Following is a bit of the log just before it crash, if anyone has hints as
to what might be going on or has had similar problems please let me know.
Thanks

part of Log follows:
java.lang.NullPointerException
at jsp_servlet._prod50._comn.__login._jspService(__login.java:277)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:120)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:922)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:886)
at
weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
Manager.java:269)
at
weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:380)
at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:268)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)
java.lang.NullPointerException
at jsp_servlet._prod50._comn.__login._jspService(__login.java:277)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:120)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:922)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:886)
at
weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
Manager.java:269)
at
weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:380)
at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:268)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:83)
at weblogic.servlet.internal.ChunkUtils.writeChunks(ChunkUtils.java:114)
at
weblogic.servlet.internal.ServletOutputStreamImpl.flush(ServletOutputStreamI
mpl.java:125)
at
weblogic.servlet.internal.ServletOutputStreamImpl.flushWithCheck(ServletOutp
utStreamImpl.java:343)
at
weblogic.servlet.internal.ServletOutputStreamImpl.checkForFlush(ServletOutpu
tStreamImpl.java:484)
at
weblogic.servlet.internal.ServletOutputStreamImpl.print(ServletOutputStreamI
mpl.java:197)
at weblogic.servlet.jsp.JspWriterImpl.print(JspWriterImpl.java:140)
at jsp_servlet._prod50._loca.__loca._jspService(__loca.java:412)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:120)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:922)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:886)
at
weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
Manager.java:269)
at
weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:380)
at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:268)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)
java.lang.NullPointerException
at jsp_servlet._prod50._comn.__login._jspService(__login.java:371)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:120)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:922)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:886)
at
weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
Manager.java:269)
at
weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:380)
at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:268)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)
java.lang.NullPointerException
at jsp_servlet._prod50._comn.__login._jspService(__login.java:371)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:120)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:922)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:886)
at
weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
Manager.java:269)
at
weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:380)
at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:268)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)
Login of user with roleid:3000, prod_USER
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:83)
at weblogic.servlet.internal.ChunkUtils.writeChunks(ChunkUtils.java:114)
at
weblogic.servlet.internal.ServletOutputStreamImpl.flush(ServletOutputStreamI
mpl.java:125)
at
weblogic.servlet.internal.ServletOutputStreamImpl.checkForFlush(ServletOutpu
tStreamImpl.java:488)
at
weblogic.servlet.internal.ServletOutputStreamImpl.print(ServletOutputStreamI
mpl.java:197)
at weblogic.servlet.jsp.JspWriterImpl.print(JspWriterImpl.java:140)
at jsp_servlet._prod50._comn.__myprofile._jspService(__myprofile.java:422)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:120)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:922)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:886)
at
weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
Manager.java:269)
at
weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:380)
at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:268)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)
Mon Feb 09 15:07:51 EST 2004:<I> <EJB JAR deployment
/prod/jars/reuse/ejb_RUNavigationS.jar> Exception in non-transactional EJB
invoke:
weblogic.ejb.extensions.LockTimedOutException: Lock for primaryKey:121_9:1
timed out after 300000 ms.
at
weblogic.ejb.internal.LockManagerImpl.waitForLock(LockManagerImpl.java:53)
at weblogic.ejb.internal.LockManagerImpl.lock(LockManagerImpl.java:146)
at weblogic.ejb.internal.LockManagerImpl.lock(LockManagerImpl.java:81)
at weblogic.ejb.internal.StatefulEJBCache.bind(StatefulEJBCache.java:456)
at
weblogic.ejb.internal.StatefulEJBObject.getContextForInvoke(StatefulEJBObjec
t.java:162)
at weblogic.ejb.internal.BaseEJBObject.preInvoke(BaseEJBObject.java:476)
at
com.prod.ejb.domain.runavigation.impl.RUNavigationDBeanEOImpl.getNavigation(
RUNavigationDBeanEOImpl.java:149)
at
com.prod.ejb.service.runavigation.impl.RUNavigationSBean.getNavigation(RUNav
igationSBean.java:123)
at
com.prod.ejb.service.runavigation.impl.RUNavigationSBeanEOImpl.getNavigation
(RUNavigationSBeanEOImpl.java:187)
at
com.prod.ejb.service.runavigation.impl.RUNavigationSBeanEOImpl_ServiceStub.g
etNavigation(RUNavigationSBeanEOImpl_ServiceStub.java:157)
at com.prod.applications.reuse.ui.Nav.populateCache(Nav.java:398)
at com.prod.applications.reuse.ui.Nav.<init>(Nav.java:321)
at com.prod.applications.reuse.ui.Nav.<init>(Nav.java:285)
at com.prod.applications.site.ui.SiteUI.<init>(SiteUI.java:26)
at jsp_servlet._prod50._mktp.__site._jspService(__site.java:191)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:120)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:922)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:886)
at
weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
Manager.java:269)
at
weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:380)
at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:268)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)

com.prod.applications.reuse.ui.UIException: Exception on Client Id(121)
at com.prod.applications.reuse.ui.Nav.populateCache(Nav.java:412)
at com.prod.applications.reuse.ui.Nav.<init>(Nav.java:321)
at com.prod.applications.reuse.ui.Nav.<init>(Nav.java:285)
at com.prod.applications.site.ui.SiteUI.<init>(SiteUI.java:26)
at jsp_servlet._prod50._mktp.__site._jspService(__site.java:191)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:120)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:922)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:886)
at
weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
Manager.java:269)
at
weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:380)
at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:268)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)
com.prod.applications.reuse.ui.UIException: Exception on Client Id(121)
at com.prod.applications.reuse.ui.Nav.populateCache(Nav.java:412)
at com.prod.applications.reuse.ui.Nav.<init>(Nav.java:321)
at com.prod.applications.reuse.ui.Nav.<init>(Nav.java:285)
at com.prod.applications.site.ui.SiteUI.<init>(SiteUI.java:26)
at jsp_servlet._prod50._mktp.__site._jspService(__site.java:191)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:120)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:922)
at
weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
l.java:886)
at
weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
Manager.java:269)
at
weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:380)
at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:268)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)
calling _checkReloadConfig at iteration 2030
Terminated

Rob Woollen

未讀,

2004年2月11日下午1:42:452004/2/11

收件者：

Looks like a NullPointerException in your jsp page. It's very hard for
us to tell you anything more without more information.

-- Rob

Haider Kazmi

未讀,

2004年2月13日上午10:11:082004/2/13

收件者：

It turns out that there was an OutOfMemoryException when this all starts. I
had missed this as it was a single line Exception with no stack trace in a
15MB log file :(

We had our site load increase 5 folds in a week. All the problem starts with
the OutOfMemoryException. Increasing the VM heap size to 1GB have fixed the
issue.

To me it looks as if the GC is removing valid references if there is a
OutOfMemoryException, or maybe even something in WL51 that tries to recover
from an out of memory state and decides to release referenced objects in its
cache.

Thanks Rob

"Rob Woollen" <rwoo...@please.noemail.com> wrote in message
news:402a...@newsgroups.bea.com...

0 則新訊息