Hibernate exception under high load

[mn...@sumerge.com]

I am running Hibernate 3 on WAS 6.0. It works fine but under high load (30 concurrent users or above) the application sometimes throws org.hibernate.exception.GenericJDBCException: Cannot open connection.

{code}
[1/13/09 16:25:57:579 EET] 00000066 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet Faces Servlet. Exception thrown : javax.servlet.ServletException: org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:637)
at com.ibm._jsp._viewloans._jspService(_viewloans.java:309)
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled Code))
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:117)
at com.ibm.ws.jsp.webcontainerext.JSPExtensionServletWrapper.handleRequest(JSPExtensionServletWrapper.java:171)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java(Compiled Code))
at com.ibm.faces.context.MultipartExternalContextImpl.dispatch(MultipartExternalContextImpl.java(Compiled Code))
at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java(Compiled Code))
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java(Compiled Code))
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java(Compiled Code))
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java(Compiled Code))
at javax.faces.webapp.FacesServlet.service(FacesServlet.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled Code))
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled Code))
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled Code))
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled Code))
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminaters(NewConnectionInitialReadCallback.java:201)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:103)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
---- Begin backtrace for Nested Throwables
javax.faces.el.EvaluationException: org.hibernate.exception.GenericJDBCException: Cannot open connection
at com.sun.faces.el.MethodBindingImpl.invoke(MethodBindingImpl.java:131)
at com.ibm.faces.renderkit.html_extended.ScriptCollectorRenderer.encodeBegin(ScriptCollectorRenderer.java:62)
at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java(Compiled Code))
at javax.faces.webapp.UIComponentTag.encodeBegin(UIComponentTag.java(Inlined Compiled Code))
at javax.faces.webapp.UIComponentTag.doStartTag(UIComponentTag.java(Compiled Code))
at com.ibm.faces.taglib.html_extended.ScriptCollectorTag.doStartTag(ScriptCollectorTag.java:109)
at com.ibm._jsp._viewloans._jspx_meth_hx_scriptCollector_1(_viewloans.java:1958)
at com.ibm._jsp._viewloans._jspx_meth_hx_scriptCollector_0(_viewloans.java:2004)
at com.ibm._jsp._viewloans._jspx_meth_f_view_0(_viewloans.java:2033)
at com.ibm._jsp._viewloans._jspService(_viewloans.java:300)
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled Code))
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:117)
at com.ibm.ws.jsp.webcontainerext.JSPExtensionServletWrapper.handleRequest(JSPExtensionServletWrapper.java:171)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java(Compiled Code))
at com.ibm.faces.context.MultipartExternalContextImpl.dispatch(MultipartExternalContextImpl.java(Compiled Code))
at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java(Compiled Code))
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java(Compiled Code))
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java(Compiled Code))
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java(Compiled Code))
at javax.faces.webapp.FacesServlet.service(FacesServlet.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled Code))
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled Code))
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled Code))
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled Code))
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminaters(NewConnectionInitialReadCallback.java:201)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:103)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java(Compiled Code))
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java(Inlined Compiled Code))
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java(Compiled Code))
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java(Compiled Code))
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java(Compiled Code))
at com.sumerge.barclays.persistence.dao.LoanDao.searchLoans(LoanDao.java:811)
at pagecode.Viewloans.setQueueLoansList(Viewloans.java:206)
at pagecode.Viewloans.onPageLoadBegin(Viewloans.java:445)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.sun.faces.el.MethodBindingImpl.invoke(MethodBindingImpl.java:127)
... 37 more
Caused by: com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 272826
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1146)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))
at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java(Compiled Code))
... 51 more
Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 272826
at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java:1248)
at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java(Compiled Code))
... 54 more
{code}

[rang...@gmail.com]

Either increase the number of available connections on your database
or increase the number of connections within the datasource definition
or both (hard to know which without seeing the application).

Then verify your results with a load testing tool.

[Ken Hygh]

mn...@sumerge.com wrote:
> I am running Hibernate 3 on WAS 6.0. It works fine but under high load (30 concurrent users or above) the application sometimes throws org.hibernate.exception.GenericJDBCException: Cannot open connection.
>
> {code}
> [1/13/09 16:25:57:579 EET] 00000066 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet Faces Servlet. Exception thrown : javax.servlet.ServletException: org.hibernate.exception.GenericJDBCException: Cannot open connection

[snip]

> ... 51 more
> Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 272826
> at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java:1248)
> at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compiled Code))
> at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java(Compiled Code))
> at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java(Compiled Code))
> ... 54 more
> {code}
>

Looks like you may need to increase the number of connections in your
connection pool.

Ken