Stuck Connection

[mahesh...@infosys.com]

Hi
We are running Websphere 6.0 on iSeries having Java/J2EE as our front end and DB2 on iSeries as our backend.
In a process of conducting load test, we injected 1000 concurrent users. Test went fine for 2.5 hours and then it broke suddenly with the error listed below. It is also important to mention that everything, CPU/memory was under control. WAS logs suggested that even connection pooling was not exhausted.

After the occurence of the error, even after stopping the load test, application server is not able to connect with Database server but all other pages which are not connecting with database are working fine.

We are able to successully replicate the error. Any pointer would be a great help. If any further information related to error is required, please revert.

Below error came, while trying to get connection.
Error Summary:
"SQL Exception Occured while getting the Connection java.sql.SQLException: The back-end resource is currently unavailable. Stuck connections have been detected."

Error Logs:
[9/10/07 10:06:34:674 EDT] 00000a92 SystemErr R java.sql.SQLException: The back-end resource is currently unavailable. Stuck connections have been detected.
at java.lang.Throwable.<init>(Throwable.java:194)
at java.lang.Exception.<init>(Exception.java:41)
at java.sql.SQLException.<init>(SQLException.java:80)
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1118)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:401)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:315)
at com.vbp.common.persistence.VbpDataSource.getConnection(VbpDataSource.java:124)
at com.vbp.portal.service.RegistrationService.performSSNCheck(RegistrationService.java:33)
at com.vbp.portal.action.PersonalInfoAction.execute(PersonalInfoAction.java:38)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:217)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1481)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:736)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:851)
at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:255)
at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:312)
at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:58)
at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:237)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1533)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1512)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:101)
at com.vbp.portal.util.PortalSessionTimeOutFilter.doFilter(PortalSessionTimeOutFilter.java:35)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:134)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:101)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:70)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:437)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:80)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1761)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:108)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:422)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:319)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:537)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:593)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:946)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1038)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1448)

[Ben_]

This is error almost unknown to Google and unknown to IBM Suport site.

You better open a PMR, I believe.

My first feeling is that the back-end database does not accept new
connections.

Isn't there a "caused by" clause further below in the stacktrace ?

[mahesh...@infosys.com]

Hi Ben
Thanks for your reply. There is a caused by clause in the stack trace which states:

Caused by: javax.resource.ResourceException: The back-end resource is currently unavailable. Stuck connections have been detected.

at java.lang.Throwable.<init>(Throwable.java:194)
at java.lang.Exception.<init>(Exception.java:41)

at javax.resource.ResourceException.<init>(ResourceException.java:49)
at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java:1678)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:842)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:580)

Also, I would like to highlight that database has stopped accepting any further connections only from the application server. If you connect database from outside then it is accepting the connection.

Rgds
Mahesh Mohta

[Andy Wilkinson]

There is information about stuck connections and how to configure their
detection in the InfoCenter:

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/udat_conpooladv.html

Andy