Database access error

[nsh...@wsscwater.com]

Hi all,

I got see this exception in SystemOut.log. My connection pool min/max is 10/50 for all datasources. Would you have any leads? I'm running WP 6.0.1.1 on Oracle 10.2.0.2?

[1/16/08 21:04:11:493 EST] 00000076 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/communityDS.
[1/16/08 21:04:11:504 EST] 00000076 DataStoreCont E com.ibm.wps.datastore.impl.DataStoreContext handleException EJPDB0002E: Error occurred during database access.
com.ibm.wps.util.DataBackendException: EJPDB0002E: Error occurred during database access.
at com.ibm.wps.datastore.impl.DataStoreContext.handleException(DataStoreContext.java:348)
at com.ibm.wps.datastore.impl.ResourcePersister.findInternal(ResourcePersister.java(Compiled Code))
at com.ibm.wps.datastore.impl.PortletDescriptorPersister.findInternal(PortletDescriptorPersister.java(Compiled Code))
at com.ibm.wps.datastore.impl.ResourcePersister.findInternal(ResourcePersister.java(Inlined Compiled Code))
at com.ibm.wps.datastore.impl.PortletDescriptorPersister.findAll(PortletDescriptorPersister.java(Compiled Code))
at com.ibm.wps.datastore.impl.ResourceHomeImpl.findAll(ResourceHomeImpl.java(Compiled Code))
at com.ibm.wps.datastore.impl.federation.ResourceHomeImpl.findAll(ResourceHomeImpl.java(Compiled Code))
at com.ibm.wps.services.registry.ReloadableBucket.reloadInternal(ReloadableBucket.java(Compiled Code))
at com.ibm.wps.services.registry.ReloadableBucket.reload(ReloadableBucket.java(Compiled Code))
at com.ibm.wps.services.registry.RegistryServiceImpl.reload(RegistryServiceImpl.java(Compiled Code))
at com.ibm.wps.services.registry.RegistryServiceImpl.fired(RegistryServiceImpl.java(Compiled Code))
at com.ibm.wps.services.workmanager.impl.WasAlarmListenerWrapper.fired(WasAlarmListenerWrapper.java(Compiled Code))
at com.ibm.ws.asynchbeans.AlarmImpl.callListenerMethod(AlarmImpl.java(Compiled Code))
at com.ibm.ws.asynchbeans.timer.GenericTimer.run(GenericTimer.java(Compiled Code))
at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java(Compiled Code))
at java.security.AccessController.doPrivileged1(Native Method)
at java.security.AccessController.doPrivileged(AccessController.java(Compiled Code))
at javax.security.auth.Subject.doAs(Subject.java(Compiled Code))
at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java(Compiled Code))
at com.ibm.ws.asynchbeans.J2EEContext$DoAsProxy.run(J2EEContext.java(Compiled Code))
at java.security.AccessController.doPrivileged1(Native Method)
at java.security.AccessController.doPrivileged(AccessController.java(Compiled Code))
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java(Compiled Code))
at com.ibm.ws.asynchbeans.AlarmImpl.runListenerAsCJWork(AlarmImpl.java(Compiled Code))
at com.ibm.ws.asynchbeans.am._Alarm.fireAlarm(_Alarm.java(Compiled Code))
at com.ibm.ws.asynchbeans.am._Alarm.run(_Alarm.java(Compiled Code))
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
Caused by: com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180024
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1146)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))
at com.ibm.wps.services.datastore.DomainDataStoreImpl.getConnectionInternal(DomainDataStoreImpl.java(Compiled Code))
at com.ibm.wps.services.datastore.DomainDataStoreImpl.getConnection(DomainDataStoreImpl.java(Compiled Code))
at com.ibm.wps.datastore.impl.DataStoreContext.init(DataStoreContext.java(Compiled Code))
... 26 more

[Jim Barnes]

need to see the rest of the stack trace
Jim
<br>
IBM Certified System Administrator -- WebSphere Portal V6.0, V5.1, V5.0<br>
IBM Certified Solution Developer -- WebSphere Portal V5.1, v6.0<br>
<br>
The postings on this site are my own and do not necessarily represent the positions, strategies, or opinions of IBM

[nsh...@wsscwater.com]

Here is the rest....doesn't give us anything more really...

[1/16/08 21:04:11:511 EST] 00000076 ReloadableBuc E com.ibm.wps.services.registry.ReloadableBucket reloadInternal() EJPEJ0004E: An unexpected exception occurred.

[1/16/08 21:07:11:657 EST] 00000076 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/communityDS.
[1/16/08 21:07:11:662 EST] 00000076 DataStoreCont E com.ibm.wps.datastore.impl.DataStoreContext handleException EJPDB0002E: Error occurred during database access.

com.ibm.wps.util.DataBackendException: EJPDB0002E: Error occurred during database access.
at com.ibm.wps.datastore.impl.DataStoreContext.handleException(DataStoreContext.java:348)
at com.ibm.wps.datastore.impl.ResourcePersister.findInternal(ResourcePersister.java(Compiled Code))

at com.ibm.wps.datastore.impl.ApplicationDescriptorPersister.findInternal(ApplicationDescriptorPersister.java(Compiled Code))
at com.ibm.wps.datastore.impl.ResourcePersister.findInternal(ResourcePersister.java(Inlined Compiled Code))
at com.ibm.wps.datastore.impl.ApplicationDescriptorPersister.findAll(ApplicationDescriptorPersister.java(Compiled Code))

[1/16/08 21:07:11:670 EST] 00000076 ReloadableBuc E com.ibm.wps.services.registry.ReloadableBucket reloadInternal() EJPEJ0004E: An unexpected exception occurred.

com.ibm.wps.util.DataBackendException: EJPDB0002E: Error occurred during database access.
at com.ibm.wps.datastore.impl.DataStoreContext.handleException(DataStoreContext.java:348)
at com.ibm.wps.datastore.impl.ResourcePersister.findInternal(ResourcePersister.java(Compiled Code))

at com.ibm.wps.datastore.impl.ApplicationDescriptorPersister.findInternal(ApplicationDescriptorPersister.java(Compiled Code))
at com.ibm.wps.datastore.impl.ResourcePersister.findInternal(ResourcePersister.java(Inlined Compiled Code))
at com.ibm.wps.datastore.impl.ApplicationDescriptorPersister.findAll(ApplicationDescriptorPersister.java(Compiled Code))

[Jim Barnes]

Well shows that you are running out of connections, not sure if there is a fix with it, but to alleviate it you may up the number max connections.
a thread dump during this time would be able to see what is locking these connections. Might want to open a PMR on this

[nsh...@wsscwater.com]

Thanks Jim. Going per the tuning guide, 50 should be good. Infact we have bumped up the number of allowable processes to 450 on the DB instance that WP points to. Are these numbers good? I will open a PMR as well.

[Jim Barnes]

Agreed it should be more than sufficient at 50, I am just wondering what in the environment is holding them up, Increasing the number can show if it is just some slow running threads, or if it is that the portal code is in a deadlock for some reason.(in that it uses them all up no matter what the value is.
<p>
It sounds like a code defect at this time, but without the thread dump it is hard to say where that is<br>
Jim

[hellg...@gmail.com]

Dear Jim,

we have WPS 6.0.1.5 on WAS ND 6.0.2.29 and we got problem :
[2/16/09 10:59:52:769 CET] 00000068 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/jcrdbORADS.
[2/16/09 10:59:52:993 CET] 00000068 ConnectionSer E com.ibm.icm.ci.ConnectionService ConnectionService.getConnection com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 90002
com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, timed out waiting for 90002

We got this error during normal usage of WCM.

I attached my Out, Err, Trace and Collector.

can U help me, please.

Fabrizio.

[David Yang]

Hello,
Please remember to follow technote *Recommended ratio of data source connection pool size to WebContainer thread pool size for WebSphere Portal 6.0* at http://www.ibm.com/support/docview.wss?rs=688&uid=swg21319394.

If the problem persists, then create a new PMR for this problem.

Thank you,
David Yang
WebSphere Portal support

The postings on this site are my own and do not necessarily represent the positions, strategies, or opinions of IBM.