[Dspace-tech] java.util.NoSuchElementException: Timeout waiting for idle object

[Mark H. Wood]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

We're getting about 1600 of these a day on our production system, which
makes reading my mail a bit tedious. Is there yet any more permanent
advice than "bounce the server and it'll be okay for an hour or so"?
I've restarted any number of times in the last few days but the problem
keeps coming back. The box is running 1.2.1 on Sun JRE 1.5.0-01 with
PGSQL 7.3.2 atop Red Hat Linux 9.0.

I've raised the connection pool from 30 to 200 (and I did check:
PostgreSQL is set for 256 concurrent connections so we should still have
plenty of headroom). The number of database service processes is never
seen to rise above about 15.

I know we're backlevel, but it'll take a while to coordinate upgrades
since this is the production box.

Where should I look for clues?

- --
Mark H. Wood, Lead System Programmer mw...@IUPUI.Edu
Open-source executable: $0.00. Source: $0.00 Control: priceless!

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: pgpenvelope 2.10.2 - http://pgpenvelope.sourceforge.net/

iD8DBQFDO/JZs/NR4JuTKG8RAvD9AJ4sIDmCRHHKbJNacOTwMWVUQ48/BQCfRX6t
Ie8gCiJPgr36ek/LYJi5/mo=
=CORp
-----END PGP SIGNATURE-----

[Claudia Jürgen]

Hi Mark,

looks as if you got a connection leak (you should have a lot of idle in
transaction processes too). We've got the same problem a while ago and
following

http://sourceforge.net/mailarchive/message.php?msg_id=9538936

helped.

Claudia
University Library of Dortmund

>------------------------------------------------------------------------
>
>From dspace-...@savage.ulib.IUPUI.Edu Tue Sep 27 13:32:35 2005
>Date: Tue, 27 Sep 2005 13:25:52 -0500
>From: dspace-...@savage.ulib.IUPUI.Edu
>To: IN-ULi...@Exchange.IU.Edu
>Subject: DSpace: Internal Server Error
>
>An internal server error occurred on https://idea.iupui.edu:8443/dspace:
>
>Date: 9/27/05 1:25 PM
>Session ID: 409AB85F09057ED55D8C84324701EE65
>
>-- URL Was: https://idea.iupui.edu/
>-- Method: GET
>-- Parameters were:
>
>
>Exception:
>org.apache.commons.dbcp.SQLNestedException: Connect failed, cause:
>java.util.NoSuchElementException: Timeout waiting for idle object
> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:801)
> at org.apache.commons.dbcp.PoolingDriver.connect(PoolingDriver.java:171)
> at java.sql.DriverManager.getConnection(DriverManager.java:525)
> at java.sql.DriverManager.getConnection(DriverManager.java:193)
> at org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:373)
> at org.dspace.core.Context.<init>(Context.java:102)
> at org.dspace.app.webui.util.UIUtil.obtainContext(UIUtil.java:93)
> at org.apache.jsp.index_jsp._jspService(index_jsp.java:78)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:324)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
> at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
> at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
> at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
> at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
> at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:535)
> at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
> at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
> at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
> at java.lang.Thread.run(Thread.java:595)
>
>
>
>

[abhishek.kulkarni]

Hi Helix84,

I created new instance of dspace3.1 and installation went successful.
But, hitting the URL throws following exception:-

java.util.NoSuchElementException: Timeout waiting for idle object
at

org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:958)
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at
org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:588)
at org.dspace.core.Context.<init>(Context.java:95)
at
org.dspace.app.xmlui.utils.ContextUtil.obtainContext(ContextUtil.java:72)
at
org.dspace.app.xmlui.utils.ContextUtil.obtainContext(ContextUtil.java:53)
at
org.dspace.app.xmlui.cocoon.AbstractDSpaceTransformer.setup(AbstractDSpaceTransformer.java:78)
at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
at sun.proxy.$Proxy19.setup(Unknown Source)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:356)
at
org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.prepareInternal(AbstractProcessingPipeline.java:480)
at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)



COCOON STACK TRACE:-

Failed to process pipeline
context:/jndi:/localhost/aspects/aspects.xmap - 115:34 <map:serialize
type="xml">
context:/jndi:/localhost/aspects/aspects.xmap - 114:43 <map:transform
type="PageNotFound">
context:/jndi:/localhost/aspects/aspects.xmap - 113:22 <map:generate>
context:/jndi:/localhost/themes/Mirage/sitemap.xmap - 162:34 <map:serialize
type="xhtml">
context:/jndi:/localhost/themes/Mirage/sitemap.xmap - 156:33 <map:transform
type="i18n">
context:/jndi:/localhost/themes/Mirage/sitemap.xmap - 151:51 <map:transform>
context:/jndi:/localhost/themes/Mirage/sitemap.xmap - 138:55 <map:transform
type="IncludePageMeta">
context:/jndi:/localhost/themes/Mirage/sitemap.xmap - 133:55 <map:transform
type="IncludePageMeta">
context:/jndi:/localhost/themes/Mirage/sitemap.xmap - 124:55 <map:transform
type="IncludePageMeta">
context:/jndi:/localhost/themes/Mirage/sitemap.xmap - 118:55 <map:generate
type="file">
context:/jndi:/localhost/themes/Mirage/sitemap.xmap - 115:28 <map:match>
context:/jndi:/localhost/themes/themes.xmap - 63:45 <map:mount>
context:/jndi:/localhost/themes/themes.xmap - 62:35 <map:match
type="ThemeMatcher">
context:/jndi:/localhost/sitemap.xmap - 588:94 <map:mount>


Can you please help?

Thanks,
Abhishek Kulkarni
Student assistant - Library Technology Group
Illinois Institute of Technology
Chicago, Illinois






--
View this message in context: http://dspace.2283337.n4.nabble.com/java-util-NoSuchElementException-Timeout-waiting-for-idle-object-tp4663019.html
Sent from the DSpace - Tech mailing list archive at Nabble.com.

[helix84]

Does this help?

http://dspace.2283337.n4.nabble.com/Recurring-Error-org-dspace-app-oai-PluginCrosswalk-td4662351.html#a4662359


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

[abhishek.kulkarni]

I tried setting db.statementpool = false, still no luck..

Exception in full java stack trace is: -

Caused by: org.xml.sax.SAXException: An error was encountered while
processing the 'aspect.general.PageNotFoundTransformer' Wing based
component: org.dspace.app.xmlui.aspect.general.PageNotFoundTransformer
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool
error Timeout waiting for idle object
at
org.dspace.app.xmlui.cocoon.AbstractDSpaceTransformer.handleException(AbstractDSpaceTransformer.java:109)
at
org.dspace.app.xmlui.cocoon.AbstractDSpaceTransformer.setup(AbstractDSpaceTransformer.java:93)
at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
at sun.proxy.$Proxy19.setup(Unknown Source)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:356)

... 95 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a
connection, pool error Timeout waiting for idle object
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)

at
org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:588)
at org.dspace.core.Context.<init>(Context.java:95)
at
org.dspace.app.xmlui.utils.ContextUtil.obtainContext(ContextUtil.java:72)
at
org.dspace.app.xmlui.utils.ContextUtil.obtainContext(ContextUtil.java:53)
at
org.dspace.app.xmlui.cocoon.AbstractDSpaceTransformer.setup(AbstractDSpaceTransformer.java:78)

... 101 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:958)
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

... 106 more




I tested the db connection too, it was successful though.

Attempting to connect to database:
- URL: jdbc:postgresql://localhost:5432/dspace
- Driver: org.postgresql.Driver
- Username: dspace
- Password: dspace
- Schema:

Testing connection...
Connected successfully!



Where am I going wrong?!

Thanks,
Abhishek





--
View this message in context: http://dspace.2283337.n4.nabble.com/java-util-NoSuchElementException-Timeout-waiting-for-idle-object-tp4663019p4663072.html

[helix84]

You should try to raise the number of allowed concurrent transactions
in both Postgres configuration (max_connections) and DSpace
configuration (db.maxconnections).

If you connect to your database and run "SELECT * from
pg_stat_activity;" and you see many items as "<IDLE> in transaction",
it means all your connections are used up. In particular, this was
often caused by the old oai module in DSpace 1.8 and older.

You can find more about this error in the archive of this mailing list
and in Jira. There isn't really any reliable solution besides raising
the number of connections and occasionally restarting Tomcat or
upgrading to 3.0.

[abhishek.kulkarni]

I am facing this problem after creation new instance of dspace 3.0.





--
View this message in context: http://dspace.2283337.n4.nabble.com/java-util-NoSuchElementException-Timeout-waiting-for-idle-object-tp4663019p4663076.html

[abhishek.kulkarni]

I have created new instance of dspace 3.0, and facing this problem on it.

Thanks,
Abhishek




--
View this message in context: http://dspace.2283337.n4.nabble.com/java-util-NoSuchElementException-Timeout-waiting-for-idle-object-tp4663019p4663078.html

[abhishek.kulkarni]

Full java stack trace: -

Caused by: org.xml.sax.SAXException: An error was encountered while
processing the 'aspect.general.PageNotFoundTransformer' Wing based
component: org.dspace.app.xmlui.aspect.general.PageNotFoundTransformer
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool

error Timeout waiting for idle object
at

org.dspace.app.xmlui.cocoon.AbstractDSpaceTransformer.handleException(AbstractDSpaceTransformer.java:109)
at
org.dspace.app.xmlui.cocoon.AbstractDSpaceTransformer.setup(AbstractDSpaceTransformer.java:93)

at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
at sun.proxy.$Proxy19.setup(Unknown Source)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:356)

... 92 more

Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a

connection, pool error Timeout waiting for idle object
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)

at
org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:588)
at org.dspace.core.Context.<init>(Context.java:95)
at
org.dspace.app.xmlui.utils.ContextUtil.obtainContext(ContextUtil.java:72)
at
org.dspace.app.xmlui.utils.ContextUtil.obtainContext(ContextUtil.java:53)
at
org.dspace.app.xmlui.cocoon.AbstractDSpaceTransformer.setup(AbstractDSpaceTransformer.java:78)

... 98 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:958)
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

--
View this message in context: http://dspace.2283337.n4.nabble.com/java-util-NoSuchElementException-Timeout-waiting-for-idle-object-tp4663019p4663081.html