Dspace 6.3 blank page

[Juan López]

Hi!

Every saturday my dspace 6.3 is giving me a blank page, the service keeps running but the dspace.log shows the following error:

2021-09-17 21:20:03,774 WARN  org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ SQL Error: 0, SQLState: null

2021-09-17 21:20:03,774 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ Cannot get a connection, pool error Timeout waiting for idle object

2021-09-17 21:20:03,834 ERROR org.dspace.statistics.SolrLoggerServiceImpl @ Unable to get location of request:  The address 172.16.29.2 is not in the database.

2021-09-17 21:20:03,880 WARN  org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ SQL Error: 0, SQLState: null

2021-09-17 21:20:03,880 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ Cannot get a connection, pool error Timeout waiting for idle object

2021-09-17 21:20:03,881 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!

javax.servlet.ServletException: org.apache.cocoon.ProcessingException: Sitemap: error invoking action

at <map:act type="locale"> - file:///datos/edocur60/webapps/xmlui/sitemap.xmap:895:64

at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:230)

at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)

at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:468)

at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:443)

at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:264)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

at com.sun.proxy.$Proxy160.service(Unknown Source)

at org.dspace.springmvc.CocoonView.render(CocoonView.java:113)

at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1216)

at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1001)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:626)

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:113)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:160)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.dspace.app.xmlui.cocoon.servlet.multipart.DSpaceMultipartFilter.doFilter(DSpaceMultipartFilter.java:119)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)

at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:432)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.cocoon.ProcessingException: Sitemap: error invoking action

at <map:act type="locale"> - file:///datos/edocur60/webapps/xmlui/sitemap.xmap:895:64

at org.apache.cocoon.ProcessingException.throwLocated(ProcessingException.java:111)

at org.apache.cocoon.components.treeprocessor.sitemap.ActTypeNode.invoke(ActTypeNode.java:141)

at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)

at org.apache.cocoon.components.treeprocessor.sitemap.SwitchSelectNode.invoke(SwitchSelectNode.java:88)

at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)

at org.apache.cocoon.components.treeprocessor.sitemap.HandleErrorsNode.invoke(HandleErrorsNode.java:78)

at org.apache.cocoon.components.treeprocessor.sitemap.ErrorHandlerHelper.prepareErrorHandler(ErrorHandlerHelper.java:157)

at org.apache.cocoon.components.treeprocessor.sitemap.ErrorHandlerHelper.prepareErrorHandler(ErrorHandlerHelper.java:124)

at org.apache.cocoon.components.treeprocessor.sitemap.ErrorHandlerHelper.invokeErrorHandler(ErrorHandlerHelper.java:94)

at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:89)

at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)

at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)

at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)

at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)

at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)

... 52 more

Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection

at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)

at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:124)

at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:109)

at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:221)

at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)

at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)

at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)

at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1387)

at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:352)

at com.sun.proxy.$Proxy41.beginTransaction(Unknown Source)

at org.dspace.core.HibernateDBConnection.getSession(HibernateDBConnection.java:43)

at org.dspace.core.HibernateDBConnection.configureDatabaseMode(HibernateDBConnection.java:154)

at org.dspace.core.HibernateDBConnection.setConnectionMode(HibernateDBConnection.java:140)

at org.dspace.core.Context.setMode(Context.java:701)

at org.dspace.core.Context.init(Context.java:183)

at org.dspace.core.Context.<init>(Context.java:126)

at org.dspace.app.xmlui.utils.ContextUtil.obtainContext(ContextUtil.java:95)

at org.dspace.app.xmlui.utils.ContextUtil.obtainContext(ContextUtil.java:59)

at org.dspace.app.xmlui.cocoon.DSpaceLocaleAction.act(DSpaceLocaleAction.java:109)

at org.apache.cocoon.sitemap.impl.DefaultExecutor.invokeAction(DefaultExecutor.java:55)

at org.apache.cocoon.components.treeprocessor.sitemap.ActTypeNode.invoke(ActTypeNode.java:105)

... 65 more

Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object

at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142)

at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)

at org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141)

at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:301)

at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214)

... 85 more

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449)

at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)

at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)

... 89 more

What could be causing this issue?

Hope you guys can help with this one.

Juan.

[Hrafn Malmquist]

Hi Juan

I'm having same issue.

Can you tell me more about your setup, do u am you have Apache or nginx on front of Tomcat?

How frequent and regular are these downtimes?

Do you see a lot of Client closed the connection errors?

Best, Hrafn

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/b1ccecdf-7012-49be-897a-4aa2c4ed54bcn%40googlegroups.com.

[José Geraldo]

Check your connection database. If database is UP.

Error: Cannot get a connection, pool error Timeout waiting for idle object

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/CAG9P5m_tu98sZAnWd0aUnQCRh8w8hC65-ZWLpfRPeNbjFCMY%3Dg%40mail.gmail.com.

--

At.te,

José Geraldo

[Hrafn Malmquist]

It's not whether it's up. The connection pool can't provide a connection.

[Hrafn Malmquist]

Hello everyone

It would be interesting to hear if other DSpace admins are experiencing this issue.

We've identified Google Analytics as the cause. See here:

https://jira.lyrasis.org/browse/DS-2718

https://jira.lyrasis.org/browse/DS-2904

You can confirm this by checking for the following log entries:

2015-08-24 10:33:08,143 INFO org.apache.http.impl.execchain.RetryExec @ I/O exception (java.net.SocketException) caught when processing request to

{s}->https://www.google-analytics.com:443: Network is unreachable
2015-08-24 10:33:08,143 INFO org.apache.http.impl.execchain.RetryExec @ Retrying request to {s}

->https://www.google-analytics.com:443

If this is the case the solution is to comment out and disable your Google Analytics key, thereby disabling the Google Analytics functionality.

Hope this helps

Best regards,

Hrafn

[Hrafn Malmquist]

By the way, there is a PR open with a solution:

https://github.com/DSpace/DSpace/pull/2248

[Juan López]

Hi Hrafn,

to answer your questions:

1. I have Apache in front of Tomcat

2. it happens every friday nigth until saturday. 

3. I haven't checked for any  client closed connection error.

And if Google Analitycs is the root of this issue, shouldn't we be able to do something with the robots.txt?

Thanks for your answers!

--

Juan López

Cel: 3156962824 - Wpp: 3013981138

[Michael Plate]

Hi Juan,

Am 22.09.21 um 16:44 schrieb Juan López:

> Hi Hrafn,
>
> to answer your questions:
>
> 1. I have Apache in front of Tomcat
> 2. it happens every friday nigth until saturday.

[…]

- look for the load of the system
- use netstat -napt for network connections
- when does your system / provider backups
- look on the cron jobs

this reoccuring leads to something like that

Michael

[Hrafn Malmquist]

Hi Juan

It does not matter if Apache sits in front of Tomcat. That was a red herring, I thought it might me SlowLoris attacks that Apache was possibly passing on to Tomcat, keeping ports open. Forget about that.

Provided you have Google Analytics on, check for the following in your DSpace logs:

2015-08-24 10:33:08,143 INFO org.apache.http.impl.execchain.RetryExec @ I/O exception (java.net.SocketException) caught when processing request to

{s}->https://www.google-analytics.com:443: Network is unreachable
2015-08-24 10:33:08,143 INFO org.apache.http.impl.execchain.RetryExec @ Retrying request to {s}->https://www.google-analytics.com:443

If you do I would say it's almost certain your problem lies in a bottleneck being created there which holds up the database connections. It's clear that the problem is database connection pooling.

With regards to the time this happens it makes sense for it to happen when traffic is at its highest level. It's possible also that this has got something to do with cronjobs.

Hrafn