High CPU while waiting for connection.

[Erik van Ingen]

Even though the DB is down, Hiraki will try to obtain a connection. This is of course all expected behavior. The problem is that the CPU usage goes to 100%, disturbing other processes. We have the impression that the CPU is consumed by Hiraki. 

Is it possible to avoid Hiraki consuming high CPU, in the rare case that the DB is down?

[Brett Wooldridge]

Can you get a thread dump while the CPU is at 100% usage?  HikariCP should not be burning CPU while the DB is down.  Callers to getConnection() should be blocking for connectionTimeout milliseconds, and the refill threads should back-off to one connection attempt per second.  But it is possible that they are somehow causing CPU load.  I'll try to do some testing on my end as well.

[Erik van Ingen]

Thank you for reflecting on this. We had lots of those reported in the logfiles, see below. It is not possible for me to repeat the problem, because the DB goes rarely down. 

Sep 11, 2014 5:34:55 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException 

SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container 

org.fao.fi.vme.VmeException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection 

        at org.vme.web.cache.CacheInterceptor.authorize(CacheInterceptor.java:58) 

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 

Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection 

        at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763) 

Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection 

        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54) 

Caused by: java.sql.SQLException: Timeout of 30000ms encountered waiting for connection. 

        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:207) 

        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:93) 

        at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.getConnection(HikariCPConnectionProvider.java:89) 

        at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:380) 

        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:228) 

        ... 80 more 

[Brett Wooldridge]

Thanks for that.  Unfortunately, it doesn't tell me much, timeout exceptions are expected when the DB is down.  Are you by any chance using SSL connections to the database?  Just trying to think of things that might induce load...