Hikari CP idle connections

[Ashley Willis]

Hi,

We are currently monitoring Sakai db connections in AppDynamics using JMX and noticed some "strange" behavior.

In AppDynamics we see idle connections occasionally spike while active connections remain relatively low. For example, a few minutes ago one of our app servers went from 25 total connections to 103 while active connections never exceeded 3 leaving us guessing as to why Hikari would spin up 100 idle connections if roughly 22 of the current connections in the pool are idle. We are configured for minimumIdle = 10 and idleTimeout=600000. The 103 connections lasted for 10 minutes and then drop back to around 25 which seems like 12 too many (3 active + 10 min idle). The db doesn't seem to be stressed that we can tell, and slow query log is relatively quiet. FWIW, minEvictableIdleTimeMillis=1800000 and timeBetweenEvictionRunsMillis = 900000 but those don't seem to be an issue as it cleans the connections, just the idle connections are higher than expected and no explanation for the spikes.

Originally we were assuming the connection spike was a code issue until we graphed the active to idle ratio. Now it seems like maybe a Hikari bug or configuration issue. Has anyone else seen behavior like this with Hikari or have any thoughts on what else we might look for? Has anyone on Sakai 11.x upgraded Hikari to a newer version?

We are on Sakai 11.2ish, MySQL 5.6.40, mysql-connector-java-5.1.39 and Hikari 2.4.6

Thanks,

Ashley

UNC Chapel Hill

[Philippe Rancourt]

We experienced the kind of spike you are describing. At the end, it was an Hikari bug and upgrading to hikari 2.6.1 resolved the issue.

https://jira.sakaiproject.org/browse/SAK-39357 

https://github.com/brettwooldridge/HikariCP/issues/835

[Ashley Willis]

Thanks for the confirmation and links!

Ashley