DB connections leaking

[ste...@powerinbox.com]

Hi all,

We use HikariCP with Play! 2.6.10. Our application will run fine for days, and then all of our DB connections will leak, leaving the application unable to make any queries. We have leakDetectionThreshold turned on, so we get stack traces for leaked connections like:

2018-01-24 06:29:00,857 - [WARN] - from com.zaxxer.hikari.pool.ProxyLeakTask in 
HikariPool-2 housekeeper 
Connection leak detection triggered for com.mysql.jdbc.JDBC4Connection@65cd084 on thread pool-1-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.jav
a:85)
        at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
        at play.api.db.DefaultDatabase.withConnection(Databases.scala:152)
        at play.api.db.DefaultDatabase.withConnection(Databases.scala:148)
        at models.summaries.ActionSummary$.listByStripe(ActionSummary.scala:137)
 

We only use the connections through Play's withConnection, so they should be returned to the pool automatically. A thread dump, when the application is in a broken state, shows that all threads inside of a withConnection blocks are stuck on...

"application-akka.mysql-context-122" #32142 prio=5 os_prio=0 tid=0x00007fca7812a
000 nid=0x28e6 waiting on condition [0x00007fca7541c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006cf097300> (a java.util.concurrent.Sync
hronousQueue$TransferQueue)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215
)
        at java.util.concurrent.SynchronousQueue$TransferQueue.awaitFulfill(Sync
hronousQueue.java:764)
        at java.util.concurrent.SynchronousQueue$TransferQueue.transfer(Synchron
ousQueue.java:695)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at com.zaxxer.hikari.util.ConcurrentBag.borrow(ConcurrentBag.java:157)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:165)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
        at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
        at play.api.db.DefaultDatabase.withConnection(Databases.scala:152)
        at play.api.db.DefaultDatabase.withConnection(Databases.scala:148)
 

...waiting for a connection to be available, which should mean that none are currently holding a connection. I have no idea how any connection could possible be leaked, but apparently all of them have. We see logging lines like:

2018-01-24 06:19:21,297 - [DEBUG] - from com.zaxxer.hikari.pool.HikariPool in application-akka.mysql-context-129 
HikariPool-2 - Timeout failure stats (total=10, active=10, idle=0, waiting=15)
 

The only unusual thing we are doing is calling setNetworkTimeout on each connection we obtain, sometimes with a timeout as low as 10 seconds. This is done to ensure that queries fail fast if we lose connection to the DB.

I'm not sure what to do next debugging this. It looks like maybe a potential issue between Hikari and Play!, or something broken with MySQL and setNetworkTimeout. Has anyone see anything similar, or have ideas as to what to try?

Thanks

Stephen

[Marcos Pereira]

Hi Stephen, thanks for the clear report here.

To all, there is already a pull request to fix the problem: https://github.com/playframework/playframework/pull/8198

Best.

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/e6b0e0ff-3fea-4f0e-bddd-0111b8306aa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marcos Pereira

Software Engineer, Lightbend.com