A problem about JDBC connection leak

[Jeremy Yang]

Exception Log：

java.util.concurrent.ExecutionException: com.google.gwtorm.server.OrmException: Cannot open database connection

        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)

        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)

        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)

        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)

        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2346)

        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318)

        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)

        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)

        at com.google.common.cache.LocalCache.get(LocalCache.java:3934)

        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)

        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)

        at com.google.gerrit.server.git.ChangeCache.get(ChangeCache.java:67)

        at com.google.gerrit.server.git.VisibleRefFilter.visibleChanges(VisibleRefFilter.java:153)

        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:79)

        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:142)

        at com.google.gerrit.server.git.VisibleRefFilter.getAdvertisedRefs(VisibleRefFilter.java:131)

        at org.eclipse.jgit.transport.AbstractAdvertiseRefsHook.advertiseRefs(AbstractAdvertiseRefsHook.java:62)

        at org.eclipse.jgit.transport.UploadPack.sendAdvertisedRefs(UploadPack.java:785)

        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:664)

        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:630)

        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:80)

        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

        at java.util.concurrent.FutureTask.run(FutureTask.java:166)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:370)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

        at java.lang.Thread.run(Thread.java:722)

Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection

        at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)

        at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)

        at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22.<init>(Unknown Source)

        at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22_Factory_GwtOrm$$23.open(Unknown Source)

        at com.google.gwtorm.jdbc.Database.open(Database.java:122)

        at com.google.gerrit.server.git.ChangeCache$Loader.load(ChangeCache.java:91)

        at com.google.gerrit.server.git.ChangeCache$Loader.load(ChangeCache.java:81)

        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)

        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)

        ... 28 more

Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)

        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)

        at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)

        ... 36 more

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)

        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

        ... 38 more

Database Config:

type = postgresql

poolLimit= 500

poolMaxIdle = 16

connectionPool = true

Monitor:

From the javamelody monitor,we can observe that the count of  active jdbc connections increase as the time went on.

At the end,when the count exceeded the limit(500),the gerrit cannot accept new request.

As for me,I have commit some code to the BasicDataSource Class in gerrit,but something wrong happened which is that I could not find the log of removed connections:

Change-Id: I8b650990d4e2a127e805cfba0a66b2ff1872ca72

Have you guys encountered this problem ?Or any good advices to debug or solve it?