Gerrit 2.8 dies - can't get DB connection (to H2!)

[James Nord]

Hi all,

We have what appears to be a DB Connection leak in 2.8 (not observed in 2.5).

All access to the server is rejected with 

com.google.inject.ProvisionException: Guice provision errors:

1) Cannot open ReviewDb

  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)

  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error

        at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1014)

<snip>

Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection

        at com.google.gwtorm.jdbc.Database.newConnection(Database.java:129)

        at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:41)

<snip>

Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)

        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)

        at com.google.gwtorm.jdbc.Database.newConnection(Database.java:127)

        ... 71 more

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)

        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

        ... 73 more

In this state netstat reported only 3 open connections to the gerrit https interface, and the sshd (gerrit show-connections) reported only 4 connections.

This is using the H2 in memory database - not mySql or any other remote DB.

The only other issues in the log are 

WARN  org.eclipse.jetty.io.nio : javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?

Not sure if that is related or not?

Restarting the server obviously fixes this.  As its a production server (we haven't seen this on the test instance) I can't attach a debugger but can obviously turn on logging if there is any areas that need investigation.

A quick trawl of the issues showed https://code.google.com/p/gerrit/issues/detail?id=1066 

[James Nord]

In case anyone else sees similar there is a serious bug in the h2 backend that causes a deadlock.

https://code.google.com/p/gerrit/issues/detail?id=2365