gerrit Cannot open database connection

1,064 views
Skip to first unread message

geyu...@gmail.com

unread,
Oct 15, 2017, 11:05:20 PM10/15/17
to Repo and Gerrit Discussion
- Ubuntu 14.04.4 LTS
- Gerrit 13.3

- error once a week
how to fix it?

- GERRIT CONFIG：
[database]
type = postgresql
hostname = 10.112.37.33
database = review
username = gerrit2
        connectionPool = true
poolLimit = 512
poolMaxIdle = 64
poolMaxWait = 2min
[index]
type = LUCENE
[auth]
type = LDAP
[ldap]
        ....
[sendemail]

[container]
user = git
javaHome = /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre
javaOptions = -Xmx80g -Xms16g -XX:MaxMetaspaceSize=8g -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
[core]
        packedGitLimit = 4g
        packedGitWindowSize = 16k
[sshd]
listenAddress = *:29418
commandStartThreads = 3
maxConnectionsPerUser = 5
maxAuthTries = 3
threads = 64
[httpd]
listenUrl = http://*:8088/
maxThreads = 100


- POSTGRESQL CONFIG
listen_addresses = '*'          # what IP address(es) to listen on;
port = 5432                             # (change requires restart)
max_connections = 1000                  # (change requires restart)

- ERROR LOG
[2017-10-12 11:39:38,898] [Index-Batch-22] ERROR com.google.gerrit.server.index.change.ReindexAfterUpdate : Failed to reindex changes after Event[platform/packages/apps/YellowPage,refs/heads/v6-alpha: e98054b8f16c9b0874663b449b65a3bb95e2a3d7 -> 7e2a19e27e2df46e47ca355a0a6425ee96858c5f]
com.google.gwtorm.server.OrmException: Cannot open database connection
at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$16.<init>(Unknown Source)
at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$16_Factory_GwtOrm$$17.open(Unknown Source)
at com.google.gwtorm.jdbc.Database.open(Database.java:122)
at com.google.gerrit.server.schema.NotesMigrationSchemaFactory.open(NotesMigrationSchemaFactory.java:40)
at com.google.gerrit.server.schema.NotesMigrationSchemaFactory.open(NotesMigrationSchemaFactory.java:25)
at com.google.gerrit.server.util.ManualRequestContext.<init>(ManualRequestContext.java:37)
at com.google.gerrit.server.util.OneOffRequestContext.open(OneOffRequestContext.java:48)
at com.google.gerrit.server.index.change.ReindexAfterUpdate$Task.call(ReindexAfterUpdate.java:107)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:417)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
... 20 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 22 more
............
[2017-10-12 11:39:39,029] [HTTP-950723] ERROR com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener : Cannot open database connection
com.google.gwtorm.server.OrmException: Cannot open database connection
at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$16.<init>(Unknown Source)
at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$16_Factory_GwtOrm$$17.open(Unknown Source)
at com.google.gwtorm.jdbc.Database.open(Database.java:122)
at com.google.gerrit.server.schema.NotesMigrationSchemaFactory.open(NotesMigrationSchemaFactory.java:40)
at com.google.gerrit.server.schema.NotesMigrationSchemaFactory.open(NotesMigrationSchemaFactory.java:25)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener.createReviewNotes(RefUpdateListener.java:98)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener.access$000(RefUpdateListener.java:37)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener$1.run(RefUpdateListener.java:65)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener.onGitReferenceUpdated(RefUpdateListener.java:91)
at com.google.gerrit.server.extensions.events.GitReferenceUpdated.fire(GitReferenceUpdated.java:119)
at com.google.gerrit.server.extensions.events.GitReferenceUpdated.fire(GitReferenceUpdated.java:76)
at com.google.gerrit.server.git.NotesBranchUtil.updateRef(NotesBranchUtil.java:264)
at com.google.gerrit.server.git.NotesBranchUtil.commitNotes(NotesBranchUtil.java:154)
at com.google.gerrit.server.git.NotesBranchUtil.commitAllNotes(NotesBranchUtil.java:105)
at com.googlesource.gerrit.plugins.reviewnotes.CreateReviewNotes.commitNotes(CreateReviewNotes.java:203)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener.createReviewNotes(RefUpdateListener.java:106)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener.access$000(RefUpdateListener.java:37)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener$1.run(RefUpdateListener.java:65)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener.onGitReferenceUpdated(RefUpdateListener.java:91)
at com.google.gerrit.server.extensions.events.GitReferenceUpdated.fire(GitReferenceUpdated.java:119)
at com.google.gerrit.server.extensions.events.GitReferenceUpdated.fire(GitReferenceUpdated.java:99)
at com.google.gerrit.server.git.BatchUpdate.execute(BatchUpdate.java:442)
at com.google.gerrit.server.git.MergeOp.integrateIntoHistory(MergeOp.java:478)
at com.google.gerrit.server.git.MergeOp.merge(MergeOp.java:433)
at com.google.gerrit.server.change.Submit.apply(Submit.java:225)
at com.google.gerrit.server.change.Submit.apply(Submit.java:83)
at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:341)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:122)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:60)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:201)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178)
at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:132)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:105)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
... 71 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 73 more
......
[2017-10-12 11:39:53,838] [HTTP-952509] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/170363/detail?O=10004
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Cannot open ReviewDb
  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70) (via modules: com.google.gerrit.server.config.GerritGlobalModule -> com.google.gerrit.server.util.ThreadLocalRequestContext$1)
  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028)
at com.google.gerrit.lucene.LuceneChangeIndex.toChangeData(LuceneChangeIndex.java:468)
at com.google.gerrit.lucene.LuceneChangeIndex.access$1000(LuceneChangeIndex.java:108)
at com.google.gerrit.lucene.LuceneChangeIndex$ChangeDataResults.toList(LuceneChangeIndex.java:393)
at com.google.gerrit.server.index.change.IndexedChangeQuery$1.toList(IndexedChangeQuery.java:108)
at com.google.gerrit.server.query.QueryProcessor.query(QueryProcessor.java:196)
at com.google.gerrit.server.query.QueryProcessor.query(QueryProcessor.java:139)
at com.google.gerrit.server.query.QueryProcessor.query(QueryProcessor.java:122)
at com.google.gerrit.server.query.InternalQuery.query(InternalQuery.java:71)
at com.google.gerrit.server.query.change.InternalChangeQuery.byLegacyChangeId(InternalChangeQuery.java:122)
at com.google.gerrit.server.ChangeFinder.find(ChangeFinder.java:63)
at com.google.gerrit.server.change.ChangesCollection.parse(ChangesCollection.java:81)
at com.google.gerrit.server.change.ChangesCollection.parse(ChangesCollection.java:38)
at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:247)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:122)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:60)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:201)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178)
at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:132)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:105)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$16.<init>(Unknown Source)
at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$16_Factory_GwtOrm$$17.open(Unknown Source)
at com.google.gwtorm.jdbc.Database.open(Database.java:122)
at com.google.gerrit.server.schema.NotesMigrationSchemaFactory.open(NotesMigrationSchemaFactory.java:40)
at com.google.gerrit.server.schema.NotesMigrationSchemaFactory.open(NotesMigrationSchemaFactory.java:25)
at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:46)
at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27)
at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)
at com.google.gerrit.server.util.ThreadLocalRequestContext$1$$FastClassByGuice$$75e0eb90.invoke(<generated>)
at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
... 56 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
... 72 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 74 more

Luca Milanesio

unread,
Oct 16, 2017, 3:59:48 AM10/16/17
to geyu...@gmail.com, Repo and Gerrit Discussion
You should trace the DB connections utilisation and see if this is a leak or a misconfiguration of the DB connection pool.
You have 512 connections available which is *A LOT*, I am quite surprised you can use all of them :-(

Why don't you start installing the JavaMelody plugin and getting some graphs?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Gustaf Lundh

unread,
Nov 13, 2017, 7:09:09 AM11/13/17
to Repo and Gerrit Discussion
We are seeing a similar issue in Gerrit 2.14.5.

Every 5th minute a database connection is created and then not released from the connection pool. This continues until the pool is full and Gerrit hangs. We are also running on Ubuntu Trusty with Postgres. Did you ever track down the issue? We are currently working around the issue by disabling the database pool (which of course have performance costs).

/Gustaf

Sven Selberg

unread,
Nov 13, 2017, 7:50:00 AM11/13/17
to Repo and Gerrit Discussion
It is really steady as clockwork [1}

A new, unused/unusable idle connection is added to the pool every five minutes.
If you look at pg_stats_activity they are marked as idle, but as far as Gerrit is concerned they are active [1].
For the unusable connections pg_stat_activity.query is "select 1" (unfortunately we did not save the output from pg_stat_activity with the locked connections and we cannot reproduce it in dev or staging).

/Sven

[1]

Sven Selberg

unread,
Nov 13, 2017, 7:52:08 AM11/13/17
to Repo and Gerrit Discussion

Hugo Arès

unread,
Nov 13, 2017, 2:29:00 PM11/13/17
to Repo and Gerrit Discussion
"SELECT 1" is the default validation query we use to confirm that a connection is still valid before returning it into the pool.

What is your configuration for the following parameters in gerrit.config:

database.poolLimit
database.poolMaxIdle
database.poolMaxWait

I am asking because the default poolMaxIdle is 16 and in your graph, it slowly increase up to 13 which could be "normal" if
you did not configure database.poolMaxIdle.

Gustaf Lundh

unread,
Nov 13, 2017, 2:36:25 PM11/13/17
to Repo and Gerrit Discussion
This is the database configuration, minus username and database configuration options.

[database]
type = postgresql
poolLimit = 128
poolMaxWait = 3 minutes

Sven only posted a part of the graph, which would more clearly show the leakage intervals.

Here is the full graph where we went from clean start to out of connections in the pool in 8 horus:


Best regards
Gustaf

Gustaf Lundh

unread,
Nov 13, 2017, 2:39:15 PM11/13/17
to Repo and Gerrit Discussion
Please note that this is during a period with very little traffic (Sunday evening)

/Gustaf

Hugo Arès

unread,
Nov 13, 2017, 2:48:09 PM11/13/17
to Repo and Gerrit Discussion
I just check on my side (postgres and gerrit 2.14.5.1) and we are not suffering this connection leak.

What plugins do you have?

Gustaf Lundh

unread,
Nov 13, 2017, 2:56:46 PM11/13/17
to Repo and Gerrit Discussion
Ok. I found our leak.

Apparently this is the issue we are having:


With our Kibana instance down last night, I had a harder time than usual tracking down the suspects. :/
 
Anyways. We had a script executing suexec commands every 5th minute. Or rather, it is not the suexec that is causing the leak, but every single login from the internal "Gerrit Code Review" account.

I.e. I executed this line 10 times:

   ssh "Gerrit Code Review"@localhost -p 29418 -i path/to/key suexec --as user gerrit version   

And I tied up 10 connections in the pool which cannot be released.

With that in mind it should not be that hard to track down the root cause.

/Gustaf

Gustaf Lundh

unread,
Nov 13, 2017, 3:02:53 PM11/13/17
to Repo and Gerrit Discussion
> Or rather, it is not the suexec that is causing the leak, but every single login from the internal "Gerrit Code Review" account.

Sorry. I wrote this wrong. It is indeed the suexec that is a cause of our leak. I tried other commands like query, but no leak as long as suexec is not used.

/Gustaf

Hugo Arès

unread,
Nov 14, 2017, 3:34:12 PM11/14/17
to Repo and Gerrit Discussion

ld...@audiokinetic.com

unread,
Oct 11, 2018, 12:43:00 PM10/11/18
to Repo and Gerrit Discussion
Hi, I have installed javamelody but i can't seem to figure out how to print a graph like yours. Could you help?


On Monday, November 13, 2017 at 7:52:08 AM UTC-5, Sven Selberg wrote:

Paul Tung

unread,
Oct 23, 2018, 6:55:37 AM10/23/18
to Repo and Gerrit Discussion
Hi, I know it's a old post but seems it happened on 2.15.5 again as below, any idea?
OS: CentOS 7.5.1804
Gerrit: 2.15.5
Gerrit Config:
[database]
        type = h2
        database = db/ReviewDB
        connectionPool = true
        pollLimit = 35
[index]
        type = LUCENE
[auth]
        #type = DEVELOPMENT_BECOME_ANY_ACCOUNT
        type = OAUTH
        #type = OPENID
        gitBasicAuthPolicy = HTTP
[sshd]
        listenAddress = *:7799
        #threads = 2
        #batchThreads = 1
[cache]
        #directory = cache
        directory = /dev/shm
        h2CacheSize = 512m
        h2AutoServer = true
[cache "web_sessions"]
        maxAge = 1 month
[gc]
        startTime = Sun 04:17
        interval = 1 week
[plugins]
        allowRemoteAdmin = true
[receive]
        enableSignedPush = false
[container]
        user = gerrit
        javaHome = /usr/java/jre1.8.0_152
        heapLimit = 1g
[core]
        packedGitLimit = 256m
        packedGitOpenFiles = 512
error_log:
[2018-10-23 18:18:12,709] [SSH git-receive-pack /auto_deploy (paul_tung)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user paul_tung account 1000001) during git-receive-pack '/auto_deploy'
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Cannot open ReviewDb
  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:69) (via modules: com.google.gerrit.server.config.GerritGlobalModule -> com.google.gerrit.server.util.ThreadLocalRequestContext$1)
  while locating com.google.gerrit.reviewdb.server.ReviewDb
    for the 26th parameter of com.google.gerrit.server.git.receive.ReceiveCommits.<init>(ReceiveCommits.java:420)
  while locating com.google.gerrit.server.git.receive.ReceiveCommits annotated with @com.google.inject.internal.UniqueAnnotations$Internal(value=9)
  at com.google.gerrit.server.git.receive.AsyncReceiveCommits.<init>(AsyncReceiveCommits.java:191)
  while locating com.google.gerrit.server.git.receive.AsyncReceiveCommits annotated with @com.google.inject.internal.UniqueAnnotations$Internal(value=9)

1 error
        at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:226)
        at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1053)
        at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836)
        at com.sun.proxy.$Proxy165.create(Unknown Source)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:93)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:97)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:64)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:479)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:558)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
        at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
        at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:43)
        at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$13.<init>(Unknown Source)
        at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$13_Factory_GwtOrm$$14.open(Unknown Source)
        at com.google.gwtorm.jdbc.Database.open(Database.java:122)
        at com.google.gerrit.server.schema.NotesMigrationSchemaFactory.open(NotesMigrationSchemaFactory.java:39)
        at com.google.gerrit.server.schema.NotesMigrationSchemaFactory.open(NotesMigrationSchemaFactory.java:25)
        at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:46)
        at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27)
        at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:69)
        at com.google.gerrit.server.util.ThreadLocalRequestContext$1$$FastClassByGuice$$75e0eb90.invoke(<generated>)
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
        at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:173)
        at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)
        at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
        at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836)
        at com.google.gerrit.server.git.receive.$Proxy166.create(Unknown Source)
        at com.google.gerrit.server.git.receive.AsyncReceiveCommits.<init>(AsyncReceiveCommits.java:232)
        at com.google.gerrit.server.git.receive.AsyncReceiveCommits$$FastClassByGuice$$70104080.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
        ... 15 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
        ... 44 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
        ... 46 more




Hugo Arès於 2017年11月15日星期三 UTC+8上午4時34分12秒寫道：

luca.mi...@gmail.com

unread,
Oct 23, 2018, 7:49:22 AM10/23/18
to Paul Tung, Repo and Gerrit Discussion


Sent from my iPhone

On 23 Oct 2018, at 12:50, Paul Tung <baiba...@gmail.com> wrote:

Hi, I know it's a old post but seems it happened on 2.15.5 again as below, any idea?

2.15.5 is broken: either wait for 2.15.6 or downgrade to 2.15.4

--

Paul Tung

unread,
Oct 23, 2018, 8:41:03 PM10/23/18
to Repo and Gerrit Discussion
Thanks lucamilanesio!!
I will downgrade to 2.15.4 and try again.

One more question: how or where can I get the announcement or info like this?


lucamilanesio於 2018年10月23日星期二 UTC+8下午7時49分22秒寫道：
Reply all
Reply to author
Forward
0 new messages