Facing performance issues in Gerrit newer version V2.11

[Anjan Tummalapalli]

Hello Team,

I have upgraded my existing gerrit version V2.10.2 with V2.11 in my production environment..

We are facing lot of performance issues after upgradation due to which we have stopped our production server

1. Inline edit feature not working
     -> Able to edit changes and saving. While trying to publish edit, server is getting struck and unable to do anything in gerrit. If it is being done by any user in organization, all other users are also getting affected with this operation..

2. Same happening for rebase operation also

3. Top yello banner with 'working..' is still visible even after event completed and not allowing to navigate to other pages. During this time not allowing others as well to do anything until i restart server..

4. Server got slow down after upgrade, sometimes unavailable also by saying below messages
     -> Server Error
     -> Forbidden Error

i am coming across below errors most often in logs

[2015-04-18 02:11:57,369] WARN  org.eclipse.jetty.servlet.ServletHandler : /
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Cannot open ReviewDb
  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70) (via modules: com.google.gerrit.server.config.GerritGlobalModule -> com.google.gerrit.server.util.ThreadLocalRequestContext$1)
  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1018)
    at com.google.gerrit.server.IdentifiedUser.getAccountDiffPreference(IdentifiedUser.java:273)
    at com.google.gerrit.httpd.raw.HostPageServlet.doGet(HostPageServlet.java:193)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
    at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
    at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
    at com.google.gerrit.httpd.auth.openid.XrdsFilter.doFilter(XrdsFilter.java:32)
    at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
    at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
    at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:62)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
    at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
    at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:497)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
    at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22.<init>(Unknown Source)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22_Factory_GwtOrm$$23.open(Unknown Source)
    at com.google.gwtorm.jdbc.Database.open(Database.java:122)
    at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:45)
    at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27)
    at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)
    at com.google.gerrit.server.util.ThreadLocalRequestContext$1$$FastClassByGuice$$75e0eb90.invoke(<generated>)
    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:230)
    at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:148)
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:54)
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:46)
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1009)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1059)
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1005)
    ... 42 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
    ... 59 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
    ... 61 more
[2015-04-18 02:11:57,370] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Cannot open ReviewDb
  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70) (via modules: com.google.gerrit.server.config.GerritGlobalModule -> com.google.gerrit.server.util.ThreadLocalRequestContext$1)
  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1018)
    at com.google.gerrit.server.IdentifiedUser.getAccountDiffPreference(IdentifiedUser.java:273)
    at com.google.gerrit.httpd.raw.HostPageServlet.doGet(HostPageServlet.java:193)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
    at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
    at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
    at com.google.gerrit.httpd.auth.openid.XrdsFilter.doFilter(XrdsFilter.java:32)
    at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
    at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
    at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:62)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
    at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
    at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:497)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
    at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22.<init>(Unknown Source)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22_Factory_GwtOrm$$23.open(Unknown Source)
    at com.google.gwtorm.jdbc.Database.open(Database.java:122)
    at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:45)
    at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27)
    at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)
    at com.google.gerrit.server.util.ThreadLocalRequestContext$1$$FastClassByGuice$$75e0eb90.invoke(<generated>)
    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:230)
    at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:148)
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:54)
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:46)
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1009)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1059)
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1005)
    ... 42 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
    ... 59 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
    ... 61 more
[2015-04-18 02:14:16,777] ERROR com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID : Unable to authenticate user "com.google.gerrit.extensions.auth.oauth.OAuthUserInfo@3c87e2e2"
com.google.gerrit.server.account.AccountException: Cannot lookup account 104371949137296892053
    at com.google.gerrit.server.account.AccountManager.lookup(AccountManager.java:95)
    at com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID.authenticateAndRedirect(OAuthSessionOverOpenID.java:126)
    at com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID.login(OAuthSessionOverOpenID.java:104)
    at com.google.gerrit.httpd.auth.openid.OAuthWebFilterOverOpenID.doFilter(OAuthWebFilterOverOpenID.java:95)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
    at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:62)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
    at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:497)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
    at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22.<init>(Unknown Source)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22_Factory_GwtOrm$$23.open(Unknown Source)
    at com.google.gwtorm.jdbc.Database.open(Database.java:122)
    at com.google.gerrit.server.account.AccountManager.lookup(AccountManager.java:86)
    ... 40 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
    ... 45 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
    ... 47 more
[2015-04-18 02:32:57,899] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/?n=25&O=81
com.google.inject.ProvisionException: Unable to provision, see the following errors:

[2015-04-18 05:07:02,997] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/20373/revisions/2abc2a1c9861ad1537ddb5436adc0b61c514daed/files/src%2Fclasses%2FBatchHandler.cls/diff?context=ALL&ignore-whitespace=CHANGED
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Cannot open ReviewDb
  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70) (via modules: com.google.gerrit.server.config.GerritGlobalModule -> com.google.gerrit.server.util.ThreadLocalRequestContext$1)
  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1018)
    at com.google.gerrit.server.ChangeUtil.findChanges(ChangeUtil.java:487)
    at com.google.gerrit.server.change.ChangesCollection.parse(ChangesCollection.java:88)
    at com.google.gerrit.server.change.ChangesCollection.parse(ChangesCollection.java:42)
    at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:229)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
    at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
    at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
    at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:62)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
    at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:497)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
    at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22.<init>(Unknown Source)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22_Factory_GwtOrm$$23.open(Unknown Source)
    at com.google.gwtorm.jdbc.Database.open(Database.java:122)
    at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:45)
    at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27)
    at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)
    at com.google.gerrit.server.util.ThreadLocalRequestContext$1$$FastClassByGuice$$75e0eb90.invoke(<generated>)
    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:230)
    at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:148)
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:54)
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:46)
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1009)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1059)
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1005)
    ... 48 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
    ... 65 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
    ... 67 more
[2015-04-18 05:07:33,248] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/20373/edit
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Cannot open ReviewDb
  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70) (via modules: com.google.gerrit.server.config.GerritGlobalModule -> com.google.gerrit.server.util.ThreadLocalRequestContext$1)
  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1018)
    at com.google.gerrit.server.ChangeUtil.findChanges(ChangeUtil.java:487)
    at com.google.gerrit.server.change.ChangesCollection.parse(ChangesCollection.java:88)
    at com.google.gerrit.server.change.ChangesCollection.parse(ChangesCollection.java:42)
    at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:229)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
    at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
    at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
    at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:62)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
    at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
    at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:497)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
    at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22.<init>(Unknown Source)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22_Factory_GwtOrm$$23.open(Unknown Source)
    at com.google.gwtorm.jdbc.Database.open(Database.java:122)
    at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:45)
    at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27)
    at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)
    at com.google.gerrit.server.util.ThreadLocalRequestContext$1$$FastClassByGuice$$75e0eb90.invoke(<generated>)
    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:230)
    at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:148)
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:54)
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:46)
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1009)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1059)
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1005)
    ... 48 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
    ... 65 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
    ... 67 more


Here is my server hardware information

Linux distribution: Red Hat Enterprise Linux Server release 6.5

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvde1      493G   64G  424G  13% /

Hardware location: Amazon EC2 instance

Model        vCPU    Mem (GiB)    SSD Storage  (GB)
c3.4xlarge    16           30               2 x 160

FYI: I am using H2 database (default with server)

To make production stable, i can't even downgrade my production environment..

Can someone please help me ASAP for getting out of this situation..

Thanks & Regards
Anjan

[Luca Milanesio]

It seems that everything is related to your DB connection pool getting exhausted.

Have you traced the DB connections usage to see if increases over time? How many connections you have in your pool?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Pursehouse]

I see javamelody in the stack trace. Are you using the javamelody plugin?  If so, is it rebuilt against the 2.11 API? Does the error go away if the plugin is removed or disabled?

[Anjan Tummalapalli]

Hi David / Luca, Thanks for quick reply..

Luca, May i know how to check database connection pool size?

David, I have tried by disabling javamelody plugin and rebased patchset, this time rebase event itself is not getting completed. Previously event is completed and getting struck, this time event also not completed and struck..

Gettig this error in error_log

[2015-04-19 00:55:54,718] ERROR com.google.gerrit.server.git.ReloadSubmitQueueOp : Cannot reload MergeQueue

com.google.gwtorm.server.OrmException: Cannot open database connection
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
    at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22.<init>(Unknown Source)
    at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22_Factory_GwtOrm$$23.open(Unknown Source)
    at com.google.gwtorm.jdbc.Database.open(Database.java:122)

    at com.google.gerrit.server.util.ManualRequestContext.<init>(ManualRequestContext.java:37)
    at com.google.gerrit.server.util.OneOffRequestContext.open(OneOffRequestContext.java:48)
    at com.google.gerrit.server.git.ReloadSubmitQueueOp.run(ReloadSubmitQueueOp.java:54)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
    at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)

    ... 15 more

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

    ... 17 more

FYI: Javamelody plugin showing version as v2.11-rc1-89-g6b498cd

[Matthias Sohn]

On Sun, Apr 19, 2015 at 7:05 AM, Anjan Tummalapalli <anjan.tum...@gmail.com> wrote:

Hi David / Luca, Thanks for quick reply..

Luca, May i know how to check database connection pool size?

the connection pool configuration is described here

https://gerrit-documentation.storage.googleapis.com/Documentation/2.11/config-gerrit.html#database

[Luca Milanesio]

Hi Anjan,

Have you checked on your gerrit.config?

Luca

Sent from my iPhone

[Anjan Tummalapalli]

Here is my gerrit.conf

[gerrit]
        basePath = git
        canonicalWebUrl = http://gerrit.gainsight.com:8080/
[database]
        type = h2
        database = db/ReviewDB
[auth]
        type = OPENID
        trustedOpenID=^.*$
[sendemail]
        smtpServer = localhost
[container]
        user = gerrit2
        javaHome = /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
[sshd]
        listenAddress = *:29418
        threads = 100
        loginGraceTime = 120s
        batchThreads = 5
        commandStartThreads = 5
        maxConnectionsPerUser = 512
[httpd]
        listenUrl = http://*:8080/
        maxThreads = 2000
        maxQueued = 0
        maxWait = 120s
[cache]
        directory = cache
[commentlink "changeid"]
        match = (I[0-9a-f]{8,40})
        link = "#q,$1,n,z"
[commentlink "jira"]
        match = ([A-Z0-9]+-\\d+)
        link = https://jbarasoftware.jira.com/browse/$1
[index]
        type = LUCENE
[gitweb]
        cgi = /var/www/git/gitweb.cgi
        url = http://gerrit.gainsight.com:8080/gitweb
[plugins]
        allowRemoteAdmin = true
[plugin "gerrit-oauth-provider-google-oauth"]
        client-id = <client id>
        client-secret = <client secret>
        link-to-existing-openid-accounts = true

[Anjan Tummalapalli]

Hi Luca,

I tried to do fresh installation of v2.11 in another directory by copyingbelow ones

db (ReviewDB.h2.db and ReviewDB.trace.db)

git respositories

etc directory contents

This time all features working fine but not recognizing me as admin. And lost all groups in older one.

How to retain all of them from old setup to new one?

But seeing this error more times in error_log

[2015-04-19 13:39:26,352] WARN  com.google.gerrit.sshd.GerritServerSession : Exception caught
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:302)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:45)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:694)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

how can we resolve this?

[Anjan Tummalapalli]

I am listed in Administrators group but not getting admin permissions..

refs/meta/config of all projects may be not copied.. how can we get that back?

And also observed that when we are trying to query in search tab, it is not auto suggesting keywords like project, branch, owner etc.. but listing its values like projects names and users email ids..    Is this expected or wrong from my side?

[Anjan Tummalapalli]

One more observation:

I tried to remove All-Projects.git and All-Users.git in new one and copied from old setup, got admin permissions and groups back but facing performance issues what i said initially like inline edit and rebase problems..

So i reset All-Projects.git and All-Users.git projects back what i got from fresh setup, admin permissions gone but no performance issues found, everytrhing working fine..

Is there any link with these projects?

[lucamilanesio]

One thing pops-up immediately: database.type = h2.

I think it was quite clear from the Documentation pages that H2 is intended to be used for test/PoC set-up and not really for production.

I would opt for PostgreSQL or MySQL if I were you ;-)

Luca.

[Matthias Sohn]

On Sun, Apr 19, 2015 at 6:40 PM, Anjan Tummalapalli <anjan.tum...@gmail.com> wrote:

Here is my gerrit.conf

[gerrit]
        basePath = git
        canonicalWebUrl = http://gerrit.gainsight.com:8080/
[database]
        type = h2
        database = db/ReviewDB

you didn't configure database.poolLimit hence your DB connection pool uses the default of 8.

This means you can't serve more than 8 concurrent requests as Gerrit typically needs one

database connection per concurrent request. But you allow sshd to server 100 concurrent threads

and httpd even 2000 (which seems huge). When the load of your server reaches 8 concurrent

requests it will still accept more concurrent requests without queuing since both daemons

have much higher thread limits but they can't proceed since the db connection pool is exhausted.

The documentation for the configuration option "database.poolLimit"  says:

"This limit must be several units higher than the total number of httpd and sshd threads as

some request processing code paths may need multiple connections."

I think you should reduce the thread limits for both sshd and httpd and adjust the database.poolLimit

according to the above rule.

 

[auth]
        type = OPENID
        trustedOpenID=^.*$
[sendemail]
        smtpServer = localhost
[container]
        user = gerrit2
        javaHome = /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
[sshd]
        listenAddress = *:29418
        threads = 100

this seems too high, the documentation says this defaults to 1.5 times the

number of CPUs you have, do you have 60 cores available ?

use the ssh command gerrit show-queue to check which tasks are currently executed

https://gerrit-documentation.storage.googleapis.com/Documentation/2.11/cmd-show-queue.html

        loginGraceTime = 120s
        batchThreads = 5
        commandStartThreads = 5
        maxConnectionsPerUser = 512

why does a single user need 512 connections in parallel ?

 

[httpd]
        listenUrl = http://*:8080/
        maxThreads = 2000

this limit seems too high, I doubt that your server can serve 2000 concurrent http requests.

If httpd.maxThreads and sshd.threads are too large the server accepts more concurrent

requests than the available CPUs can handle and it's going to be overloaded.

 

        maxQueued = 0
        maxWait = 120s
[cache]
        directory = cache

[commentlink "changeid"]
        match = (I[0-9a-f]{8,40})
        link = "#q,$1,n,z"
[commentlink "jira"]
        match = ([A-Z0-9]+-\\d+)
        link = https://jbarasoftware.jira.com/browse/$1
[index]
        type = LUCENE
[gitweb]
        cgi = /var/www/git/gitweb.cgi
        url = http://gerrit.gainsight.com:8080/gitweb
[plugins]
        allowRemoteAdmin = true
[plugin "gerrit-oauth-provider-google-oauth"]
        client-id = <client id>
        client-secret = <client secret>
        link-to-existing-openid-accounts = true

you should consider to increase the JGit buffer cache using option "core.packedGitLimit" (default is 10MB).

JGit loves memory so give it enough, rule of thumb is use half of maxHeap size. Best the JGit buffer cache is

large enough to fit the total size of all repositories used in parallel.

Are you running git gc regularily ?

-Matthias

[Anjan Tummalapalli]

Hi Matthias,

Here is the changed gerrit.config

[gerrit]
        basePath = git
        canonicalWebUrl = http://gerrit.gainsight.com:8080/
[database]
        type = h2
        database = db/ReviewDB

        poolLimit = 500
[auth]
        type = OPENID
        trustedOpenID = ^.*$

[sendemail]
        smtpServer = localhost
[container]
        user = gerrit2
        javaHome = /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre

        heapLimit = 1g
[core]
        packedGitLimit = 2g

[sshd]
        listenAddress = *:29418

        threads = 75

        loginGraceTime = 120s
        batchThreads = 5
        commandStartThreads = 5

        maxConnectionsPerUser = 10

[httpd]
        listenUrl = http://*:8080/
        maxThreads = 200

        maxQueued = 0
        maxWait = 120s
[cache]
        directory = cache
[commentlink "changeid"]
        match = (I[0-9a-f]{8,40})
        link = "#q,$1,n,z"
[commentlink "jira"]
        match = ([A-Z0-9]+-\\d+)
        link = https://jbarasoftware.jira.com/browse/$1
[index]
        type = LUCENE
[gitweb]
        cgi = /var/www/git/gitweb.cgi
        url = http://gerrit.gainsight.com:8080/gitweb
[plugins]
        allowRemoteAdmin = true
[plugin "gerrit-oauth-provider-google-oauth"]
        client-id = <client id>
        client-secret = <client secret>
        link-to-existing-openid-accounts = true

I restarted gerrit after that, and tried to do inline edit for a change. Again it is getting struck showing 'working...' continuously.

No errors / exceptions observed in error_log

Do i need to do anything else?

[Anjan Tummalapalli]

I am running git gc for all projects daily once with automated job in jenkins..

[Matthias Sohn]

create some thread dumps, the stack traces in the thread dumps should reveal what's going on. 

-Matthias

[Anjan Tummalapalli]

Here is the error logs i captured from error_log

[2015-04-20 01:56:51,959] WARN  com.google.gerrit.sshd.GerritServerSession : Exception caught

java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:302)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:45)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:694)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

[2015-04-20 01:56:52,059] WARN  org.eclipse.jetty.servlet.ServletHandler : /logout

com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error in custom provider, java.lang.IllegalStateException: Response is committed
  at com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID.class(OAuthSessionOverOpenID.java:53)
  while locating com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID

1 error
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1018)

        at com.google.gerrit.httpd.auth.openid.OAuthOverOpenIDLogoutServlet.doLogout(OAuthOverOpenIDLogoutServlet.java:53)
        at com.google.gerrit.httpd.HttpLogoutServlet.doGet(HttpLogoutServlet.java:86)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)

        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)

        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException: Response is committed
        at org.eclipse.jetty.server.Request.getSession(Request.java:1399)
        at org.eclipse.jetty.server.Request.getSession(Request.java:1377)
        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)
        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)
        at com.google.inject.servlet.ServletScopes$3$1.get(ServletScopes.java:151)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)

        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1009)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1059)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1005)

        ... 37 more
[2015-04-20 01:56:52,062] WARN  org.eclipse.jetty.server.HttpChannel : /logout

com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error in custom provider, java.lang.IllegalStateException: Response is committed
  at com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID.class(OAuthSessionOverOpenID.java:53)
  while locating com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID

1 error
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1018)

        at com.google.gerrit.httpd.auth.openid.OAuthOverOpenIDLogoutServlet.doLogout(OAuthOverOpenIDLogoutServlet.java:53)
        at com.google.gerrit.httpd.HttpLogoutServlet.doGet(HttpLogoutServlet.java:86)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)

        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)

        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException: Response is committed
        at org.eclipse.jetty.server.Request.getSession(Request.java:1399)
        at org.eclipse.jetty.server.Request.getSession(Request.java:1377)
        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)
        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:238)
        at com.google.inject.servlet.ServletScopes$3$1.get(ServletScopes.java:151)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)

        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1009)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1059)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1005)

        ... 37 more
[2015-04-20 01:56:52,063] WARN  org.eclipse.jetty.server.HttpChannel : Could not send response error 500: com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error in custom provider, java.lang.IllegalStateException: Response is committed
  at com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID.class(OAuthSessionOverOpenID.java:53)
  while locating com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID

1 error


[2015-04-20 02:06:28,669] WARN  org.eclipse.jetty.server.HttpChannel : Commit failed
java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
        at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:161)
        at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

[2015-04-20 02:06:28,671] WARN  org.eclipse.jetty.servlet.ServletHandler : /login/%23%2F
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
        at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:136)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:378)
        at java.io.OutputStream.write(OutputStream.java:75)
        at com.google.gerrit.httpd.auth.openid.LoginForm.sendHtml(LoginForm.java:324)
        at com.google.gerrit.httpd.auth.openid.LoginForm.sendForm(LoginForm.java:312)
        at com.google.gerrit.httpd.auth.openid.LoginForm.doGet(LoginForm.java:133)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)

        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)

        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
        at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:161)
        at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        ... 1 more

[2015-04-20 02:06:28,673] WARN  org.eclipse.jetty.server.HttpChannel : Could not send response error 500: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
[2015-04-20 02:07:29,380] ERROR com.google.gerrit.httpd.auth.openid.OAuthSessionOverOpenID : Illegal request state 'DJ-NCr1fqJvkfX_EPewz0GBXu2jvHfZypZFn' on OAuthProtocol OAuthSession [token=null, user=null]

[David Ostrovsky]

You can ignore this failure. This happens, when you you restart Gerrit

and try to logout previous active session: [1].

[1] https://github.com/davido/gerrit-oauth-provider/issues/9

[Anjan Tummalapalli]

Can anyone help in this observation:

I tried to remove All-Projects.git and All-Users.git in new one and copied from old setup, got admin permissions and groups back but facing performance issues what i said initially like inline edit and rebase problems..

So i reset All-Projects.git and All-Users.git projects back what i got from fresh setup, admin permissions gone but no performance issues found, everytrhing working fine..

I used same gerrit.config, db and same environment full but working without any issues / errors

Is there any link with these projects?

[Jon Bettcher]

I'm experiencing a performance issue with the 2.11 release as well, but just on change submission.  I'm not entirely sure if it is the same as Anjan's problem, but I thought I would post here for a data point.

What I'm experiencing is that the submit phase of a change causes gerrit to spin on the CPU quite a bit.  I find that pushing a new change for review or publishing a new patchset after an edit within the browser are both fast.  If the change is submitted either a) by the web submit button or b) by push-closing directly to refs/heads/master, either way the CPU use seems much greater than Gerrit 2.10.

Configuration:

Local gerrit instance, Mysql reviewdb (poolLimit 64), Java 7, 2.5g heap.

Gerrit 2.11 upgrade completed with a site init, full rm -r of cache/* and index/*, then a gerrit reindex.

A test repo with working copy checked out.  'origin' should be the gerrit 2.11 server under test.

The repo that demonstrates this submit-slowness has the following object counts (as measured from gerrit's copy):

count: 8702

size: 52744

in-pack: 1016484

packs: 5

size-pack: 1210287

prune-packable: 0

garbage: 22

size-garbage: 4371

Test setup:

git fetch origin master && git co FETCH_HEAD

echo "another test" >> foofile.txt && git add foofile.txt && git commit -m "test push again" && git push origin HEAD:refs/for/master

<now get ready to profile>

git push origin HEAD:refs/heads/master

My result:

$ time git push origin HEAD:refs/heads/master

Total 0 (delta 0), reused 0 (delta 0)

remote: Processing changes: closed: 1, refs: 1, done

To ssh://gerrit.local:29418/test-repo

   9ea5f70..15bb965  HEAD -> master

real    0m31.499s

user    0m0.055s

sys     0m0.097s

The final push will eventually close the change and email comments, but it does take some time.  Here's a screenshot of jvisualvm's CPU profiling while the submit is running:

It seems to be a fair amount of thrashing going on with the lucene index.  If profile snapshots would help as well, I can send those along.

I'll keep digging on my side, but just thought I'd share in case anyone else has noticed the same.

[David Pursehouse]

Performance degradation of the submit button was also reported in issue
3300 [1].

[1] https://code.google.com/p/gerrit/issues/detail?id=3300

On 04/21/2015 07:28 AM, Jon Bettcher wrote:
> I'm experiencing a performance issue with the 2.11 release as well, but
> just on change submission. I'm not entirely sure if it is the same as
> Anjan's problem, but I thought I would post here for a data point.
>
> What I'm experiencing is that the submit phase of a change causes gerrit
> to spin on the CPU quite a bit. I find that pushing a new change for
> review or publishing a new patchset after an edit within the browser are
> both fast. If the change is submitted either a) by the web submit
> button or b) by push-closing directly to refs/heads/master, either way
> the CPU use seems much greater than Gerrit 2.10.
>

> *Configuration:*

> Local gerrit instance, Mysql reviewdb (poolLimit 64), Java 7, 2.5g heap.
> Gerrit 2.11 upgrade completed with a site init, full rm -r of cache/*
> and index/*, then a gerrit reindex.
>
> A test repo with working copy checked out. 'origin' should be the
> gerrit 2.11 server under test.
> The repo that demonstrates this submit-slowness has the following object
> counts (as measured from gerrit's copy):
> count: 8702
> size: 52744
> in-pack: 1016484
> packs: 5
> size-pack: 1210287
> prune-packable: 0
> garbage: 22
> size-garbage: 4371
>

> *Test setup:*

> git fetch origin master && git co FETCH_HEAD
> echo "another test" >> foofile.txt && git add foofile.txt && git commit
> -m "test push again" && git push origin HEAD:refs/for/master
> <now get ready to profile>
> git push origin HEAD:refs/heads/master
>

> *My result:*

> $ time git push origin HEAD:refs/heads/master
> Total 0 (delta 0), reused 0 (delta 0)
> remote: Processing changes: closed: 1, refs: 1, done
> To ssh://gerrit.local:29418/test-repo
> 9ea5f70..15bb965 HEAD -> master
>
> real 0m31.499s
> user 0m0.055s
> sys 0m0.097s
>
> The final push will eventually close the change and email comments, but
> it does take some time. Here's a screenshot of jvisualvm's CPU
> profiling while the submit is running:
>

> <mailto:repo-discuss%2Bunsu...@googlegroups.com>

> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the
> Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to repo-discuss...@googlegroups.com

> <mailto:repo-discuss...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com

> <mailto:repo-discuss%2Bunsu...@googlegroups.com>

> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to repo-discuss...@googlegroups.com

> <mailto:repo-discuss...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com

> <mailto:repo-discuss...@googlegroups.com>.

[Anjan Tummalapalli]

Hello Team,

Here is an update for issue in my gerrit instance

Issue is resolved by migrating database from H2 to Postgresql. Not seeing any errors now..

Thanks All who supported me in resolving the issue.. My production server is up and running now..

But somethings missing, here are my observations

-> Replication is not happening by default to all branches in a project. Only happening to specific branch when a new change is merged in respective branch in respective project.

  In previous version, it forcefully updates all branches in all projects..

-> Missing auto suggestion of keywords in "search term" text area.

Any suggestions on these missing topics..?

Thanks & Regards

Anjan

[Jon Bettcher]

Hi all -

I tracked down my performance issues on submit to the mergability checker running as part of the request thread rather than in the background.  Not sure if something changed with the defaults for 2.11, but this change in my gerrit.config caused submits/ref-updates to be fast again:

[index]

type = LUCENE

threads = 4

When the threads are unspecified, gerrit defaults to 1, which seems to cause the RecieveCommits task to wait for the result of the various index and mergability-checker futures.

My project has a fair number of open changes, so a ref-updated operation still uses up the same CPU resources, but at least this operation is deferred rather than blocking receive commits this way. 

Hope that helps someone else out there.