Startup error after Gerrit was upgraded to version 3.2.14

[adrien...@gmail.com]

Hi,

We upgraded our Gerrit from 3.1.8 to 3.2.14 successfully.

However after indexing (took five hours), an attempt to start Gerrit for the first time shows the below entries in the error log and Gerrit would not start.

Any help from the experts will be appreciated at this point:

[2021-12-14T21:56:10.454-0500] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon
java.lang.IllegalStateException: Cannot start HTTP daemon
        at com.google.gerrit.pgm.http.jetty.JettyServer$Lifecycle.start(JettyServer.java:109)
        at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:95)
        at com.google.gerrit.pgm.Daemon.start(Daemon.java:372)
        at com.google.gerrit.pgm.Daemon.run(Daemon.java:275)
        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:224)
        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:120)
        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:65)
        at Main.main(Main.java:28)
Caused by: com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.NoSuchMethodError: 'void com.google.gerrit.server.events.EventBroker.<init>(com.google.gerrit.server.plugincontext.PluginSetContext, com.google.gerrit.server.plugincontext.PluginSetContext, com.google.gerrit.server.permissions.PermissionBackend, com.google.gerrit.server.project.ProjectCache, com.google.gerrit.server.notedb.ChangeNotes$Factory)'
  at com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedAwareEventBroker.<init>(Unknown Source)
  while locating com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedAwareEventBroker
  while locating com.google.gerrit.server.events.EventDispatcher
    for the 1st parameter of com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedEventHandler.<init>(Unknown Source)
>(Unknown Source)
  at com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedEventHandler.class(Unknown Source)
  while locating com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedEventHandler
    for the 1st parameter of com.ericsson.gerrit.plugins.highavailability.forwarder.rest.EventRestApiServlet.<init>(Unknown Source)
  at com.ericsson.gerrit.plugins.highavailability.forwarder.rest.EventRestApiServlet.class(Unknown Source)
  while locating com.ericsson.gerrit.plugins.highavailability.forwarder.rest.EventRestApiServlet

1 error
        at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:226)
        at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1097)
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1126)
        at com.google.inject.servlet.ServletDefinition.init(ServletDefinition.java:110)
        at com.google.inject.servlet.ManagedServletPipeline.init(ManagedServletPipeline.java:80)
        at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:102)
        at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:232)
        at com.google.gerrit.httpd.plugins.HttpPluginServlet.load(HttpPluginServlet.java:191)
        at com.google.gerrit.httpd.plugins.HttpPluginServlet.install(HttpPluginServlet.java:171)
        at com.google.gerrit.httpd.plugins.HttpPluginServlet.init(HttpPluginServlet.java:151)
        at com.google.inject.servlet.ServletDefinition.init(ServletDefinition.java:119)
        at com.google.inject.servlet.ManagedServletPipeline.init(ManagedServletPipeline.java:80)
        at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:102)
        at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:232)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:140)
        at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:731)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
        at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:755)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:423)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.Server.doStart(Server.java:387)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at com.google.gerrit.pgm.http.jetty.JettyServer$Lifecycle.start(JettyServer.java:95)
        ... 12 more
Caused by: java.lang.NoSuchMethodError: 'void com.google.gerrit.server.events.EventBroker.<init>(com.google.gerrit.server.plugincontext.PluginSetContext, com.google.gerrit.server.plugincontext.PluginSetContext, com.google.gerrit.server.permissions.PermissionBackend, com.google.gerrit.server.project.ProjectCache, com.google.gerrit.server.notedb.ChangeNotes$Factory)'
        at com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedAwareEventBroker.<init>(ForwardedAwareEventBroker.java:36)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at com.google.inject.internal.DefaultConstructionProxyFactory$ReflectiveProxy.newInstance(DefaultConstructionProxyFactory.java:126)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1094)
        ... 45 more

Regards,

Adrien

[Matthias Sohn]

On Wed, Dec 15, 2021 at 3:05 PM adrien...@gmail.com <adrien...@gmail.com> wrote:

Hi,

We upgraded our Gerrit from 3.1.8 to 3.2.14 successfully.

However after indexing (took five hours), an attempt to start Gerrit for the first time shows the below entries in the error log and Gerrit would not start.

Any help from the experts will be appreciated at this point:

did you also upgrade all Gerrit plugins and lib modules you have installed to their latest 3.2 build ?

Note that 3.2 is now EOL [1], you can still find the archived plugin builds here

https://archive-ci.gerritforge.com/view/Plugins-stable-3.2/job/

[1] https://gerrit-review.googlesource.com/c/homepage/+/317484

-Matthias

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/9520e74d-c56a-402f-815c-97c33da4d715n%40googlegroups.com.

[adrien...@gmail.com]

Hi Matthias,

Yes we did upgraded all the plugins as below.

*** Plugins

***

Installing plugins.

Install plugin codemirror-editor version v3.2.14 [Y/n]?

codemirror-editor v3.1.8 is already installed, overwrite it [Y/n]?

Updated codemirror-editor to v3.2.14

Install plugin commit-message-length-validator version v3.2.14 [Y/n]?

commit-message-length-validator v3.1.8 is already installed, overwrite it [Y/n]?

Updated commit-message-length-validator to v3.2.14

Install plugin delete-project version v3.2.14 [Y/n]?

delete-project v3.1.8 is already installed, overwrite it [Y/n]?

Updated delete-project to v3.2.14

Install plugin download-commands version v3.2.14 [Y/n]?

download-commands v3.1.8 is already installed, overwrite it [Y/n]?

Updated download-commands to v3.2.14

Install plugin gitiles version v3.2.14 [Y/n]?

gitiles v3.1.8 is already installed, overwrite it [Y/n]?

Updated gitiles to v3.2.14

Install plugin hooks version v3.2.14 [Y/n]?

hooks v3.1.8 is already installed, overwrite it [Y/n]?

Updated hooks to v3.2.14

Install plugin plugin-manager version v3.2.14 [Y/n]?

plugin-manager v3.1.8 is already installed, overwrite it [Y/n]?

Updated plugin-manager to v3.2.14

Install plugin replication version v3.2.14 [Y/n]?

replication v3.1.8 is already installed, overwrite it [Y/n]?

Updated replication to v3.2.14

Install plugin reviewnotes version v3.2.14 [Y/n]?

reviewnotes v3.1.8 is already installed, overwrite it [Y/n]?

Updated reviewnotes to v3.2.14

Install plugin singleusergroup version v3.2.14 [Y/n]?

singleusergroup v3.1.8 is already installed, overwrite it [Y/n]?

Updated singleusergroup to v3.2.14

Install plugin webhooks version v3.2.14 [Y/n]?

webhooks v3.1.8 is already installed, overwrite it [Y/n]?

Updated webhooks to v3.2.14

Initializing plugins.

*** high-availability Plugin

***

 

Configure high-availability    [y/N]? N

 

Migrating data to schema 182 ...

Found a total of 161 zombie draft refs in All-Users repo.

Cleanup percentage = 100

Number of zombie refs to be cleaned = 161

Deleted 161/161 zombie draft refs (10 seconds)

Migrating data to schema 183 ...

Regards,

Adrien

[Matthias Sohn]

I don't see an entry for the high-availability plugin in this log, maybe you didn't upgrade it ? 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/3e7b1d4a-6d50-42f2-96fd-0b87bc1c96den%40googlegroups.com.

[adrien...@gmail.com]

Hi Matthias,

Yes I did not updated it as shown below.

Should I just update it by downloading the 3.2.14 version of that plugin and once again attempt to start Gerrit?

*** high-availability Plugin

***

 

Configure high-availability    [y/N]? N

Regards,

Adrien

[Matthias Sohn]

On Wed 15. Dec 2021 at 19:03, adrien...@gmail.com <adrien...@gmail.com> wrote:

Hi Matthias,

Yes I did not updated it as shown below.

Should I just update it by downloading the 3.2.14 version of that plugin and once again attempt to start Gerrit?

Yes

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f3c5b871-60a7-4dee-b92b-0d8140815dcbn%40googlegroups.com.

[Nasser Grainawi]

On Dec 15, 2021, at 7:05 AM, adrien...@gmail.com <adrien...@gmail.com> wrote:

Hi,

We upgraded our Gerrit from 3.1.8 to 3.2.14 successfully.

However after indexing (took five hours), an attempt to start Gerrit for the first time shows the below entries in the error log and Gerrit would not start.

I know this wasn’t the point of your thread, but can I ask how you ran reindexing? What command did you use, # of threads, settings in gerrit.config, etc? Can you share the caches output from the end of the reindexing log?

There were very significant offline reindex performance improvements in 3.2.14 [1] and you should be able to complete it in much less time.

[1] https://www.gerritcodereview.com/3.2.html#3214

[adrien...@gmail.com]

Hi Matthias,

I was able to start Gerrit after updating the ha plugin compatible with v3.2.14.

Thanks.

Adrien

[adrien...@gmail.com]

Hi Nas,

I just actually used the command -->  java -jar gerrit.war reindex -d $GERRIT_SITE

In conjunction with these common settings:

[index]
        type = lucene

[auth]
        type = LDAP
        gitBasicAuthPolicy = HTTP

[container]
        heapLimit = 4g

[sshd]
        listenAddress = *:29418
        threads = 32
        waitTimeout = 30s

[cache]
        directory = cache

Regards,

Adrien 

[Luca Milanesio]

On 15 Dec 2021, at 18:54, adrien...@gmail.com <adrien...@gmail.com> wrote:

Hi Matthias,

I was able to start Gerrit after updating the ha plugin compatible with v3.2.14.

The plugins expose in their manifest the “expected” version of Gerrit API: I believe we should check that at startup and send a *clearer* message to the Gerrit admin.

WDYT?

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/c9964a80-3c5e-49e0-bf5a-4133b20b2f8bn%40googlegroups.com.

[Nasser Grainawi]

On Dec 15, 2021, at 11:58 AM, adrien...@gmail.com <adrien...@gmail.com> wrote:

Hi Nas,

Please avoid top-posting.

I just actually used the command -->  java -jar gerrit.war reindex -d $GERRIT_SITE

In conjunction with these common settings:

[index]
        type = lucene

[auth]
        type = LDAP
        gitBasicAuthPolicy = HTTP

[container]
        heapLimit = 4g

[sshd]
        listenAddress = *:29418
        threads = 32
        waitTimeout = 30s

[cache]
        directory = cache

Ok, I’m a bit surprised you’re using mostly defaults there. If you’re running a Gerrit server with enough data to take 5 hours to reindex, you can probably benefit from adjusting some settings.

If the 5 hour time is a problem for you, you’ll want to change those defaults. I would especially look at the caches output from the end of your reindexing log and ensure you’ve configured diskLimit values for each of the caches with a D in the first column to at least the size of the Space column. After doing that, we were able to reduce reindex time by 70% when we reused the cache/ dir for an offline reindex after populating it with a “test” reindex that used a snapshot of production data. We covered more on this in the recent user summit talk: https://youtu.be/V5QVf4DWUZ8

Nasser

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/7b7c7ce4-75c4-4f39-9fab-d058f3a3afddn%40googlegroups.com.