com.gerritforge.gerrit.globalrefdb.validation.SharedRefDbConfiguration Error during Gerrit HA setup

647 views
Skip to first unread message

anish....@gmail.com

unread,
Jan 2, 2023, 12:29:06 PM1/2/23
to Repo and Gerrit Discussion
Hello All,

Trying to setup a Gerrit High Availability setup using the Active-Active configuration using the High Availability Plugin. Document followed is  

https://gerrit.googlesource.com/plugins/high-availability/+/refs/heads/stable-3.6/src/main/resources/Documentation/config.md

OS Ubuntu 20.04 
java --version
openjdk 11.0.17 2022-10-18
OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu220.04)
OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu220.04, mixed mode, sharing)

As Per document pre-requisites: 

  • Gerrit v2.14.20 or later --> 3.6
  • Externally mounted filesystem shared among the cluster --> NFS share of /git
  • ReviewDb on an external DataBase Server --> External Postgress Database
  • Load-balancer (HAProxy or similar) --> haproxy


Initialized the gerrit with the external DB and required configuration.

gerrit_home/etc/gerrit.config
*********************
[gerrit]
        basePath = /fsxtest/shared/git
        canonicalWebUrl = http://awsmblx108as605.ad.com:8080
        serverId = 82c23693-4ff8-4232-80ea-c18ac9ad0499
[container]
        javaOptions = "-Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance"
        javaOptions = "-Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance"
        user = gerrit
        javaHome = /usr/lib/jvm/java-11-openjdk-amd64
[index]
        type = lucene
[auth]
        type = LDAP
        gitBasicAuthPolicy = HTTP
        userNameCaseInsensitive = true
[ldap]
        server = ldaps://ldapbd.ad.test.com
        username = gerrit
        accountBase = DC=ad,DC=test,DC=com
        groupBase = DC=ad,DC=test,DC=com
[receive]
        enableSignedPush = false
[sendemail]
        smtpServer = smtpbd.test.com
        smtpServerPort = 25
        smtpUser = gerrit
[sshd]
        listenAddress = *:29418
[httpd]
        listenUrl = proxy-https://*:8080/
[cache]
        directory = cache
[accountPatchReviewDb]
        url = jdbc:postgresql://hiawsmb-rds.ap-south-1.rds.amazonaws.com:5432/gerritha?user=ger&password=pass
********************
Create a high-availability.config with the below contents:

**************
[main]
  sharedDirectory = /fsxtest/shared/git

[peerInfo]
  strategy = static

[peerInfo "static"]
  url = http://awsmblx108as606.ad.test.com:8080

[autoReindex]
  enabled = false

[http]
  user = gerrit
  password = OB6EmDPfqhN3NDdclg

[healthcheck]
  enable = true

**************

When we start the Server it fails with the below Error only when the ha plugin is loaded in the plugin directory.

**************************************
[2023-01-02T13:44:00.358Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin gitiles, version v3.6.3
[2023-01-02T13:44:00.427Z] [main] ERROR com.google.gerrit.pgm.Daemon : Thread main threw exception
com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lcom/gerritforge/gerrit/globalrefdb/validation/SharedRefDbConfiguration;
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
        at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4941)
        at com.google.inject.internal.FailableCache.get(FailableCache.java:54)
        at com.google.inject.internal.ConstructorInjectorStore.get(ConstructorInjectorStore.java:49)
        at com.google.inject.internal.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:148)
        at com.google.inject.internal.InjectorImpl.initializeJitBinding(InjectorImpl.java:606)
        at com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:943)
        at com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:863)
        at com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:301)
        at com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:224)
        at com.google.inject.internal.InjectorImpl.getProviderOrThrow(InjectorImpl.java:1092)
        at com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1121)
        at com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1086)
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1138)
        at com.google.gerrit.server.plugins.ServerPlugin.startPlugin(ServerPlugin.java:212)
        at com.google.gerrit.server.plugins.ServerPlugin.start(ServerPlugin.java:182)
        at com.google.gerrit.server.plugins.PluginLoader.runPlugin(PluginLoader.java:512)
        at com.google.gerrit.server.plugins.PluginLoader.rescan(PluginLoader.java:426)
        at com.google.gerrit.server.plugins.PluginLoader.start(PluginLoader.java:331)
        at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:95)
        at com.google.gerrit.pgm.Daemon.start(Daemon.java:386)
        at com.google.gerrit.pgm.Daemon.run(Daemon.java:285)
        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:224)
        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:120)
        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:65)
        at Main.main(Main.java:29)
Caused by: java.lang.NoClassDefFoundError: Lcom/gerritforge/gerrit/globalrefdb/validation/SharedRefDbConfiguration;
        at java.base/java.lang.Class.getDeclaredFields0(Native Method)
        at java.base/java.lang.Class.privateGetDeclaredFields(Class.java:3061)
        at java.base/java.lang.Class.getDeclaredFields(Class.java:2248)
        at com.google.inject.internal.DeclaredMembers.getDeclaredFields(DeclaredMembers.java:40)
        at com.google.inject.spi.InjectionPoint.getDeclaredFields(InjectionPoint.java:800)
        at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:710)
        at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:423)
        at com.google.inject.internal.ConstructorBindingImpl.getInternalDependencies(ConstructorBindingImpl.java:173)
        at com.google.inject.internal.InjectorImpl.getInternalDependencies(InjectorImpl.java:671)
        at com.google.inject.internal.InjectorImpl.cleanup(InjectorImpl.java:628)
        at com.google.inject.internal.InjectorImpl.initializeJitBinding(InjectorImpl.java:614)
        at com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:943)
        at com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:863)
        at com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:301)
        at com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:224)
        at com.google.inject.internal.InjectorImpl.createParameterInjector(InjectorImpl.java:1035)
        at com.google.inject.internal.InjectorImpl.getParametersInjectors(InjectorImpl.java:1022)
        at com.google.inject.internal.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:70)
        at com.google.inject.internal.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:30)
        at com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:38)
        at com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:34)
        at com.google.inject.internal.FailableCache$1.load(FailableCache.java:43)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
        ... 33 more
Caused by: java.lang.ClassNotFoundException: com.gerritforge.gerrit.globalrefdb.validation.SharedRefDbConfiguration
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:904)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
**************************************

It looks to be a defect in 3.3.4 version but is it also an issue with the Open JDK or some configuration issue/

Any leads is greatly apreciated.

Thank you,
Anish

Fabio Ponciroli

unread,
Jan 2, 2023, 12:48:35 PM1/2/23
to anish....@gmail.com, Repo and Gerrit Discussion
Hi Anish,


It is trying to load the SharedRefDb. Try installing the SharedRefDb library [1] in your lib directory (I am assuming you are running Gerrit 3.6).

Let us know if it works.

Thanks,
Ponch

 
It looks to be a defect in 3.3.4 version but is it also an issue with the Open JDK or some configuration issue/

Any leads is greatly apreciated.

Thank you,
Anish

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d8028d44-2c58-4d12-a863-c5f16ad947e9n%40googlegroups.com.

anish....@gmail.com

unread,
Jan 2, 2023, 1:16:43 PM1/2/23
to Repo and Gerrit Discussion
Hello Ponch,

Yes, I just downloaded and added the below files in the lib directory and the server started up just fine.

drwxrwxr-x  2 gerrit gerrit    4096 Jan  2 18:01 .
drwxr-xr-x 14 gerrit gerrit    4096 Jan  2 18:02 ..
-rw-r--r--  1 gerrit gerrit   73632 Nov 17 14:27 global-refdb-3.6.3.jar
-rw-rw-r--  1 gerrit gerrit 1046770 Nov 23 14:31 postgresql-42.5.1.jar
gerrit@awsmblx108as605:/data/gerrit/lib$

The new Logs provide the following information:

***********************
root@awsmblx108as605:/data/gerrit/logs# tail -f error_log
[2023-01-02T18:09:54.344Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SshCommandStart' queue
[2023-01-02T18:09:55.049Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Stream-Worker' queue
[2023-01-02T18:09:55.051Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Interactive-Worker' queue
[2023-01-02T18:09:55.052Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Batch-Worker' queue
[2023-01-02T18:09:55.054Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'MigrateExternalIdCase' queue
[2023-01-02T18:09:55.079Z] [main] WARN  com.google.gerrit.server.config.GitwebCgiConfig : gitweb not installed (no /usr/lib/cgi-bin/gitweb.cgi found)
[2023-01-02T18:09:56.437Z] [main] INFO  org.eclipse.jetty.util.log : Logging initialized @9257ms to org.eclipse.jetty.util.log.Slf4jLog
[2023-01-02T18:09:56.663Z] [main] INFO  com.google.gerrit.server.git.SystemReaderInstaller : Set JGit's SystemReader to read system config from /data/gerrit/etc/jgit.config
[2023-01-02T18:09:56.669Z] [main] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 992m
[2023-01-02T18:09:57.322Z] [main] WARN  com.google.gerrit.server.project.PeriodicProjectListCacheWarmer : project_list cache warmer is disabled
[2023-01-02T18:09:57.326Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /data/gerrit/plugins
[2023-01-02T18:09:57.459Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin codemirror-editor, version v3.6.3
[2023-01-02T18:09:57.538Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin commit-message-length-validator, version v3.6.3
[2023-01-02T18:09:57.655Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin delete-project, version v3.6.3
[2023-01-02T18:09:57.764Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin download-commands, version v3.6.3
[2023-01-02T18:09:57.817Z] [main] INFO  com.google.gerrit.server.config.FileBasedGlobalPluginConfigProvider : No /data/gerrit/etc/gitiles.config; assuming defaults
[2023-01-02T18:09:57.864Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin gitiles, version v3.6.3
[2023-01-02T18:09:57.947Z] [main] INFO  com.ericsson.gerrit.plugins.highavailability.Configuration : Loading configuration from /data/gerrit/etc/high-availability.config
[2023-01-02T18:09:57.993Z] [main] INFO  com.gerritforge.gerrit.globalrefdb.validation.SharedRefDbConfiguration : Loading configuration from /data/gerrit/etc/high-availability.config
[2023-01-02T18:09:58.075Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin high-availability, version v3.3.0-124-gd5fd7d9993
[2023-01-02T18:09:58.165Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin hooks, version v3.6.3
[2023-01-02T18:09:58.268Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin plugin-manager, version v3.6.3
[2023-01-02T18:09:58.274Z] [plugin-manager-preloader] INFO  com.googlesource.gerrit.plugins.manager.OnStartStop : Start-up: pre-loading list of plugins from registry
[2023-01-02T18:09:58.470Z] [main] WARN  com.googlesource.gerrit.plugins.replication.DestinationConfigParser : Replication config does not exist or it's empty; not replicating
[2023-01-02T18:09:58.498Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin replication, version v3.6.3
[2023-01-02T18:09:58.568Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin reviewnotes, version v3.6.3
[2023-01-02T18:09:58.652Z] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6c0c1109[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5d9b28b8[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@41138192]]]] INFO  com.googlesource.gerrit.plugins.deleteproject.fs.RepositoryCleanupTask : Cleaning up expired git repositories...
[2023-01-02T18:09:58.666Z] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6c0c1109[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5d9b28b8[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@41138192]]]] INFO  com.googlesource.gerrit.plugins.deleteproject.fs.RepositoryCleanupTask : Cleaning up expired git repositories... Done
[2023-01-02T18:09:58.711Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin singleusergroup, version v3.6.3
[2023-01-02T18:09:58.804Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin webhooks, version v3.6.3
[2023-01-02T18:09:58.810Z] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "accountDeactivation".
[2023-01-02T18:09:58.837Z] [main] INFO  com.google.gerrit.sshd.SshDaemon : Started Gerrit APACHE-SSHD-2.8.0 on *:29418
[2023-01-02T18:09:58.842Z] [main] INFO  org.eclipse.jetty.server.Server : jetty-9.4.36.v20210114; built: 2021-01-14T16:44:28.689Z; git: 238ec6997c7806b055319a6d11f8ae7564adc0de; jvm 11.0.17+8-post-Ubuntu-1ubuntu220.04
[2023-01-02T18:09:58.897Z] [main] INFO  org.eclipse.jetty.server.session : DefaultSessionIdManager workerName=node0
[2023-01-02T18:09:58.899Z] [main] INFO  org.eclipse.jetty.server.session : No SessionScavenger set, using defaults
[2023-01-02T18:09:58.902Z] [main] INFO  org.eclipse.jetty.server.session : node0 Scavenging every 600000ms
[2023-01-02T18:09:59.045Z] [main] INFO  com.googlesource.gerrit.plugins.gitiles.HttpModule : No /data/gerrit/etc/gitiles.config; assuming defaults
[2023-01-02T18:09:59.075Z] [WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6fa76d4d[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@655f063d[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@6b98f4e6]]]] INFO  com.ericsson.gerrit.plugins.highavailability.websession.file.FileBasedWebSessionCacheCleaner.CleanupTask : Cleaning up expired file based websessions...
[2023-01-02T18:09:59.078Z] [WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6fa76d4d[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@655f063d[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@6b98f4e6]]]] INFO  com.ericsson.gerrit.plugins.highavailability.websession.file.FileBasedWebSessionCacheCleaner.CleanupTask : Cleaning up expired file based websessions...Done
[2023-01-02T18:10:00.086Z] [main] INFO  org.eclipse.jetty.server.handler.ContextHandler : Started o.e.j.s.ServletContextHandler@f777d41{/,null,AVAILABLE}
[2023-01-02T18:10:00.100Z] [main] INFO  org.eclipse.jetty.server.AbstractConnector : Started ServerConnector@359f3401{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
[2023-01-02T18:10:00.101Z] [main] INFO  org.eclipse.jetty.server.Server : Started @12923ms
[2023-01-02T18:10:00.103Z] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.6.3 ready
[2023-01-02T18:10:50.840Z] [plugin-manager-preloader] INFO  com.googlesource.gerrit.plugins.manager.OnStartStop : 71 plugins successfully pre-loaded

*******************************

I presume the logs looks good. 

The real question, why is this not documented in the plugin  configuration ReadMe. Also I'm using a HA plugin and not a multisite plugin..............How is the  global-refdb-3.6.3.jar file even applicable here????

Thank you,
Anish

Fabio Ponciroli

unread,
Jan 2, 2023, 1:30:18 PM1/2/23
to anish....@gmail.com, Repo and Gerrit Discussion
Hi Anish,

Cool! I am glad to hear it worked fine!
 

The real question, why is this not documented in the plugin  configuration ReadMe.

It is a bug that was introduced at some point. The SharedRefDB configuration should be conditionally loaded depending if you are using it or not.
At the moment adding the sharedrefdb library is just a workaround for a proper fix.

Can you please raise an issue for it [1]?

Feel free to contribute with a fix or an update to the documentation.

Thanks,
Ponch

Reply all
Reply to author
Forward
0 new messages