Gerrit Multisite Setup

1,149 views
Skip to first unread message

haridas...@gmail.com

unread,
May 2, 2019, 1:36:42 AM5/2/19
to Repo and Gerrit Discussion
Hi Team,

I am trying to setup Gerrit Multisite Plugin. The plugin documentation says that currently it supports only one active master (RW site) accepting modifications on the Git repositories or the review data. I believe, the health check plugin will be deciding the RW Site based on certain parameters. I assume, connections will be received by HAProxy which intern will validate with the "health check plugin" and redirect the connections to the active RW site. I couldn't find any related configurations or documents. It would be of great help, if someone can point me to the right direction. 

Thanks,
Dipu H

Luca Milanesio

unread,
May 2, 2019, 2:24:18 AM5/2/19
to haridas...@gmail.com, Luca Milanesio, Repo and Gerrit Discussion, Fabio Ponciroli

On 2 May 2019, at 06:04, haridas...@gmail.com wrote:

Hi Team,

I am trying to setup Gerrit Multisite Plugin. The plugin documentation says that currently it supports only one active master (RW site) accepting modifications on the Git repositories or the review data.

Yes, however things are evolving very quickly on the plugins' development and very soon you'll be able to enable multiple writes on any site on any repos.
Tiago has a series of changes for making this possible without any risk of split-brain:

I believe, the health check plugin will be deciding the RW Site based on certain parameters. I assume, connections will be received by HAProxy which intern will validate with the "health check plugin" and redirect the connections to the active RW site.


I couldn't find any related configurations or documents. It would be of great help, if someone can point me to the right direction. 

Hi Dipu,
the plugin's configuration docs can be found at:

and for creating a test site, just use the setup.sh script developed by Tiago at:

Ponch is working on creating a Dockerized test environment also with the following series of changes:

HTH

Luca.


Thanks,
Dipu H

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matthias Sohn

unread,
May 2, 2019, 8:50:08 AM5/2/19
to Luca Milanesio, haridas...@gmail.com, Repo and Gerrit Discussion, Fabio Ponciroli
On Thu, May 2, 2019 at 8:24 AM Luca Milanesio <luca.mi...@gmail.com> wrote:


On 2 May 2019, at 06:04, haridas...@gmail.com wrote:

Hi Team,

I am trying to setup Gerrit Multisite Plugin. The plugin documentation says that currently it supports only one active master (RW site) accepting modifications on the Git repositories or the review data.

Yes, however things are evolving very quickly on the plugins' development and very soon you'll be able to enable multiple writes on any site on any repos.
Tiago has a series of changes for making this possible without any risk of split-brain:

I believe, the health check plugin will be deciding the RW Site based on certain parameters. I assume, connections will be received by HAProxy which intern will validate with the "health check plugin" and redirect the connections to the active RW site.

Exactly, you can find a sample HAProxy configuration at:

I couldn't find any related configurations or documents. It would be of great help, if someone can point me to the right direction. 

Hi Dipu,
the plugin's configuration docs can be found at:

and for creating a test site, just use the setup.sh script developed by Tiago at:

I am trying to set this up based on current gerrit core and the multi-site plugin from stable-2.16 branch.

Zookeeper and Kafka are up and running, Gerrit instance 1 and 2 come up and I can logon as admin.

All the additional plugins load but the multi-site plugin fails to load with a Guice Exception,
interestingly its Kafka consumer anyway seems to connect to Kafka [1].

-Matthias

[1]
[2019-05-02 14:33:09,989] [main] WARN  com.google.gerrit.server.plugins.PluginLoader : Cannot load plugin multi-site
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) No implementation for com.google.gerrit.server.events.EventDispatcher was bound.
  while locating com.google.gerrit.server.events.EventDispatcher
    for the 1st parameter of com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedEventHandler.<init>(ForwardedEventHandler.java:41)
  while locating com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedEventHandler
    for the 1st parameter of com.googlesource.gerrit.plugins.multisite.kafka.router.StreamEventRouter.<init>(StreamEventRouter.java:29)
  while locating com.googlesource.gerrit.plugins.multisite.kafka.router.StreamEventRouter
    for the 4th parameter of com.googlesource.gerrit.plugins.multisite.kafka.consumer.StreamEventSubscriber.<init>(StreamEventSubscriber.java:44)
  at com.google.gerrit.extensions.registration.DynamicSet.bind(DynamicSet.java:108) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule)

2) No implementation for com.google.gerrit.server.events.EventDispatcher was bound.
  while locating com.google.gerrit.server.events.EventDispatcher
    for the 1st parameter of com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedEventHandler.<init>(ForwardedEventHandler.java:41)
  while locating com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedEventHandler
    for the 1st parameter of com.googlesource.gerrit.plugins.multisite.kafka.router.StreamEventRouter.<init>(StreamEventRouter.java:29)
  while locating com.googlesource.gerrit.plugins.multisite.kafka.router.StreamEventRouter
    for the 4th parameter of com.googlesource.gerrit.plugins.multisite.kafka.consumer.CacheEvictionEventSubscriber.<init>(CacheEvictionEventSubscriber.java:44)
  at com.google.gerrit.extensions.registration.DynamicSet.bind(DynamicSet.java:108) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule)

3) A binding to com.googlesource.gerrit.plugins.multisite.validation.MultiSiteRepository$Factory was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.config.FactoryModule.factory(FactoryModule.java:43) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.validation.ValidationModule -> com.google.inject.assistedinject.FactoryModuleBuilder$1)

4) A binding to com.googlesource.gerrit.plugins.multisite.validation.MultiSiteRefDatabase$Factory was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.config.FactoryModule.factory(FactoryModule.java:43) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.validation.ValidationModule -> com.google.inject.assistedinject.FactoryModuleBuilder$1)

5) A binding to com.googlesource.gerrit.plugins.multisite.validation.MultiSiteRefUpdate$Factory was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.config.FactoryModule.factory(FactoryModule.java:43) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.validation.ValidationModule -> com.google.inject.assistedinject.FactoryModuleBuilder$1)

6) A binding to com.googlesource.gerrit.plugins.multisite.validation.MultiSiteBatchRefUpdate$Factory was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.config.FactoryModule.factory(FactoryModule.java:43) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.validation.ValidationModule -> com.google.inject.assistedinject.FactoryModuleBuilder$1)

7) A binding to com.google.gerrit.extensions.registration.DynamicSet<com.googlesource.gerrit.plugins.multisite.forwarder.CacheEvictionForwarder> was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.registration.DynamicSet.setOf(DynamicSet.java:84) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.forwarder.ForwarderModule)

8) A binding to com.google.gerrit.extensions.registration.DynamicSet<com.googlesource.gerrit.plugins.multisite.forwarder.IndexEventForwarder> was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.registration.DynamicSet.setOf(DynamicSet.java:84) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.forwarder.ForwarderModule)

9) A binding to com.google.gerrit.extensions.registration.DynamicSet<com.googlesource.gerrit.plugins.multisite.forwarder.ProjectListUpdateForwarder> was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.registration.DynamicSet.setOf(DynamicSet.java:84) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.forwarder.ForwarderModule)

10) A binding to com.google.gerrit.extensions.registration.DynamicSet<com.googlesource.gerrit.plugins.multisite.forwarder.StreamEventForwarder> was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.registration.DynamicSet.setOf(DynamicSet.java:84) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.forwarder.ForwarderModule)

11) A binding to com.google.gerrit.extensions.registration.DynamicSet<com.googlesource.gerrit.plugins.multisite.kafka.consumer.AbstractKafkaSubcriber> was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.registration.DynamicSet.setOf(DynamicSet.java:84) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule)

12) A binding to com.google.gerrit.extensions.registration.DynamicSet<com.googlesource.gerrit.plugins.multisite.kafka.consumer.DroppedEventListener> was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.google.gerrit.extensions.registration.DynamicSet.setOf(DynamicSet.java:84) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule)

13) A binding to com.google.gson.Gson annotated with interface com.googlesource.gerrit.plugins.multisite.broker.BrokerGson was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.Module.configure(Module.java:114)

14) A binding to com.googlesource.gerrit.plugins.multisite.MessageLogger was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.Module.configure(Module.java:89)

15) A binding to java.util.UUID annotated with @com.googlesource.gerrit.plugins.multisite.InstanceId() was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.Module.getInstanceId(Module.java:124)

16) A binding to java.util.concurrent.Executor annotated with interface com.googlesource.gerrit.plugins.multisite.cache.CacheExecutor was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.cache.CacheModule.configure(CacheModule.java:28) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.cache.CacheModule)

17) A binding to java.util.concurrent.Executor annotated with interface com.googlesource.gerrit.plugins.multisite.event.EventExecutor was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.event.EventModule.configure(EventModule.java:26) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.event.EventModule)

18) A binding to com.googlesource.gerrit.plugins.multisite.broker.BrokerSession was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.forwarder.broker.BrokerForwarderModule.configure(BrokerForwarderModule.java:39) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.forwarder.broker.BrokerForwarderModule)

19) A binding to java.util.concurrent.Executor annotated with interface com.googlesource.gerrit.plugins.multisite.index.IndexExecutor was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.index.IndexModule.configure(IndexModule.java:31) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.index.IndexModule)

20) A binding to java.util.concurrent.ScheduledExecutorService annotated with interface com.googlesource.gerrit.plugins.multisite.index.ForwardedIndexExecutor was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.index.IndexModule.configure(IndexModule.java:32) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.index.IndexModule)

21) A binding to com.googlesource.gerrit.plugins.multisite.index.ProjectChecker was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.index.IndexModule.configure(IndexModule.java:41) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.index.IndexModule)

22) A binding to com.googlesource.gerrit.plugins.multisite.index.ChangeCheckerImpl$Factory was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.index.IndexModule.configure(IndexModule.java:43) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.index.IndexModule -> com.google.inject.assistedinject.FactoryModuleBuilder$1)

23) A binding to org.apache.kafka.common.serialization.Deserializer<byte[]> was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule.configure(KafkaConsumerModule.java:39) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule)

24) A binding to org.apache.kafka.common.serialization.Deserializer<com.googlesource.gerrit.plugins.multisite.kafka.consumer.SourceAwareEventWrapper> was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule.configure(KafkaConsumerModule.java:40) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule)

25) A binding to java.util.concurrent.Executor annotated with interface com.googlesource.gerrit.plugins.multisite.kafka.consumer.ConsumerExecutor was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule.configure(KafkaConsumerModule.java:43) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.consumer.KafkaConsumerModule)

26) A binding to com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedIndexEventRouter was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedEventRouterModule.configure(ForwardedEventRouterModule.java:22) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedEventRouterModule)

27) A binding to com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedCacheEvictionEventRouter was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedEventRouterModule.configure(ForwardedEventRouterModule.java:23) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedEventRouterModule)

28) A binding to com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedProjectListUpdateRouter was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedEventRouterModule.configure(ForwardedEventRouterModule.java:24) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedEventRouterModule)

29) A binding to com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedStreamEventRouter was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedEventRouterModule.configure(ForwardedEventRouterModule.java:25) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.kafka.router.ForwardedEventRouterModule)

30) A binding to com.google.gerrit.server.git.GitRepositoryManager was already configured at com.google.gerrit.server.plugins.CopyConfigModule.getGitRepositoryManager() (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.CopyConfigModule).
  at com.googlesource.gerrit.plugins.multisite.validation.ValidationModule.configure(ValidationModule.java:42) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.validation.ValidationModule)

31) A binding to com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.SharedRefEnforcement was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.validation.ValidationModule.configure(ValidationModule.java:45) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.validation.ValidationModule)

32) A binding to com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.SharedRefDatabase was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.zookeeper.ZkValidationModule.configure(ZkValidationModule.java:34) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.validation.ValidationModule -> com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.zookeeper.ZkValidationModule)

33) A binding to org.apache.curator.framework.CuratorFramework was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.zookeeper.ZkValidationModule.configure(ZkValidationModule.java:35) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.validation.ValidationModule -> com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.zookeeper.ZkValidationModule)

34) A binding to org.apache.curator.RetryPolicy annotated with @com.google.inject.name.Named(value=ZkLockRetryPolicy) was already configured at com.google.gerrit.server.plugins.PluginGuiceEnvironment$2.configure(PluginGuiceEnvironment.java:549) (via modules: com.google.gerrit.server.plugins.PluginGuiceEnvironment$1 -> com.google.gerrit.server.plugins.PluginGuiceEnvironment$2).
  at com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.zookeeper.ZkValidationModule.configure(ZkValidationModule.java:36) (via modules: com.googlesource.gerrit.plugins.multisite.Module -> com.googlesource.gerrit.plugins.multisite.validation.ValidationModule -> com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.zookeeper.ZkValidationModule)

34 errors
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:159)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:237)
at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:242)
at com.google.gerrit.server.plugins.ServerPlugin.startPlugin(ServerPlugin.java:205)
at com.google.gerrit.server.plugins.ServerPlugin.start(ServerPlugin.java:175)
at com.google.gerrit.server.plugins.PluginLoader.runPlugin(PluginLoader.java:495)
at com.google.gerrit.server.plugins.PluginLoader.rescan(PluginLoader.java:423)
at com.google.gerrit.server.plugins.PluginLoader.start(PluginLoader.java:328)
at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:95)
at com.google.gerrit.pgm.Daemon.start(Daemon.java:368)
at com.google.gerrit.pgm.Daemon.run(Daemon.java:272)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:226)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:122)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:66)
at Main.main(Main.java:28)
[2019-05-02 14:33:10,105] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin replication, version v2.16.7-7-ged5c9f2
[2019-05-02 14:33:10,154] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin reviewnotes, version v2.16.7
[2019-05-02 14:33:10,188] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin singleusergroup, version v2.16.7
[2019-05-02 14:33:10,235] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin websession-flatfile, version 44eaa18e7c
[2019-05-02 14:33:10,246] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "accountDeactivation".
[2019-05-02 14:33:10,247] [OnlineNoteDbMigrator] INFO  com.google.gerrit.server.notedb.rebuild.OnlineNoteDbMigrator : Starting online NoteDb migration
[2019-05-02 14:33:10,247] [OnlineNoteDbMigrator] INFO  com.google.gerrit.server.notedb.rebuild.OnlineNoteDbMigrator : Online index schema upgrades will be deferred until NoteDb migration is complete
[2019-05-02 14:33:10,248] [main] INFO  com.googlesource.gerrit.plugins.multisite.kafka.consumer.MultiSiteKafkaConsumerRunner : starting consumers
[2019-05-02 14:33:10,250] [main] INFO  com.googlesource.gerrit.plugins.multisite.broker.kafka.KafkaSession : Connect to localhost:9092...
[2019-05-02 14:33:10,250] [pool-1-thread-1] INFO  com.googlesource.gerrit.plugins.multisite.kafka.consumer.AbstractKafkaSubcriber : Kafka consumer subscribing to topic [gerrit_index] for event family [INDEX_EVENT]
[2019-05-02 14:33:10,250] [pool-1-thread-2] INFO  com.googlesource.gerrit.plugins.multisite.kafka.consumer.AbstractKafkaSubcriber : Kafka consumer subscribing to topic [gerrit_stream] for event family [STREAM_EVENT]
[2019-05-02 14:33:10,250] [pool-1-thread-3] INFO  com.googlesource.gerrit.plugins.multisite.kafka.consumer.AbstractKafkaSubcriber : Kafka consumer subscribing to topic [gerrit_cache_eviction] for event family [CACHE_EVENT]
[2019-05-02 14:33:10,250] [pool-1-thread-4] INFO  com.googlesource.gerrit.plugins.multisite.kafka.consumer.AbstractKafkaSubcriber : Kafka consumer subscribing to topic [gerrit_list_project] for event family [PROJECT_LIST_EVENT] 

Luca Milanesio

unread,
May 2, 2019, 8:55:28 AM5/2/19
to Matthias Sohn, Luca Milanesio, haridas...@gmail.com, Repo and Gerrit Discussion, Fabio Ponciroli

On 2 May 2019, at 13:49, Matthias Sohn <matthi...@gmail.com> wrote:

On Thu, May 2, 2019 at 8:24 AM Luca Milanesio <luca.mi...@gmail.com> wrote:


On 2 May 2019, at 06:04, haridas...@gmail.com wrote:

Hi Team,

I am trying to setup Gerrit Multisite Plugin. The plugin documentation says that currently it supports only one active master (RW site) accepting modifications on the Git repositories or the review data. 

Yes, however things are evolving very quickly on the plugins' development and very soon you'll be able to enable multiple writes on any site on any repos.
Tiago has a series of changes for making this possible without any risk of split-brain:

I believe, the health check plugin will be deciding the RW Site based on certain parameters. I assume, connections will be received by HAProxy which intern will validate with the "health check plugin" and redirect the connections to the active RW site.

Exactly, you can find a sample HAProxy configuration at:

I couldn't find any related configurations or documents. It would be of great help, if someone can point me to the right direction. 

Hi Dipu,
the plugin's configuration docs can be found at:

and for creating a test site, just use the setup.sh script developed by Tiago at:

I am trying to set this up based on current gerrit core and the multi-site plugin from stable-2.16 branch.

Zookeeper and Kafka are up and running, Gerrit instance 1 and 2 come up and I can logon as admin.

All the additional plugins load but the multi-site plugin fails to load with a Guice Exception,

Even if it is *technically built as a plugin*, it needs to installed as libModule, not as a plugin.

HTH

Luca.

David Ostrovsky

unread,
May 2, 2019, 9:09:03 AM5/2/19
to Repo and Gerrit Discussion

Am Donnerstag, 2. Mai 2019 14:55:28 UTC+2 schrieb lucamilanesio:
On 2 May 2019, at 13:49, Matthias Sohn <matthi...@gmail.com> wrote:

On Thu, May 2, 2019 at 8:24 AM Luca Milanesio <luca.m...@gmail.com> wrote:


On 2 May 2019, at 06:04, harida...@gmail.com wrote:

Hi Team,

I am trying to setup Gerrit Multisite Plugin. The plugin documentation says that currently it supports only one active master (RW site) accepting modifications on the Git repositories or the review data. 

Yes, however things are evolving very quickly on the plugins' development and very soon you'll be able to enable multiple writes on any site on any repos.
Tiago has a series of changes for making this possible without any risk of split-brain:

I believe, the health check plugin will be deciding the RW Site based on certain parameters. I assume, connections will be received by HAProxy which intern will validate with the "health check plugin" and redirect the connections to the active RW site.

Exactly, you can find a sample HAProxy configuration at:

I couldn't find any related configurations or documents. It would be of great help, if someone can point me to the right direction. 

Hi Dipu,
the plugin's configuration docs can be found at:

and for creating a test site, just use the setup.sh script developed by Tiago at:

I am trying to set this up based on current gerrit core and the multi-site plugin from stable-2.16 branch.

Zookeeper and Kafka are up and running, Gerrit instance 1 and 2 come up and I can logon as admin.

All the additional plugins load but the multi-site plugin fails to load with a Guice Exception,

Even if it is *technically built as a plugin*, it needs to installed as libModule, not as a plugin.


We could do better and mark the deployment kind of plugin as "libModule" to prevent
plugin loader facility in gerrit core to load it from `gerrit_site/plugins` directory. Moreover
plugin-manager could also read META-INF/MANIFEST.MF deploymentKind = libModule,
and deploy the plugin to the directory: `gerrit_site/lib`.

Matthias Sohn

unread,
May 2, 2019, 10:13:45 AM5/2/19
to Luca Milanesio, haridas...@gmail.com, Repo and Gerrit Discussion, Fabio Ponciroli
On Thu, May 2, 2019 at 2:55 PM Luca Milanesio <luca.mi...@gmail.com> wrote:


On 2 May 2019, at 13:49, Matthias Sohn <matthi...@gmail.com> wrote:

On Thu, May 2, 2019 at 8:24 AM Luca Milanesio <luca.mi...@gmail.com> wrote:


On 2 May 2019, at 06:04, haridas...@gmail.com wrote:

Hi Team,

I am trying to setup Gerrit Multisite Plugin. The plugin documentation says that currently it supports only one active master (RW site) accepting modifications on the Git repositories or the review data. 

Yes, however things are evolving very quickly on the plugins' development and very soon you'll be able to enable multiple writes on any site on any repos.
Tiago has a series of changes for making this possible without any risk of split-brain:

I believe, the health check plugin will be deciding the RW Site based on certain parameters. I assume, connections will be received by HAProxy which intern will validate with the "health check plugin" and redirect the connections to the active RW site.

Exactly, you can find a sample HAProxy configuration at:

I couldn't find any related configurations or documents. It would be of great help, if someone can point me to the right direction. 

Hi Dipu,
the plugin's configuration docs can be found at:

and for creating a test site, just use the setup.sh script developed by Tiago at:

I am trying to set this up based on current gerrit core and the multi-site plugin from stable-2.16 branch.

Zookeeper and Kafka are up and running, Gerrit instance 1 and 2 come up and I can logon as admin.

All the additional plugins load but the multi-site plugin fails to load with a Guice Exception,

Even if it is *technically built as a plugin*, it needs to installed as libModule, not as a plugin.

I ran the setup script without parameters which did all this automatically:
gerrit (stable-2.16 *)]$ ./plugins/multi-site/setup_local_env/setup.sh

Apparently the multi-site plugin was copied also to the plugins folder of each site. I deleted these copies and
restarted both Gerrit instances which fixed this error which was logged earlier.

-Matthias

Luca Milanesio

unread,
May 2, 2019, 10:27:39 AM5/2/19
to Matthias Sohn, Han-Wen Nienhuys, Luca Milanesio, haridas...@gmail.com, Repo and Gerrit Discussion, Fabio Ponciroli
Cool, we are planning to demo the multi-site plugin for Gerrit v3.0 in Munch :-)

@Han-Wen: can the demo be recorded at the Hackathon? It would be nice to share them to a wider audience afterward.

Luca.

Han-Wen Nienhuys

unread,
May 2, 2019, 10:31:59 AM5/2/19
to Luca Milanesio, Matthias Sohn, haridas...@gmail.com, Repo and Gerrit Discussion, Fabio Ponciroli
On Thu, May 2, 2019 at 4:27 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

Cool, we are planning to demo the multi-site plugin for Gerrit v3.0 in Munch :-)

@Han-Wen: can the demo be recorded at the Hackathon? It would be nice to share them to a wider audience afterward.


I think our room should have infrastructure to record things. We can record you, provided you sign a waiver.

Luca.


--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Luca Milanesio

unread,
May 2, 2019, 10:34:13 AM5/2/19
to Han-Wen Nienhuys, Luca Milanesio, Matthias Sohn, haridas...@gmail.com, Repo and Gerrit Discussion, Fabio Ponciroli

On 2 May 2019, at 15:31, Han-Wen Nienhuys <han...@google.com> wrote:



On Thu, May 2, 2019 at 4:27 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

Cool, we are planning to demo the multi-site plugin for Gerrit v3.0 in Munch :-)

@Han-Wen: can the demo be recorded at the Hackathon? It would be nice to share them to a wider audience afterward.


I think our room should have infrastructure to record things. We can record you, provided you sign a waiver.

That makes sense, of course.
Thanks for clarifying.

Luca.

haridas...@gmail.com

unread,
May 2, 2019, 7:12:57 PM5/2/19
to Repo and Gerrit Discussion
Thanks a lot for the update. I could setup the haproxy to direct connections based on healthcheck plugin status.

However, I face the below error when a upload a change.

[2019-05-02 15:45:35,045] [ReceiveCommits-2] ERROR com.google.gerrit.server.git.receive.ReceiveCommits : Error collecting groups for changes [CONTEXT RECEIVE_ID="dipuh_test-1556837135032-4c67b9de" ]
com.google.gwtorm.server.OrmException: java.util.concurrent.ExecutionException: java.io.IOException: Unable to update ref 'refs/sequences/changes', cannot open the local ref on the local DB

Caused by: java.io.IOException: Unable to update ref 'refs/sequences/changes', the local objectId 'refs/sequences/changes' is not equal to the one in the shared ref datasuper
        at com.googlesource.gerrit.plugins.multisite.validation.MultiSiteRefUpdate.checkSharedDBForRefUpdate(MultiSiteRefUpdate.java:77)


I was using psql as backend on Gerrit 2.14. Now migrated from 2.14 to 2.16 and moved to notes db. 

The direct pushes are happening fine and replicates across sites. Tried with a new repo created as well, but uploads still failed to proceed.

git push origin HEAD:refs/for/master
Counting objects: 3, done.
Writing objects: 100% (3/3), 280 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Processing changes: refs: 1, done
remote: error: internal error while processing changes
To ssh://****/dipuh_test
 ! [remote rejected] HEAD -> refs/for/master (internal server error)
error: failed to push some refs to 'ssh://*****/dipuh_test'

Am I missing something? Please advice.

Thanks,
Dipu H


On Wednesday, May 1, 2019 at 11:24:18 PM UTC-7, lucamilanesio wrote:


On 2 May 2019, at 06:04, harida...@gmail.com wrote:

Hi Team,

I am trying to setup Gerrit Multisite Plugin. The plugin documentation says that currently it supports only one active master (RW site) accepting modifications on the Git repositories or the review data.

Yes, however things are evolving very quickly on the plugins' development and very soon you'll be able to enable multiple writes on any site on any repos.
Tiago has a series of changes for making this possible without any risk of split-brain:

I believe, the health check plugin will be deciding the RW Site based on certain parameters. I assume, connections will be received by HAProxy which intern will validate with the "health check plugin" and redirect the connections to the active RW site.

Exactly, you can find a sample HAProxy configuration at:

I couldn't find any related configurations or documents. It would be of great help, if someone can point me to the right direction. 

Hi Dipu,
the plugin's configuration docs can be found at:

and for creating a test site, just use the setup.sh script developed by Tiago at:

Ponch is working on creating a Dockerized test environment also with the following series of changes:

HTH

Luca.


Thanks,
Dipu H

--
--

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

Luca Milanesio

unread,
May 2, 2019, 7:22:40 PM5/2/19
to haridas...@gmail.com, Luca Milanesio, Repo and Gerrit Discussion

On 3 May 2019, at 00:12, haridas...@gmail.com wrote:

Thanks a lot for the update. I could setup the haproxy to direct connections based on healthcheck plugin status.

However, I face the below error when a upload a change.

[2019-05-02 15:45:35,045] [ReceiveCommits-2] ERROR com.google.gerrit.server.git.receive.ReceiveCommits : Error collecting groups for changes [CONTEXT RECEIVE_ID="dipuh_test-1556837135032-4c67b9de" ]
com.google.gwtorm.server.OrmException: java.util.concurrent.ExecutionException: java.io.IOException: Unable to update ref 'refs/sequences/changes', cannot open the local ref on the local DB

Caused by: java.io.IOException: Unable to update ref 'refs/sequences/changes', the local objectId 'refs/sequences/changes' is not equal to the one in the shared ref datasuper
        at com.googlesource.gerrit.plugins.multisite.validation.MultiSiteRefUpdate.checkSharedDBForRefUpdate(MultiSiteRefUpdate.java:77)

You ended up into a split-brain situation: I am currently reviewing a fix for this.

Bear with me, should be merged by today or max tomorrow.

Luca.


More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

haridas...@gmail.com

unread,
May 2, 2019, 7:29:14 PM5/2/19
to Repo and Gerrit Discussion
Thanks Luca. Appreciate the quick help.

Dipu H 

haridas...@gmail.com

unread,
May 7, 2019, 12:28:34 PM5/7/19
to Repo and Gerrit Discussion
Hi Luca,

I tried with the latest plugin from plugin-multi-site-bazel-stable-2.16 Jenkins job, build 44. I get the below error now:

[2019-05-07 09:11:51,327] [ReceiveCommits-1] WARN  org.apache.curator.utils.ZKPaths : The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.
[2019-05-07 09:11:51,362] [ReceiveCommits-1] WARN  com.googlesource.gerrit.plugins.multisite.validation.RefUpdateValidator : Failure while running with policy enforcement REQUIRED. Error message: Local ref refs/sequences/changes (ObjectId=3b2deb7a1496e5b59a535a39c213bc0cd8574e7e) on project All-Projects is out of sync with the shared ref-db [CONTEXT RECEIVE_ID="dipuh_test-1557245511245-670b4b9b" ]
com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.OutOfSyncException: Local ref refs/sequences/changes (ObjectId=3b2deb7a1496e5b59a535a39c213bc0cd8574e7e) on project All-Projects is out of sync with the shared ref-db
.
.
.
[2019-05-07 09:11:51,380] [ReceiveCommits-1] ERROR com.google.gerrit.server.git.receive.ReceiveCommits : Error collecting groups for changes [CONTEXT RECEIVE_ID="dipuh_test-1557245511245-670b4b9b" ]
com.google.gwtorm.server.OrmException: java.util.concurrent.ExecutionException: com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.OutOfSyncException: Local ref refs/sequences/changes (ObjectId=3b2deb7a1496e5b59a535a39c213bc0cd8574e7e) on project All-Projects is out of sync with the shared ref-db

Did I miss something ? :)

Thanks,
Dipu H

Luca Milanesio

unread,
May 7, 2019, 12:47:47 PM5/7/19
to haridas...@gmail.com, Luca Milanesio, Repo and Gerrit Discussion

On 7 May 2019, at 09:28, haridas...@gmail.com wrote:

Hi Luca,

I tried with the latest plugin from plugin-multi-site-bazel-stable-2.16 Jenkins job, build 44. I get the below error now:

[2019-05-07 09:11:51,327] [ReceiveCommits-1] WARN  org.apache.curator.utils.ZKPaths : The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.
[2019-05-07 09:11:51,362] [ReceiveCommits-1] WARN  com.googlesource.gerrit.plugins.multisite.validation.RefUpdateValidator : Failure while running with policy enforcement REQUIRED. Error message: Local ref refs/sequences/changes (ObjectId=3b2deb7a1496e5b59a535a39c213bc0cd8574e7e) on project All-Projects is out of sync with the shared ref-db [CONTEXT RECEIVE_ID="dipuh_test-1557245511245-670b4b9b" ]
com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.OutOfSyncException: Local ref refs/sequences/changes (ObjectId=3b2deb7a1496e5b59a535a39c213bc0cd8574e7e) on project All-Projects is out of sync with the shared ref-db

Are you pushing to the two sites behind a reverse proxy in round-robin?
Is the problem persistent or transient?

Luca.

.
.
.
[2019-05-07 09:11:51,380] [ReceiveCommits-1] ERROR com.google.gerrit.server.git.receive.ReceiveCommits : Error collecting groups for changes [CONTEXT RECEIVE_ID="dipuh_test-1557245511245-670b4b9b" ]
com.google.gwtorm.server.OrmException: java.util.concurrent.ExecutionException: com.googlesource.gerrit.plugins.multisite.validation.dfsrefdb.OutOfSyncException: Local ref refs/sequences/changes (ObjectId=3b2deb7a1496e5b59a535a39c213bc0cd8574e7e) on project All-Projects is out of sync with the shared ref-db

Did I miss something ? :)

Thanks,
Dipu H

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

haridas...@gmail.com

unread,
May 7, 2019, 1:53:14 PM5/7/19
to Repo and Gerrit Discussion
Hi Luca,

Thanks for the quick reply. 

Yes, I have an haproxy in front, with the below configuration:

backend ssh
    mode tcp
    option redispatch
    option httpchk GET /config/server/healthcheck~status HTTP/1.0
    balance roundrobin
    timeout connect 1m
    timeout server 5m
    server ssh_node1 testgit-01:29418 check inter 20s check port 8080 inter 20s
    server ssh_node2 testgit-02:29418 check inter 20s check port 8080 inter 20s backup

The error is persistent. Please find the log attached.

Thanks,
Dipu H
error.log

Tiago Palma

unread,
May 8, 2019, 8:49:27 PM5/8/19
to Matthias Sohn, Luca Milanesio, haridas...@gmail.com, Repo and Gerrit Discussion, Fabio Ponciroli
Hi Matthias,

I suspect the reason you're seeing the multi-site.jar under the plugins directory is that your release.war file contains the multi-site plugin and because the shell script installs gerrit with --install-all-plugins option, it automatically deploys it under that directory.

Hope this helps.
Tiago


--

Tiago Palma

unread,
May 8, 2019, 8:49:27 PM5/8/19
to haridas...@gmail.com, Repo and Gerrit Discussion
Hi Haridasan,

Looking at the logs and at your configuration, I suspect that you're pushing in round-robin through ssh_server1 and ssh_server2 and the replication does not have enough time to copy the git data across to the second server, hence your out of sync exceptions. 
Can you please check if your pushes are going through the same node?
Can you also let me know if you've changed notedb.accounts.sequenceBatchSize?

Tiago

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/15e83b61-7d2d-4f5b-bc53-af66bb916469%40googlegroups.com.

David Pursehouse

unread,
May 8, 2019, 8:58:49 PM5/8/19
to Tiago Palma, Matthias Sohn, Luca Milanesio, haridas...@gmail.com, Repo and Gerrit Discussion, Fabio Ponciroli
On Thu, May 9, 2019 at 9:49 AM Tiago Palma <tiago....@gmail.com> wrote:
Hi Matthias,

I suspect the reason you're seeing the multi-site.jar under the plugins directory is that your release.war file contains the multi-site plugin and because the shell script installs gerrit with --install-all-plugins option, it automatically deploys it under that directory.

This seems like something that should be improved.  I.e. for bundled plugins that are not actually plugins but library modules, the initialization program should not install them into the plugins folder.
 

David Ostrovsky

unread,
May 9, 2019, 1:10:18 AM5/9/19
to Repo and Gerrit Discussion

On Thursday, May 9, 2019 at 2:58:49 AM UTC+2, David Pursehouse wrote:
On Thu, May 9, 2019 at 9:49 AM Tiago Palma <tiago...@gmail.com> wrote:
Hi Matthias,

I suspect the reason you're seeing the multi-site.jar under the plugins directory is that your release.war file contains the multi-site plugin and because the shell script installs gerrit with --install-all-plugins option, it automatically deploys it under that directory.

This seems like something that should be improved.  I.e. for bundled plugins that are not actually plugins but library modules, the initialization program should not install them into the plugins folder.

Right. Have you noticed my previous comment? I think right now the
plugin loader doesn't know whether it is a regular plugin or a lib module.


Dipu Haridas

unread,
May 9, 2019, 1:42:37 AM5/9/19
to Repo and Gerrit Discussion
Hi Tiago,

My haproxy configuration is the below:

backend ssh
    mode tcp
    option redispatch
    option httpchk GET /config/server/healthcheck~status HTTP/1.0
    balance roundrobin
    timeout connect 5m
    timeout server 5m
    server ssh_node1 testgit-01:29418 check inter 20s check port 8080 inter 20s
    server ssh_node2 testgit-02:29418 check inter 20s check port 8080 inter 20s backup 

I believe, since the second server (ssh_node2) is mentioned as "backup", connections should not go directly to this server, unless the health check on ssh_node1 fails. I have verified that connections from the proxy are directed to ssh_node1.

I am trying only one upload at a time and I am the only one user uploading. Pushes to the repository is working fine and syncing to the second site without any issue. Only the uploads fail.I mean "git push" works, but "git push origin HEAD:refs/for/master" fails. Is there any change in references from Gerrit 2.14 to 2.16?

>> Can you also let me know if you've changed notedb.accounts.sequenceBatchSize?

I haven't set any custom settings for nodedb, rather than converting the psql db to note. Could you please guide me to the correct setting for this?

Thanks,
Dipu H
 

Luca Milanesio

unread,
May 9, 2019, 2:43:59 AM5/9/19
to haridas...@gmail.com, Luca Milanesio, Repo and Gerrit Discussion

On 7 May 2019, at 18:53, haridas...@gmail.com wrote:

Hi Luca,

Thanks for the quick reply. 

Yes, I have an haproxy in front, with the below configuration:

backend ssh
    mode tcp
    option redispatch
    option httpchk GET /config/server/healthcheck~status HTTP/1.0
    balance roundrobin
    timeout connect 1m
    timeout server 5m
    server ssh_node1 testgit-01:29418 check inter 20s check port 8080 inter 20s
    server ssh_node2 testgit-02:29418 check inter 20s check port 8080 inter 20s backup

What about the HTTP configuration?

The error is persistent. Please find the log attached.

That means that the shared-refdb on Zookeeper is consistently misaligned with the local-refdb on the filesystem. That means that the ref was updated by another master and the replication plugin isn't working for some reasons.
Can you share your replication.config?

Luca.


Thanks,
Dipu H

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
<error.log>

Luca Milanesio

unread,
May 9, 2019, 2:46:20 AM5/9/19
to David Ostrovsky, Luca Milanesio, Repo and Gerrit Discussion
Right, two things then:
1) We should have a specific annotation for libModules in MANIFEST.MF
2) The '--install-all-plugins' should only install the jars with the plugin annotation

WDYT?

Luca.




--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Luca Milanesio

unread,
May 9, 2019, 2:58:28 AM5/9/19
to Dipu Haridas, Luca Milanesio, Repo and Gerrit Discussion

On 9 May 2019, at 06:42, Dipu Haridas <haridas...@gmail.com> wrote:

Hi Tiago,

My haproxy configuration is the below:

backend ssh
    mode tcp
    option redispatch
    option httpchk GET /config/server/healthcheck~status HTTP/1.0
    balance roundrobin
    timeout connect 5m
    timeout server 5m
    server ssh_node1 testgit-01:29418 check inter 20s check port 8080 inter 20s
    server ssh_node2 testgit-02:29418 check inter 20s check port 8080 inter 20s backup 

I believe, since the second server (ssh_node2) is mentioned as "backup", connections should not go directly to this server, unless the health check on ssh_node1 fails. I have verified that connections from the proxy are directed to ssh_node1.

OK, what happened then is that your existing Zookeeper instance has old outdated values, possibly coming from the 
Just reset Zookeeper and it should work again.



I am trying only one upload at a time and I am the only one user uploading. Pushes to the repository is working fine and syncing to the second site without any issue. Only the uploads fail.I mean "git push" works, but "git push origin HEAD:refs/for/master" fails. Is there any change in references from Gerrit 2.14 to 2.16?

2.14? Are you trying to use the multi-site with Gerrit v2.14?


>> Can you also let me know if you've changed notedb.accounts.sequenceBatchSize?

I haven't set any custom settings for nodedb, rather than converting the psql db to note. Could you please guide me to the correct setting for this?

No specific setting is needed for that.

Luca.


Thanks,
Dipu H
 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Dipu Haridas

unread,
May 9, 2019, 3:02:19 AM5/9/19
to Repo and Gerrit Discussion
Hi Luca,

 >> 2.14? Are you trying to use the multi-site with Gerrit v2.14?

I am trying multisite on Gerrit 2.16. Was wondering if there is any reference changes from 2.14 to 2.16. 

Let me restart the zookeeper instances and try.

Thanks,
Dipu H

Tiago Palma

unread,
May 9, 2019, 3:38:38 AM5/9/19
to Dipu Haridas, Repo and Gerrit Discussion
Hi Dipu,

To reset Zookeeper actually means removing the znode that is outdated from zookeeper. You can do this via the zookeeper cli.
I'm not aware of reference changes from 2.14 to 2.16.

Hope this helps.
Tiago

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2722500d-9d76-4d2b-91eb-dbec03eac883%40googlegroups.com.

Dipu Haridas

unread,
May 9, 2019, 3:47:00 AM5/9/19
to Repo and Gerrit Discussion
I could upload the reviews, after resetting  zookeeper :)

Thanks,
Dipu H


On Thursday, May 9, 2019 at 12:38:38 AM UTC-7, Tiago Palma wrote:
Hi Dipu,

To reset Zookeeper actually means removing the znode that is outdated from zookeeper. You can do this via the zookeeper cli.
I'm not aware of reference changes from 2.14 to 2.16.

Hope this helps.
Tiago

On Thu, 9 May 2019 at 08:02, Dipu Haridas <harida...@gmail.com> wrote:
Hi Luca,

 >> 2.14? Are you trying to use the multi-site with Gerrit v2.14?

I am trying multisite on Gerrit 2.16. Was wondering if there is any reference changes from 2.14 to 2.16. 

Let me restart the zookeeper instances and try.

Thanks,
Dipu H

--
--
To unsubscribe, email repo-d...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

Luca Milanesio

unread,
May 9, 2019, 4:46:01 AM5/9/19
to Dipu Haridas, Luca Milanesio, Repo and Gerrit Discussion

On 9 May 2019, at 08:47, Dipu Haridas <haridas...@gmail.com> wrote:

I could upload the reviews, after resetting  zookeeper :)

Cool ! From now on, that shouldn't happen anymore. If it does, feel free to come back and open an issue, as it could be a new bug.

Luca.


More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/7110508b-892b-4109-a532-8de22ae623e7%40googlegroups.com.

Dipu Haridas

unread,
May 9, 2019, 3:49:50 PM5/9/19
to Repo and Gerrit Discussion
Hi,

My current haproxy configuration is as below:

backend read-backendnodes
    mode http
    balance source
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    default-server inter 10s fall 3 rise 2
    option httpchk GET /config/server/healthcheck~status HTTP/1.0
    http-check expect status 200
    server node1 testgit-01:8080 check inter 10s
    server node2 testgit-02:8080 check inter 10s backup

backend ssh
    mode tcp
    option redispatch
    option httpchk GET /config/server/healthcheck~status HTTP/1.0
    balance roundrobin
    timeout connect 5m
    timeout server 5m
    server ssh_node1 testgit-01:29418 check inter 20s check port 8080 inter 20s
    server ssh_node2 testgit-02:29418 check inter 20s check port 8080 inter 20s backup

If I stop gerrit on testgit-01, connections are not redirecting to testgit-02. I manually checked and the health check status on  testgit-02 is fine. If I remove "backup" option and make both of them active in haproxy, connections are directed properly to the available node. Is this the suggested configuration, as I read that only one master it supports currently. Please advise.

Thanks,
Dipu H

Luca Milanesio

unread,
May 9, 2019, 3:54:19 PM5/9/19
to Dipu Haridas, Luca Milanesio, Repo and Gerrit Discussion
You should check the HAProxy log, it will tell you what's wrong and have specific messages for the healthcheck.

Luca.


Thanks,
Dipu H

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Dipu Haridas

unread,
May 9, 2019, 6:00:29 PM5/9/19
to Repo and Gerrit Discussion
Oops. That was the basic thing, I could have checked. Thanks for the pointer. It helped.

Thanks,
Dipu H

On Thursday, May 9, 2019 at 12:54:19 PM UTC-7, lucamilanesio wrote:

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages