event-kafka plugin - message

218 views
Skip to first unread message

malvika

unread,
Aug 4, 2023, 2:40:10 PM8/4/23
to Repo and Gerrit Discussion
Hi Community,

I am seeing messages with multi-site setup ( Two leaders (RW) and haproxy, kafka/zookeeper) 

pull replication is not working and I am seeing "Not completed task"

A snippet of the error log - 

[2023-08-04T16:55:26.151Z] [kafka-subscriber-3[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@15323855[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@1fca5325[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@110dbeac]]]] INFO  org.apache.kafka.clients.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-3, groupId=instance-1] Successfully joined group with generation 2

[2023-08-04T16:55:26.154Z] [kafka-subscriber-3[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@15323855[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@1fca5325[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@110dbeac]]]] INFO  org.apache.kafka.clients.consumer.internals.ConsumerCoordinator : [Consumer clientId=consumer-3, groupId=instance-1] Setting newly assigned partitions [gerrit_stream-0]


Thanks,

Malvika

malvika

unread,
Aug 4, 2023, 5:41:11 PM8/4/23
to Repo and Gerrit Discussion
I want more details about the multi-site problem  I am seeing a problem with replication. Is the plugin working with 3.7.4?

Gerrit 
===== 
3.7.4

a snippet of gerrit config  (both are same on instance -1 and instance 2) 

[gerrit]

      installModule = com.gerritforge.gerrit.eventbroker.BrokerApiModule

      installModule = com.googlesource.gerrit.modules.gitrefsfilter.RefsFilterModule

      installModule = com.googlesource.gerrit.plugins.replication.ReplicationExtensionPointModule

      installModule = com.googlesource.gerrit.plugins.replication.pull.ReplicationExtensionPointModule

      installDbModule = com.googlesource.gerrit.plugins.multisite.GitModule

      installModule = com.googlesource.gerrit.plugins.multisite.Module

      serverId = <...>

      instanceId = instance-2

[plugin "events-kafka"]

      bootstrapServers = <ipadd>:9092

      groupId = instance-2

      sendAsync = true

      numberOfSubscribers = 6

      securityProtocol = PLAINTEXT

      pollingIntervalMs = 1000

      enableAutoCommit = true

      autoCommitIntervalMs = 1000

      autoOffsetReset = latest


pull replication cfg on instance 1

[remote]

  url = ssh://gerrit@<instance  2address>:<path to git>/git/${name}.git

  apiUrl = http://<:instance 2  address:8080>

  fetch = +refs/*:refs/*

  mirror = true

  timeout = 600

  rescheduleDelay = 15

  replicationDelay = 15

  createMissingRepositories = true

  replicateProjectDeletions = true

  replicateHiddenProjects = true

log from pull replication on instance 1

2023-08-04 21:05:41,184] Ref event received: CREATE on project test2:refs/meta/config - 0000000000000000000000000000000000000000 => 7c0284bd21e7605d60c02c1c9c3e2cd905f3fd38

[2023-08-04 21:05:41,285] Ref event received: UPDATE on project test2:refs/multi-site/version - fee7e4a6d558733c25e9d43928abac8f4b116ace => 2f0138e0a7eedc3ab22b1c3cbb7c42d63aeee8fb

[2023-08-04 21:05:41,292] Ref event received: UPDATE on project test2:refs/multi-site/version - 2f0138e0a7eedc3ab22b1c3cbb7c42d63aeee8fb => 93f147cb886f06cbb32eedc64f54db570aeb40b1

[2023-08-04 21:05:41,293] Ref event received: CREATE on project test2:refs/heads/master - 0000000000000000000000000000000000000000 => 92404394e0ddce148daf3fb3cab91d718f8a68e4

error log

============

023-08-04T19:23:25.214Z] [kafka-subscriber-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5350b333[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@59a8d843[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@576f2d06]]]] WARN  com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedIndexProjectHandler : project test1 is not up-to-date. Rescheduling

[2023-08-04T19:23:25.214Z] [kafka-subscriber-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5350b333[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@59a8d843[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@576f2d06]]]] WARN  com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedIndexProjectHandler : Retrying for the #1 time to index project test1 after 30000 msecs

[2023-08-04T19:23:44.192Z] [Forwarded-Index-Event-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@39ed4066[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@33152b16[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3dd2c304]]]] WARN  com.googlesource.gerrit.plugins.multisite.index.GroupCheckerImpl : Commit AnyObjectId[ec1f644bff716ab1b6091c0a7093389c78529a38] does not exist in All-Users

[2023-08-04T19:23:44.193Z] [Forwarded-Index-Event-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@39ed4066[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@33152b16[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3dd2c304]]]] WARN  com.googlesource.gerrit.plugins.multisite.index.GroupCheckerImpl : Group '4b57377b29b2449a531980ed81454563665f0313' NOT up-to-date: sha1 'AnyObjectId[ec1f644bff716ab1b6091c0a7093389c78529a38]' still missing in All-Users

[2023-08-04T19:23:44.193Z] [Forwarded-Index-Event-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@39ed4066[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@33152b16[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3dd2c304]]]] WARN  com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedIndexGroupHandler : group 4b57377b29b2449a531980ed81454563665f0313 is not up-to-date. Rescheduling

[2023-08-04T19:23:44.193Z] [Forwarded-Index-Event-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@39ed4066[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@33152b16[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3dd2c304]]]] WARN  com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedIndexGroupHandler : Retrying for the #2 time to index group 4b57377b29b2449a531980ed81454563665f0313 after 30000 msecs

[2023-08-04T19:23:44.196Z] [kafka-producer-network-thread | producer-1] INFO  org.apache.kafka.clients.Metadata : Cluster ID: R7ZfD93zRci1QC9-R0tvrw

[2023-08-04T19:23:55.216Z] [Forwarded-Index-Event-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@485a3102[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@42a25a72[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@4c65ac56]]]] WARN  com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedIndexProjectHandler : project test1 is not up-to-date. Rescheduling

[2023-08-04T19:23:55.216Z] [Forwarded-Index-Event-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@485a3102[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@42a25a72[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@4c65ac56]]]] WARN  com.googlesource.gerrit.plugins.multisite.forwarder.ForwardedIndexProjectHandler : Retrying for the #2 time to index project test1 after 30000 msecs

[2023-08-04T19:24:14.195Z] [Forwarded-Index-Event-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5601b63[N


Thanks in advance 


Luca Milanesio

unread,
Aug 5, 2023, 1:52:06 AM8/5/23
to Repo and Gerrit Discussion, Luca Milanesio, malvika

On 5 Aug 2023, at 00:41, malvika <malvik...@gmail.com> wrote:

I want more details about the multi-site problem  I am seeing a problem with replication. Is the plugin working with 3.7.4?

Which replication plugin do you use? Which replication problem you see?

Luca.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/07cce98d-e646-4969-811f-5a6fb9fc648en%40googlegroups.com.

malvika

unread,
Aug 17, 2023, 5:56:23 PM8/17/23
to Repo and Gerrit Discussion

Hi Luca,

It was my config. Both plugins: replication & pull-replication are working. However, I see both plugins are doing replication as opposite to what I was expecting that nothing will happen via the replication plugin and will have a message " 
"Nothing to replicate or something..."

replication config on both Gerrit (R/W) nodes 

[gerrit]

      autoReload = true

      replicateOnStartup = false

      instanceId = phx-dpro-leader-01-dev-gerrit


[remote <instanceId-2>]

      apiUrl = http://instanceId-2:8080

      fetch = +refs/*:refs/*

      .....

.     <more config>


[remote "<instanceId-2>-push"]

      ......

     .......

      push = +refs/*:refs/

      replicationDelay = 50

      url = gerrit@<instanceId-2>:/home/gerrit/gerrit-repos/git/${name}.git

[replication]

      excludeRefs = refs/multi-site/version

      lockErrorMaxRetries = 5

      consumeStreamEvents = true

      maxRetries = 5

~                                        

a snippet of my logs 

Pull-replication log 
==============

[2023-08-17 21:31:05,233] Pull replication REST API apply object to http://<instanceId - 2>:8080 for test-demo1:refs/meta/config - [{commitObject=081543412968f5dfdcaa1ba343526d7f1ca9ef96 (COMMIT) treeObject=537ab9e414cd82d953805d589180813f33c693b2 (TREE) blobs=[a782fcaf8f8f413d313e58d4f0238e214800328c (BLOB)]}]

[2023-08-17 21:31:05,516] Pull replication REST API apply object to http://<instanceId - 2>:8080 COMPLETED for test-demo1:refs/meta/config - [{commitObject=081543412968f5dfdcaa1ba343526d7f1ca9ef96 (COMMIT) treeObject=537ab9e414cd82d953805d589180813f33c693b2 (TREE) blobs=[a782fcaf8f8f413d313e58d4f0238e214800328c (BLOB)]}], HTTP Result: FAILED, status=500 'Not found: test-demo1

' - time:280.110686 ms

[2023-08-17 21:31:06,617] Missing project test-demo1 created, HTTP Result:OK

[2023-08-17 21:31:06,618] Pull replication REST API fetch to http://<<instanceId - 2>>:8080 for test-demo1:refs/meta/config

[2023-08-17 21:31:06,644] Pull replication REST API fetch to http://<<instanceId - 2>:8080 COMPLETED for test-demo1:refs/meta/config, HTTP Result: OK - time:26 ms

[2023-08-17 21:31:06,694] Ref event received: CREATE on project test-demo1:refs/heads/master - 0000000000000000000000000000000000000000 => 58416a1988940c04cdcf7ec208076fbae0047d21

[2023-08-17 21:31:06,695] RevisionData is {commitObject=58416a1988940c04cdcf7ec208076fbae0047d21 (COMMIT) treeObject=4b825dc642cb6eb9a060e54bf8d69288fbee4904 (TREE) } for test-demo1:refs/heads/master

[2023-08-17 21:31:06,696] Pull replication REST API apply object to http://<instanceId - 2>:8080 for test-demo1:refs/heads/master - [{commitObject=58416a1988940c04cdcf7ec208076fbae0047d21 (COMMIT) treeObject=4b825dc642cb6eb9a060e54bf8d69288fbee4904 (TREE) }]

[2023-08-17 21:31:06,709] Pull replication REST API apply object to http://<instanceId - 2>:8080 COMPLETED for test-demo1:refs/heads/master - [{commitObject=58416a1988940c04cdcf7ec208076fbae0047d21 (COMMIT) treeObject=4b825dc642cb6eb9a060e54bf8d69288fbee4904 (TREE) }], HTTP Result: OK - time:12.797041 ms

Replication log 
==============

log from instance 1 

[2023-08-17 21:31:06,684] scheduling replication test-demo1:[refs/multi-site/version] => gerrit@<instanceId - 2>/home/gerrit/gerrit-repos/git/test-demo1.git [CONTEXT PLUGIN="gerrit" PLUGIN="multi-site" PLUGIN="replication" project="test-demo1" request="REST /projects/*" ]

[2023-08-17 21:31:06,685] consolidated test-demo1:[refs/multi-site/version] => [5af026c7] push gerrit@<instanceId - 2>:/home/gerrit/gerrit-repos/git/test-demo1.git [refs/meta/config refs/multi-site/version (+1)] with an existing pending push [CONTEXT PLUGIN="gerrit" PLUGIN="multi-site" PLUGIN="replication" project="test-demo1" request="REST /projects/*" ]

[2023-08-17 21:31:06,692] scheduling replication test-demo1:[refs/multi-site/version] => gerrit@<instanceId - 2>:/home/gerrit/gerrit-repos/git/test-demo1.git [CONTEXT PLUGIN="gerrit" PLUGIN="multi-site" PLUGIN="replication" project="test-demo1" request="REST /projects/*" ]

[2023-08-17 21:31:06,693] consolidated test-demo1:[refs/multi-site/version] => [5af026c7] push gerrit@<instanceId - 2>:/home/gerrit/gerrit-repos/git/test-demo1.git [refs/meta/config refs/multi-site/version (+1)] with an existing pending push [CONTEXT PLUGIN="gerrit" PLUGIN="multi-site" PLUGIN="replication" project="test-demo1" request="REST /projects/*" ]

[2023-08-17 21:31:55,182] Replication to gerrit@<instanceId - 2>:/home/gerrit/gerrit-repos/git/test-demo1.git started... [CONTEXT PLUGIN="replication" project="test-demo1" pushOneId="5af026c7" request="REST /projects/*" ]

[2023-08-17 21:31:55,204] Push to gerrit@<instanceId - 2>:/home/gerrit/gerrit-repos/git/test-demo1.git references: RemoteRefUpdate{refSpec=refs/meta/config:refs/meta/config, status=NOT_ATTEMPTED, id=(null)..AnyObjectId[081543412968f5dfdcaa1ba343526d7f1ca9ef96], force=yes, delete=no, ffwd=no}, RemoteRefUpdate{refSpec=refs/heads/master:refs/heads/master, status=NOT_ATTEMPTED, id=(null)..AnyObjectId[58416a1988940c04cdcf7ec208076fbae0047d21], force=yes, delete=no, ffwd=no}, RemoteRefUpdate{refSpec=refs/multi-site/version:refs/multi-site/version, status=NOT_ATTEMPTED, id=(null)..AnyObjectId[939f8a006cc7233768990772caece572f0c44878], force=yes, delete=no, ffwd=no} [CONTEXT PLUGIN="replication" project="test-demo1" pushOneId="5af026c7" request="REST /projects/*" ]

[2023-08-17 21:31:56,073] Replication to gerrit@<instanceId - 2>:/home/gerrit/gerrit-repos/git/test-demo1.git completed in 890ms, 50008ms delay, 0 retries [CONTEXT PLUGIN="replication" project="test-demo1" pushOneId="5af026c7" request="REST /projects/*" ]


Thanks,
Malvika



Reply all
Reply to author
Forward
0 new messages