Replication between gerrit issue

759 views
Skip to first unread message

Mk

unread,
Jul 18, 2022, 2:21:14 AM7/18/22
to Repo and Gerrit Discussion

I’m trying to enable replication from our existing gerrit to newer gerrit instance.

Source (Master) gerrit environment details.

Gerrit version – 3.2.3

Total repos – 600+

replication.config has below information.

 [remote "replica-1"]

  url = ssh://ad...@gerrit-review.com:29418/${name}.git

  push = refs/*:refs/*

  timtout = 30

  threads = 3

  replicatePermissions = true

  replicateHiddenProjects = true

  mirror = false

  replicationDelay = 0

  createMissingRepositories = true

  lockErrorMaxRetries = 5

 Target (Replica) gerrit environment details.

 Gerrit version – 3.6.1

Total repos – 2 (All-Projects.git  All-Users.git)

 replication_log from source server as follows.

 [2022-07-18 06:10:02,880] Created remote repository: ssh://ad...@gerrit-review.com:29418/member/test.git [CONTEXT pushOneId="9cce7596" ]

[2022-07-18 06:10:02,881] Missing repository created; retry replication to ssh://ad...@gerrit-review.com:29418/ member/test.git [CONTEXT pushOneId="9cce7596" ]

[2022-07-18 06:10:02,927] Created remote repository: ssh://ad...@gerrit-review.com:29418/lab/usa/project.git [CONTEXT pushOneId="dcd86d61" ]

[2022-07-18 06:10:02,927] Missing repository created; retry replication to ssh://ad...@gerrit-review.com:29418/lab/usa/project.git [CONTEXT pushOneId="dcd86d61" ]

 Please give me an advise to resolve this issue.

Matthias Sohn

unread,
Jul 18, 2022, 2:35:33 AM7/18/22
to Mk, Repo and Gerrit Discussion
Which issue do you have ?

-Matthias

Mk

unread,
Jul 18, 2022, 2:44:11 AM7/18/22
to Repo and Gerrit Discussion
Hello Matthias,

Except (All-Projects.git  & All-Users.git) i can't see other repos in my target server and in the source replication log constitutionally showing above provided message.

Still i'm i missing any configurations to replicate my source server gerrit data?

Matthias Sohn

unread,
Jul 18, 2022, 3:41:29 AM7/18/22
to Mk, Repo and Gerrit Discussion
On Mon, Jul 18, 2022 at 8:44 AM Mk <moha...@gmail.com> wrote:
Hello Matthias,


Please avoid top posting on this list as it makes it hard to follow the discussion thread.
Instead use interleaved posting and reply inline.
 
Except (All-Projects.git  & All-Users.git) i can't see other repos in my target server and in the source replication log constitutionally showing above provided message.

Still i'm i missing any configurations to replicate my source server gerrit data?

You didn't set gerrit.replicateOnStartup = true. This means replication is triggered only when some ref is updated.
If for a repository there was no refupdate since you configured replication then nothing will happen.
You can use the replicate start --all command to trigger an initial replication for all repos.
 
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/ef805cb3-f531-4363-b34a-aaee6da7cf1en%40googlegroups.com.

Mk

unread,
Jul 18, 2022, 4:29:13 AM7/18/22
to Repo and Gerrit Discussion
Please find my reply in line.

On Monday, July 18, 2022 at 1:11:29 PM UTC+5:30 Matthias Sohn wrote:
On Mon, Jul 18, 2022 at 8:44 AM Mk <moha...@gmail.com> wrote:
Hello Matthias,


Please avoid top posting on this list as it makes it hard to follow the discussion thread.
Instead use interleaved posting and reply inline.
 
Except (All-Projects.git  & All-Users.git) i can't see other repos in my target server and in the source replication log constitutionally showing above provided message.

Still i'm i missing any configurations to replicate my source server gerrit data?

You didn't set gerrit.replicateOnStartup = true. This means replication is triggered only when some ref is updated.
If for a repository there was no refupdate since you configured replication then nothing will happen.
You can use the replicate start --all command to trigger an initial replication for all repos.

My updated configurations file as follows,

$ cat replication.config
[remote "replica-1"]
  url = ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/${name}.git
  push = +refs/*:refs/*
  threads = 3
  timtout = 30
  rescheduleDelay = 15
  replicationDelay = 5
  mirror = true
  createMissingRepositories = true
  replicateProjectDeletions = true
  replicateHiddenProjects = true
  replicatePermissions = true

[gerrit]
  autoReload = true
  replicateOnStartup = false

[replication]
  lockErrorMaxRetries = 5
  maxRetries = 5

$ cat ~/.ssh/config
Host replica-1
  Hostname gerrit-review.com
  User admin
  Port 29418
  IdentityFile ~/.ssh/id_rsa
  StrictHostKeyChecking no
  PreferredAuthentications publickey

I have been using the below command to start the replication.

$ ssh -p 29418 ad...@gerrit-review.com replication start --all

Still my replication log shows below message and repos are not getting synced with my target server.

[2022-07-18 08:01:55,896] scheduling replication All-Projects:..all.. => ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/All-Projects.git
[2022-07-18 08:01:55,909] scheduled All-Projects:..all.. => [d1ca5d8d] push ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/All-Projects.git to run after 5s
[2022-07-18 08:01:55,912] scheduling replication All-Users:..all.. => ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/All-Users.git
[2022-07-18 08:01:55,913] scheduled All-Users:..all.. => [11e8f5f9] push ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/All-Users.git to run after 5s
[2022-07-18 08:01:55,936] scheduling replication android/framework:..all.. => ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/android/framework.git

[2022-07-18 08:02:00,911] Replication to ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/All-Projects.git started... [CONTEXT pushOneId="d1ca5d8d" ]
[2022-07-18 08:02:00,915] Replication to ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/All-Users.git started... [CONTEXT pushOneId="11e8f5f9" ]
[2022-07-18 08:02:00,939] Replication to ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/android/framework.git started... [CONTEXT pushOneId="51de6d58" ]
[2022-07-18 08:02:01,414] Created remote repository: ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/All-Projects.git [CONTEXT pushOneId="d1ca5d8d" ]
[2022-07-18 08:02:01,416] Missing repository created; retry replication to ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/All-Projects.git [CONTEXT pushOneId="d1ca5d8d" ]
[2022-07-18 08:02:01,423] Replication to ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/linux-uxstarted... [CONTEXT pushOneId="913f0571" ]
[2022-07-18 08:02:01,469] Created remote repository: ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/android/framework.git [CONTEXT pushOneId="51de6d58" ]
[2022-07-18 08:02:01,469] Missing repository created; retry replication to ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/android/framework.git [CONTEXT pushOneId="51de6d58" ]
[2022-07-18 08:02:01,473] Replication to ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/core/test-repo.git started... [CONTEXT pushOneId="d1457df6" ]
[2022-07-18 08:02:01,525] Created remote repository: ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/All-Users.git [CONTEXT pushOneId="11e8f5f9" ]

Matthias Sohn

unread,
Jul 18, 2022, 5:06:24 AM7/18/22
to Mk, Repo and Gerrit Discussion
Whenever replication is triggered a task is persisted as a file under $gerrit_site/data/replication/ref-updates.
It progresses from building to waiting to running during the lifecycle of the task.
When it finishes successfully the corresponding task file is deleted.

You can observe running replication tasks using the gerrit show-queue command

The replication log is written to $gerrit_site/logs/replication_log, you can use the pushOneId to filter the logs for a single
task being processed. In case there are errors they are logged to $gerrit_site/logs/error_log, again you can use the
pushOneId to filter for errors in error_log for a specific replication task.

Did you follow the recommendations for ssh configuration in 
 

Mk

unread,
Jul 18, 2022, 6:03:41 AM7/18/22
to Repo and Gerrit Discussion
All Git repositories have been replicated successfully after (index start projects --force) executing for replica server.

Since i have "push = +refs/*:refs/*" in my replication.config file, but changes are missing in my replica server.  How can I fix this problem?

Matthias Sohn

unread,
Jul 18, 2022, 2:41:58 PM7/18/22
to Mk, Repo and Gerrit Discussion
On Mon, Jul 18, 2022 at 12:03 PM Mk <moha...@gmail.com> wrote:
All Git repositories have been replicated successfully after (index start projects --force) executing for replica server.

Since i have "push = +refs/*:refs/*" in my replication.config file, but changes are missing in my replica server.  How can I fix this problem?

On Monday, July 18, 2022 at 2:36:24 PM UTC+5:30 Matthias Sohn wrote:
On Mon, Jul 18, 2022 at 10:29 AM Mk <moha...@gmail.com> wrote:
Please find my reply in line.

On Monday, July 18, 2022 at 1:11:29 PM UTC+5:30 Matthias Sohn wrote:
On Mon, Jul 18, 2022 at 8:44 AM Mk <moha...@gmail.com> wrote:
Hello Matthias,


Please avoid top posting on this list as it makes it hard to follow the discussion thread.
Instead use interleaved posting and reply inline.
 
Except (All-Projects.git  & All-Users.git) i can't see other repos in my target server and in the source replication log constitutionally showing above provided message.

Still i'm i missing any configurations to replicate my source server gerrit data?

You didn't set gerrit.replicateOnStartup = true. This means replication is triggered only when some ref is updated.
If for a repository there was no refupdate since you configured replication then nothing will happen.
You can use the replicate start --all command to trigger an initial replication for all repos.

My updated configurations file as follows,

$ cat replication.config
[remote "replica-1"]
  url = ssh://ad...@gerrit-review.com:29418/gerrit-storage/projects/${name}.git
  push = +refs/*:refs/*
  threads = 3
  timtout = 30

fix typo: s/timtout/timeout/
 

Martin Fick

unread,
Jul 18, 2022, 3:28:11 PM7/18/22
to Mk, Repo and Gerrit Discussion
On 7/18/22 12:21 AM, Mk wrote:
replication.config has below information.

 [remote "replica-1"]

  url = ssh://ad...@gerrit-review.com:29418/${name}.git

This url look problematic, generally replication should not push to Gerrit's ssh port. Either use ssh port 22, or setup git-daemon on the git port or over http(s) on the replica to receive pushes,

-Martin



Mohan

unread,
Jul 19, 2022, 8:31:36 AM7/19/22
to Martin Fick, Repo and Gerrit Discussion
Hello Martin & Matthias,

Thanks for your response. I have successfully 
replicated the data to target. However after replication on target server replicated repositories on GUI didn't show in the list automatically. I had to run (index start projects --force) to make it available on GUI. 

Also I can't see the changes. Though I had used (push = +refs/*:refs/*). Anything additionally I need to take care to view the grrrit changes?

Even I have executed (index start changes). Still same result.

Matthias Sohn

unread,
Jul 20, 2022, 5:57:35 AM7/20/22
to Mohan, Martin Fick, Repo and Gerrit Discussion
On Tue, Jul 19, 2022 at 2:31 PM Mohan <moha...@gmail.com> wrote:
Hello Martin & Matthias,

Thanks for your response. I have successfully 
replicated the data to target. However after replication on target server replicated repositories on GUI didn't show in the list automatically. I had to run (index start projects --force) to make it available on GUI. 

Also I can't see the changes. Though I had used (push = +refs/*:refs/*). Anything additionally I need to take care to view the grrrit changes?

Even I have executed (index start changes). Still same result.

I guess the serverId of the two instances doesn't match.
If the two instances are replica of each other it should match otherwise the Gerrit instance
at the receiving end of the replication cannot resolve accounts referenced from objects
being replicated. Only do this if also the All-Users repository is replicated.


On Tue, 19 Jul 2022, 12:58 am Martin Fick, <quic_...@quicinc.com> wrote:
On 7/18/22 12:21 AM, Mk wrote:
replication.config has below information.

 [remote "replica-1"]

  url = ssh://ad...@gerrit-review.com:29418/${name}.git

This url look problematic, generally replication should not push to Gerrit's ssh port. Either use ssh port 22, or setup git-daemon on the git port or over http(s) on the replica to receive pushes,

-Martin



--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Mohan

unread,
Jul 21, 2022, 8:30:01 AM7/21/22
to Matthias Sohn, Martin Fick, Repo and Gerrit Discussion
Hi Matthias,

Thanks. Serverid mismatch was the issue. 

Another query: Have created and merged couple of changes in master node, the same not getting loaded on replica GUI.

Should I include any additional configuration in my replication.config file? Please let me know. Thank you.

Björn Pedersen

unread,
Jul 21, 2022, 9:29:37 AM7/21/22
to Repo and Gerrit Discussion
Mk schrieb am Donnerstag, 21. Juli 2022 um 14:30:01 UTC+2:
Hi Matthias,

Thanks. Serverid mismatch was the issue. 

Another query: Have created and merged couple of changes in master node, the same not getting loaded on replica GUI.


I think you have a general misunderstanding of replication here. The default is to have a primary node that serves: 
 - all write requests
 - the gui

the replicas only serve read trafffic(no gui), basically git clones/fetches. There is a basic assumption that both sides run the same gerrit version.

From what you posted I deduce that you want to use it for a zero-downtime upgrade. This will only work if you start with two primary instances on the same(!) version and either a ha-setup (see ha-plugin docs) or a multi-master setup ( see multimaster plugin docs). Then follow the information in the release notes for a zero downtime upgrade. 
 

Should I include any additional configuration in my replication.config file? Please let me know. Thank you.

Matthias Sohn

unread,
Jul 21, 2022, 10:08:46 AM7/21/22
to Mohan, Martin Fick, Repo and Gerrit Discussion
On Thu, Jul 21, 2022 at 2:29 PM Mohan <moha...@gmail.com> wrote:
Hi Matthias,

Thanks. Serverid mismatch was the issue. 

Another query: Have created and merged couple of changes in master node, the same not getting loaded on replica GUI.

Gerrit replica don't serve neither GUI nor REST API and have no search indexes.

Mohan

unread,
Jul 22, 2022, 9:09:08 AM7/22/22
to Matthias Sohn, Martin Fick, Repo and Gerrit Discussion
Hi Matthias & Bjorn,

Ok. Thanks for the clarification. I was thinking that replica too serve completely as master node.

Thanks once again for all the help.

Luca Milanesio

unread,
Jul 23, 2022, 8:01:04 AM7/23/22
to Repo and Gerrit Discussion, Luca Milanesio, Matthias Sohn, Martin Fick, Mohan

On 22 Jul 2022, at 14:08, Mohan <moha...@gmail.com> wrote:

Hi Matthias & Bjorn,

Ok. Thanks for the clarification. I was thinking that replica too serve completely as master node.

You can use a multi-site setup [1] if you want all Gerrit nodes to be R/W and be fully functional.
I have presented the Gerrit multi-site architecture at the summit in Palo Alto, recording is available at [2].

HTH

Luca.



Mohan

unread,
Jul 25, 2022, 9:59:30 PM7/25/22
to Luca Milanesio, Repo and Gerrit Discussion, Matthias Sohn, Martin Fick
Hello Luca,

Right we are interested with multi master architecture setup and shared guidelines and video will be helpful for us. Thank you.
Reply all
Reply to author
Forward
0 new messages