Pull replication with https

140 views

Skip to first unread message

Kiran Shinde

unread,

Feb 14, 2025, 8:40:18 AMFeb 14

to Repo and Gerrit Discussion

Hello everyone,

I am trying to setup a Gerrit mirror with pull replication using https refering example from 
https://gerrit.googlesource.com/plugins/pull-replication/+/refs/heads/master/example-setup/http/

I have an existing QA master with some repos and I have setup a mirror with pull replication using above example.

replication.config on mirror

[gerrit]
    autoReload = true
    replicateOnStartup = true
[replication]
    excludeRefs = ^refs/users/\\d\\d/\\d+/edit-\\d+/\\d+$
    lockErrorMaxRetries = 5
    maxRetries = 100
    useCGitClient = false
    consumeStreamEvents = false
    syncRefs="ALL REFS ASYNC"
    maxApiPayloadSize=40000
[remote "gerrit-master"]
    url = http://master-ip:port/${name}.git
    apiUrl = http://master-ip:port
    fetch = +refs/*:refs/*
    mirror = true
    timeout = 60 # In seconds
    connectionTimeout = 120000 # In mseconds
    rescheduleDelay = 15
    replicationDelay = 1
    threads = 4
    createMissingRepositories = true
    replicateProjectDeletions = true
    replicateHiddenProjects = true
    tagopt= --no-tags


replication.config on master

[gerrit]
    autoReload = true
    replicateOnStartup = false
[replication]
    excludeRefs = ^refs/users/\\d\\d/\\d+/edit-\\d+/\\d+$
    lockErrorMaxRetries = 5
    maxRetries = 100
    useCGitClient = false
    consumeStreamEvents = false
    syncRefs="ALL REFS ASYNC"
    maxApiPayloadSize=40000
[remote "mirror-http-1"]
    url = http://mirror-ip:port/${name}.git
    fetch = +refs/*:refs/*
    mirror = true
    timeout = 60 # In seconds
    connectionTimeout = 120000 # In mseconds
    rescheduleDelay = 15
    replicationDelay = 1
    threads = 4
    createMissingRepositories = true
    replicateProjectDeletions = true
    replicateHiddenProjects = true
    tagopt= --no-tags



And when I start replication, I see that All-Projects and All-Users repository is getting replicated from mirror to master (please see below logs highlighted in red).
We lost all account from the test master and we had lost access to the test master which was resolved later using backups.


[2025-02-14 14:22:56,036] scheduling replication http://mirror-ip:port/All-Projects.git:..all.. => All-Projects
[2025-02-14 14:22:56,064] scheduled [bd167ee4] fetch http://mirror-ip:port/All-Projects.git []:..all.. => All-Projects to run after 1s
[2025-02-14 14:22:57,069] [bd167ee4] ASYNC replication from http://mirror-ip:port/All-Projects.git started for refs [[..all..]] ...
[2025-02-14 14:22:57,133] [bd167ee4] Fetch references [+refs/*:refs/*] from http://mirror-ip:port/All-Projects.git
[2025-02-14 14:22:57,313] [bd167ee4] Replication from http://mirror-ip:port/All-Projects.git completed in 241ms, 1027ms delay, 0 retries

I have some questions which I am not able to find answers for by reading documentation:

1. Do we need to have replication.config and pull-replication plugin on both master and mirror?
2. If we need replication.config on master as well, then what should be the configuration?
   Are there any specific settings that need to be done on mirror only and not on master?
3. If we have replication.config on both master and mirror, what configuration tells that master should not pull from mirror?

Kiran Shinde

unread,

Feb 17, 2025, 4:25:13 AMFeb 17

to Repo and Gerrit Discussion

Can someone please help?

Best Regards,

Kiran

lucamilanesio

unread,

Feb 21, 2025, 9:04:25 AMFeb 21

to Repo and Gerrit Discussion

Hi Kiran,

On Friday, February 14, 2025 at 1:40:18 PM UTC Kiran Shinde wrote:

Hello everyone,

I am trying to setup a Gerrit mirror with pull replication using https refering example from
https://gerrit.googlesource.com/plugins/pull-replication/+/refs/heads/master/example-setup/http/

Yep, that's a good starting point.

Also, you can watch the introduction to the pull-replication plugin on GerritForge TV channel [1].

That's just *one* example, but it isn't production-ready.

Some of the options can be questionable, like the exclusion of the edits in "excludeRefs" above and the exclusion of stream events consumption.

See the full set of configuration parameters at [2].

I don't believe you want to fetch from mirror, this isn't needed.

Only if you want to replicate on an event-based using the apply-object REST-API (with msec level latency) you need to set the apiUrl (I don't see that in your config).

In your example, you are fetching from the mirror, which would wipe out your data.

I saw this happening in the past and these mistakes causing damage: I still need to think about a "safety net" for preventing disaster caused by misconfiguration.

And when I start replication, I see that All-Projects and All-Users repository is getting replicated from mirror to master (please see below logs highlighted in red).
We lost all account from the test master and we had lost access to the test master which was resolved later using backups.

Yep, that's what you've configured: the master should fetch from the mirror.

[remote "mirror-http-1"]

    url = http://mirror-ip:port/${name}.git

    fetch = +refs/*:refs/*

[2025-02-14 14:22:56,036] scheduling replication http://mirror-ip:port/All-Projects.git:..all.. => All-Projects
[2025-02-14 14:22:56,064] scheduled [bd167ee4] fetch http://mirror-ip:port/All-Projects.git []:..all.. => All-Projects to run after 1s
[2025-02-14 14:22:57,069] [bd167ee4] ASYNC replication from http://mirror-ip:port/All-Projects.git started for refs [[..all..]] ...
[2025-02-14 14:22:57,133] [bd167ee4] Fetch references [+refs/*:refs/*] from http://mirror-ip:port/All-Projects.git
[2025-02-14 14:22:57,313] [bd167ee4] Replication from http://mirror-ip:port/All-Projects.git completed in 241ms, 1027ms delay, 0 retries

The above is the behaviour consistent with your configuration.

NOTE: The example you've used was relying on a single template file for master and mirror, possibly that's where the confusion came from.

I have some questions which I am not able to find answers for by reading documentation:

1. Do we need to have replication.config and pull-replication plugin on both master and mirror?

Yes and no, depending on what you want to achieve.

2. If we need replication.config on master as well, then what should be the configuration?

Same as before: it depends.

Are there any specific settings that need to be done on mirror only and not on master?

If the mirror should just pull from master, you just need the mirror's replication.config.

3. If we have replication.config on both master and mirror, what configuration tells that master should not pull from mirror?

Well, the master doesn't know that the other is a mirror, isn't it?

In your case, you called the remote "mirror" and have a "mirror-ip" but the master has no notion that the mirror in the name is actually a read-only mirror of the master.

[remote "mirror-http-1"]

    url = http://mirror-ip:port/${name}.git

I'd suggest to go through the videos and see how the whole thing works and then try with a simpler minimal setup, where the mirror fetches from master with a replication.config on the mirror only.

HTH

[1] https://www.youtube.com/watch?v=6rubGQE9v2E

[2] https://gerrit.googlesource.com/plugins/pull-replication/+/refs/heads/master/src/main/resources/Documentation/config.md