Pull replication events

[Finn Ashford]

Hello everyone,

We are currently working on setting up pull replication between a Gerrit primary and its replicas. The replication itself appears to be functioning as expected.

Our goal is to integrate these pull replicas with our CI system. Specifically, we want the CI to poll the Gerrit primary's stream events and trigger jobs only after replication has completed, similar to how the Gerrit Trigger plugin for Jenkins works.

With push replication, the primary generates events like "ref-replicated" and "ref-replication-scheduled", which the CI can use to determine replication status.

However, we’ve noticed that these events are not generated by the primary when using pull replication.

Is there an alternative approach or recommended way to handle this scenario when working with pull replicas?

- Finn

[Luca Milanesio]

Hi Finn,

On 14 Apr 2025, at 13:57, Finn Ashford <finnas...@gmail.com> wrote:

Hello everyone,

We are currently working on setting up pull replication between a Gerrit primary and its replicas. The replication itself appears to be functioning as expected.

Our goal is to integrate these pull replicas with our CI system. Specifically, we want the CI to poll the Gerrit primary's stream events and trigger jobs only after replication has completed, similar to how the Gerrit Trigger plugin for Jenkins works.

With push replication, the primary generates events like "ref-replicated" and "ref-replication-scheduled", which the CI can use to determine replication status.

However, we’ve noticed that these events are not generated by the primary when using pull replication.

The replication plugin works asynchronously, therefore they are always generated.

The pull-replication plugin *may* work both ways, therefore those events may not be generated.

Can you share your replication.config on both ends?

Luca.

[Finn Ashford]

Hi Luca,

Thank you for your response. Please find the replication configurations for both the primary and the replica environments below.

replication.config on primary

[gerrit]

autoReload = true

replicateOnStartup = false

[replication]

maxApiPayloadSize = 40000

[remote "gerrit-push-replica-1"]

url = ssh://gerrit@10.x.x.6/var/gerrit/git/${name}.git

threads = 8

timeout = 600

createMissingRepositories = true

replicateProjectDeletions = true

replicatePermissions = true

replicateNoteDbMetaRefs = false

mirror = true

push = +refs/*:refs/*

rescheduleDelay = 15

replicationDelay = 15

[remote "gerrit-pull-replica-1"]

url = http://10.x.x.10:8080/${name}.git

apiUrl = http://10.x.x.10:8080/

fetch = +refs/*:refs/*

mirror = true

timeout = 60 # In seconds

connectionTimeout = 120000 # In mseconds

rescheduleDelay = 15

replicationDelay = 1

threads = 4

createMissingRepositories = true

replicateProjectDeletions = true

replication.config on replica

[gerrit]

autoReload = true

replicateOnStartup = true

[replication]

refsBatchSize = 20

lockErrorMaxRetries = 5

maxRetries = 100

useCGitClient = false

eventBrokerTopic = gerrit

[remote "primary-gerrit"]

url = https://primary-gerrit.net/${name}.git

fetch = +refs/*:refs/*

mirror = true

timeout = 60 # In seconds

connectionTimeout = 120000 # In mseconds

rescheduleDelay = 15

replicationDelay = 1

threads = 4

createMissingRepositories = true

replicateProjectDeletions = true

 

- Finn

Luca.

[Luca Milanesio]

> On 14 Apr 2025, at 14:18, Finn Ashford <finnas...@gmail.com> wrote:
>
>
>
> On Monday, April 14, 2025 at 3:01:56 PM UTC+2 Luca Milanesio wrote:
> Hi Finn,
>
>
>> On 14 Apr 2025, at 13:57, Finn Ashford <finnas...@gmail.com> wrote:
>>
>> Hello everyone,
>> We are currently working on setting up pull replication between a Gerrit primary and its replicas. The replication itself appears to be functioning as expected.
>> Our goal is to integrate these pull replicas with our CI system. Specifically, we want the CI to poll the Gerrit primary's stream events and trigger jobs only after replication has completed, similar to how the Gerrit Trigger plugin for Jenkins works.
>> With push replication, the primary generates events like "ref-replicated" and "ref-replication-scheduled", which the CI can use to determine replication status.
>> However, we’ve noticed that these events are not generated by the primary when using pull replication.
>
> The replication plugin works asynchronously, therefore they are always generated.
> The pull-replication plugin *may* work both ways, therefore those events may not be generated.
>
> Can you share your replication.config on both ends?
>
> Hi Luca,
>
> Thank you for your response. Please find the replication configurations for both the primary and the replica environments below.
>
> replication.config on primary
>
> [gerrit]
> autoReload = true
> replicateOnStartup = false
> [replication]
> maxApiPayloadSize = 40000

You do not specify anything in ‘replication.syncRefs’, therefore all replication operations are synchronous, see [1].
That means that relying on ‘ref-updated’ events is enough, and as soon as you get that event it means the ref is already replicated to the replicas.

HTH

Luca.

[1] https://gerrit.googlesource.com/plugins/pull-replication/+/refs/heads/master/src/main/resources/Documentation/config.md


> [remote "gerrit-push-replica-1"]
> url = ssh://ger...@10.x.x.6/var/gerrit/git/${name}.git

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/a34acbc8-7b5e-4c36-92fb-e6ee3e66671cn%40googlegroups.com.

[Finn Ashford]

That makes a lot more sense now 😊

Thank you very much, Luca, for the prompt response and clarification.

- Finn

[Finn Ashford]

Hello again,

While reviewing the ref-updated events in the context of pull replication, I noticed an interesting behavior.

We have two replicas configured. During testing, one of the replicas was intentionally brought down, while the other remained healthy. A new repository was created on the primary, and the change was successfully replicated to the healthy replica. As expected, the replication to the offline replica failed with a connection timeout.

However, despite one replica not being updated, the Gerrit primary still emitted a ref-updated event in the stream.

This suggests that the ref-updated event is emitted even if replication to all configured replicas hasn't fully succeeded.

Is there a known way to mitigate this situation?

Regards,

Finn

[Luca Milanesio]

On 15 Apr 2025, at 10:38, Finn Ashford <finnas...@gmail.com> wrote:

Hello again,

While reviewing the ref-updated events in the context of pull replication, I noticed an interesting behavior.

We have two replicas configured. During testing, one of the replicas was intentionally brought down, while the other remained healthy. A new repository was created on the primary, and the change was successfully replicated to the healthy replica. As expected, the replication to the offline replica failed with a connection timeout.

However, despite one replica not being updated, the Gerrit primary still emitted a ref-updated event in the stream.

This suggests that the ref-updated event is emitted even if replication to all configured replicas hasn't fully succeeded.

ref-updated isn’t a replication event: it just says that the ref has been updated on the primary node.

If the synchronous replication has succeeded, it has been replicated also on the replicas. In your case, the replica was offline, therefore it hasn’t succedded and it won’t be able to serve any traffic anyway.

Is there a known way to mitigate this situation?

When a node is offline, it can start replicating again when it comes back online, using two mechanisms:

- Catch-up via events-broker (see replication.eventBrokerTopic)

- replicateOnStartup = true

For more details on those settings, you can refer to [1].

Both of the above are async replication mechanisms, therefore they will emit the replication stream events.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/53199b00-fc27-44f7-84b8-ba3daa35d745n%40googlegroups.com.

[Luca Milanesio]

> On 15 Apr 2025, at 10:46, Luca Milanesio <luca.mi...@gmail.com> wrote:
>
>
>
>> On 15 Apr 2025, at 10:38, Finn Ashford <finnas...@gmail.com> wrote:
>>
>> Hello again,
>> While reviewing the ref-updated events in the context of pull replication, I noticed an interesting behavior.
>> We have two replicas configured. During testing, one of the replicas was intentionally brought down, while the other remained healthy. A new repository was created on the primary, and the change was successfully replicated to the healthy replica. As expected, the replication to the offline replica failed with a connection timeout.
>> However, despite one replica not being updated, the Gerrit primary still emitted a ref-updated event in the stream.
>> This suggests that the ref-updated event is emitted even if replication to all configured replicas hasn't fully succeeded.
>
> ref-updated isn’t a replication event: it just says that the ref has been updated on the primary node.
> If the synchronous replication has succeeded, it has been replicated also on the replicas. In your case, the replica was offline, therefore it hasn’t succedded and it won’t be able to serve any traffic anyway.
>>
>>
>> Is there a known way to mitigate this situation?
>
> When a node is offline, it can start replicating again when it comes back online, using two mechanisms:
>
> - Catch-up via events-broker (see replication.eventBrokerTopic)
> - replicateOnStartup = true
>
> For more details on those settings, you can refer to [1].
>
> Both of the above are async replication mechanisms, therefore they will emit the replication stream events.

One correction on the above statement: the Gerrit replicas *DO NOT* emit stream events, therefore you won’t see those events generated, unless you set `replica = false` on the gerrit.config and install the read-only plugin.
Apologies for the confusion.

HTH

Luca.

[Finn Ashford]

On Tuesday, April 15, 2025 at 11:53:04 AM UTC+2 Luca Milanesio wrote:

> On 15 Apr 2025, at 10:46, Luca Milanesio <luca.mi...@gmail.com> wrote:
>
>
>
>> On 15 Apr 2025, at 10:38, Finn Ashford <finnas...@gmail.com> wrote:
>>
>> Hello again,
>> While reviewing the ref-updated events in the context of pull replication, I noticed an interesting behavior.
>> We have two replicas configured. During testing, one of the replicas was intentionally brought down, while the other remained healthy. A new repository was created on the primary, and the change was successfully replicated to the healthy replica. As expected, the replication to the offline replica failed with a connection timeout.
>> However, despite one replica not being updated, the Gerrit primary still emitted a ref-updated event in the stream.
>> This suggests that the ref-updated event is emitted even if replication to all configured replicas hasn't fully succeeded.
>
> ref-updated isn’t a replication event: it just says that the ref has been updated on the primary node.
> If the synchronous replication has succeeded, it has been replicated also on the replicas. In your case, the replica was offline, therefore it hasn’t succedded and it won’t be able to serve any traffic anyway.
>>
>>
>> Is there a known way to mitigate this situation?
>
> When a node is offline, it can start replicating again when it comes back online, using two mechanisms:
>
> - Catch-up via events-broker (see replication.eventBrokerTopic)
> - replicateOnStartup = true
>
> For more details on those settings, you can refer to [1].
>
> Both of the above are async replication mechanisms, therefore they will emit the replication stream events.


One correction on the above statement: the Gerrit replicas *DO NOT* emit stream events, therefore you won’t see those events generated, unless you set `replica = false` on the gerrit.config and install the read-only plugin.
Apologies for the confusion.

If we set replica = false, will the apply-object operation from the primary to the replica still function, given that it uses a POST request?
Additionally, will git fetch continue to work on a read-only Gerrit instance?

Thanks,

Finn

 

[Luca Milanesio]

Of could it will.

Additionally, will git fetch continue to work on a read-only Gerrit instance?

Yep.

Luca.

Thanks,

Finn

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/69525f47-f453-45b3-bdf8-28a894a1624cn%40googlegroups.com.

[Luca Milanesio]

On 15 Apr 2025, at 22:41, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 15 Apr 2025, at 12:53, Finn Ashford <finnas...@gmail.com> wrote:

On Tuesday, April 15, 2025 at 11:53:04 AM UTC+2 Luca Milanesio wrote:

> On 15 Apr 2025, at 10:46, Luca Milanesio <luca.mi...@gmail.com> wrote:
>
>
>
>> On 15 Apr 2025, at 10:38, Finn Ashford <finnas...@gmail.com> wrote:
>>
>> Hello again,
>> While reviewing the ref-updated events in the context of pull replication, I noticed an interesting behavior.
>> We have two replicas configured. During testing, one of the replicas was intentionally brought down, while the other remained healthy. A new repository was created on the primary, and the change was successfully replicated to the healthy replica. As expected, the replication to the offline replica failed with a connection timeout.
>> However, despite one replica not being updated, the Gerrit primary still emitted a ref-updated event in the stream.
>> This suggests that the ref-updated event is emitted even if replication to all configured replicas hasn't fully succeeded.
>
> ref-updated isn’t a replication event: it just says that the ref has been updated on the primary node.
> If the synchronous replication has succeeded, it has been replicated also on the replicas. In your case, the replica was offline, therefore it hasn’t succedded and it won’t be able to serve any traffic anyway.
>>
>>
>> Is there a known way to mitigate this situation?
>
> When a node is offline, it can start replicating again when it comes back online, using two mechanisms:
>
> - Catch-up via events-broker (see replication.eventBrokerTopic)
> - replicateOnStartup = true
>
> For more details on those settings, you can refer to [1].
>
> Both of the above are async replication mechanisms, therefore they will emit the replication stream events.


One correction on the above statement: the Gerrit replicas *DO NOT* emit stream events, therefore you won’t see those events generated, unless you set `replica = false` on the gerrit.config and install the read-only plugin.
Apologies for the confusion.

If we set replica = false, will the apply-object operation from the primary to the replica still function, given that it uses a POST request?

Of could it will.

I need to withdraw my statement on this point; you need a fix on this line of code:

https://gerrit.googlesource.com/plugins/readonly/+/refs/heads/master/src/main/java/com/googlesource/gerrit/plugins/readonly/ReadOnly.java#90

Everything else is confirmed.

Luca.

[Finn Ashford]

Thanks again Luca.