Pull-replication and readonly plugins

[Liem Do]

Is pull-replication supported with the readonly plugin and multi-site? I see the following errors when a commit is done and replication is attempted to a secondary node which has readonly plugin enabled. If pull-replcation and readonly plugins don't work together, what are the ways you can make a node in a multi-site cluster as a passive ready-only node and still have pull-replication working on it.

[2025-07-09 13:55:07,813] Pull replication REST API apply object to https://gerrit2.example.com COMPLETED for load-test:refs/changes/42/1194742/meta - [{commitObject=9751911cd865c1982b8ea39133a76da49f8d0f22 (COMMIT) treeObject=4b825dc642cb6eb9a060e54bf8d69288fbee4904 (TREE) }], HTTP Result: FAILED, status=503 'You are trying an operation which is not allowed in this readonly instance, please use evogit-push or contact git-s...@example.com

' - time:97.15252 ms

[2025-07-09 13:55:07,813] Pull replication REST API fetch to https://gerrit2.example.com for load-test:refs/changes/42/1194742/meta

Another question I have with pull-replication is that if you have a cluster of 3 Gerrit Nodes with multi-site enabled and pull-replication. Do I have the pull-replication remotes configured for the other 3 nodes on each of the nodes? So for example, would I have the following?

GerritNode1:

  GERRIT_SITE/etc/replication/node2.config

  GERRIT_SITE/etc/replication/node3.config

GerritNode2:

  GERRIT_SITE/etc/replication/node1.config

  GERRIT_SITE/etc/replication/node3.config

GerritNode3:

  GERRIT_SITE/etc/replication/node1.config

  GERRIT_SITE/etc/replication/node2.config

 Thanks,

Liem

[Luca Milanesio]

On 9 Jul 2025, at 22:18, Liem Do <lie...@gmail.com> wrote:

Is pull-replication supported with the readonly plugin and multi-site? I see the following errors when a commit is done and replication is attempted to a secondary node which has readonly plugin enabled. If pull-replcation and readonly plugins don't work together, what are the ways you can make a node in a multi-site cluster as a passive ready-only node and still have pull-replication working on it.

Well, the reason why I developed the read-only script (then turned into a plugin) was to make a node readonly for backups or concurrent GCs.

If the node would still accept replication, then it won’t be read-only after all and backups could be inconsistent or GCs could risk to remove BLOBs or revert refs.

What would be your use-case of having the read-only node? If you are just looking to have the node servicing the Git protocol read-only, why not enabling the replica mode in gerrit.config instead?

[2025-07-09 13:55:07,813] Pull replication REST API apply object to https://gerrit2.example.com COMPLETED for load-test:refs/changes/42/1194742/meta - [{commitObject=9751911cd865c1982b8ea39133a76da49f8d0f22 (COMMIT) treeObject=4b825dc642cb6eb9a060e54bf8d69288fbee4904 (TREE) }], HTTP Result: FAILED, status=503 'You are trying an operation which is not allowed in this readonly instance, please use evogit-push or contact git-s...@example.com

' - time:97.15252 ms

[2025-07-09 13:55:07,813] Pull replication REST API fetch to https://gerrit2.example.com for load-test:refs/changes/42/1194742/meta

That’s expected.

Another question I have with pull-replication is that if you have a cluster of 3 Gerrit Nodes with multi-site enabled and pull-replication. Do I have the pull-replication remotes configured for the other 3 nodes on each of the nodes? So for example, would I have the following?

GerritNode1:

  GERRIT_SITE/etc/replication/node2.config

  GERRIT_SITE/etc/replication/node3.config

GerritNode2:

  GERRIT_SITE/etc/replication/node1.config

  GERRIT_SITE/etc/replication/node3.config

GerritNode3:

  GERRIT_SITE/etc/replication/node1.config

  GERRIT_SITE/etc/replication/node2.config

Yes, that’s correct.

HTH

Luca.

[Liem Do]

Luca,

On Wednesday, July 9, 2025 at 2:31:09 PM UTC-7 Luca Milanesio wrote:

On 9 Jul 2025, at 22:18, Liem Do <lie...@gmail.com> wrote:

Is pull-replication supported with the readonly plugin and multi-site? I see the following errors when a commit is done and replication is attempted to a secondary node which has readonly plugin enabled. If pull-replcation and readonly plugins don't work together, what are the ways you can make a node in a multi-site cluster as a passive ready-only node and still have pull-replication working on it.

Well, the reason why I developed the read-only script (then turned into a plugin) was to make a node readonly for backups or concurrent GCs.

If the node would still accept replication, then it won’t be read-only after all and backups could be inconsistent or GCs could risk to remove BLOBs or revert refs.

That make sense.

 

What would be your use-case of having the read-only node? If you are just looking to have the node servicing the Git protocol read-only, why not enabling the replica mode in gerrit.config instead?

This was the way we had originally deployed ha and multi-site with refdb disabled and push replication with direction from Gerritforge if you recall. I was just trying to setup pull-replication in our staging environment which has all the same config as we do in production. So in a multi-site cluster, if you put one node into read-only mode and after some time take it out of read-only mode while other nodes were still taking on writes, would the pull-replication have to be initiated manually to catch up with all the changes from the other nodes and would you do that from the node that was read-only or from one of the other master nodes?

[2025-07-09 13:55:07,813] Pull replication REST API apply object to https://gerrit2.example.com COMPLETED for load-test:refs/changes/42/1194742/meta - [{commitObject=9751911cd865c1982b8ea39133a76da49f8d0f22 (COMMIT) treeObject=4b825dc642cb6eb9a060e54bf8d69288fbee4904 (TREE) }], HTTP Result: FAILED, status=503 'You are trying an operation which is not allowed in this readonly instance, please use evogit-push or contact git-s...@example.com

' - time:97.15252 ms

[2025-07-09 13:55:07,813] Pull replication REST API fetch to https://gerrit2.example.com for load-test:refs/changes/42/1194742/meta

That’s expected.

Another question I have with pull-replication is that if you have a cluster of 3 Gerrit Nodes with multi-site enabled and pull-replication. Do I have the pull-replication remotes configured for the other 3 nodes on each of the nodes? So for example, would I have the following?

GerritNode1:

  GERRIT_SITE/etc/replication/node2.config

  GERRIT_SITE/etc/replication/node3.config

GerritNode2:

  GERRIT_SITE/etc/replication/node1.config

  GERRIT_SITE/etc/replication/node3.config

GerritNode3:

  GERRIT_SITE/etc/replication/node1.config

  GERRIT_SITE/etc/replication/node2.config

Yes, that’s correct.

Thanks for confirming!

 

HTH

Luca.

[Luca Milanesio]

On 10 Jul 2025, at 00:17, Liem Do <lie...@gmail.com> wrote:

Luca,

On Wednesday, July 9, 2025 at 2:31:09 PM UTC-7 Luca Milanesio wrote:

On 9 Jul 2025, at 22:18, Liem Do <lie...@gmail.com> wrote:

Is pull-replication supported with the readonly plugin and multi-site? I see the following errors when a commit is done and replication is attempted to a secondary node which has readonly plugin enabled. If pull-replcation and readonly plugins don't work together, what are the ways you can make a node in a multi-site cluster as a passive ready-only node and still have pull-replication working on it.

Well, the reason why I developed the read-only script (then turned into a plugin) was to make a node readonly for backups or concurrent GCs.

If the node would still accept replication, then it won’t be read-only after all and backups could be inconsistent or GCs could risk to remove BLOBs or revert refs.

That make sense.

 

What would be your use-case of having the read-only node? If you are just looking to have the node servicing the Git protocol read-only, why not enabling the replica mode in gerrit.config instead?

This was the way we had originally deployed ha and multi-site with refdb disabled and push replication with direction from Gerritforge if you recall. I was just trying to setup pull-replication in our staging environment which has all the same config as we do in production. So in a multi-site cluster, if you put one node into read-only mode and after some time take it out of read-only mode while other nodes were still taking on writes, would the pull-replication have to be initiated manually to catch up with all the changes from the other nodes and would you do that from the node that was read-only or from one of the other master nodes?

That’s the way pull-replication works, spot on :-)

If the node is offline or read-only, the replication is suspended and the missing events accumulated on the events-broker.

I would recommend the following sequence to bring the node in “read-only” and safely recover all the missing replication events:

1. Unload the events-* plugin (e.g. events-kafka) - all replication events generated *after* this point will be automatically replayed

2. Set the node as readonly - no new replication actions will take place

3. Unload the pull-replication plugin

Once the readonly phase is over, you can then re-enable them in reverse

1. Load the pull-replication plugin

2. Set the node as read-write

3. LOad the events-* plugin (e.g. events-kafka)

The above will assure that you can do *any sort of maintenance* without the repo moving or missing any replication events after the readonly window.

Read-only windows are useful for having a stable and consitent view of the data.

NOTE: Also remember to set the node as unheatlhy, so that nobody will access it whilst it is in read only mode.

HTH

Luca.

[2025-07-09 13:55:07,813] Pull replication REST API apply object to https://gerrit2.example.com COMPLETED for load-test:refs/changes/42/1194742/meta - [{commitObject=9751911cd865c1982b8ea39133a76da49f8d0f22 (COMMIT) treeObject=4b825dc642cb6eb9a060e54bf8d69288fbee4904 (TREE) }], HTTP Result: FAILED, status=503 'You are trying an operation which is not allowed in this readonly instance, please use evogit-push or contact git-s...@example.com

' - time:97.15252 ms

[2025-07-09 13:55:07,813] Pull replication REST API fetch to https://gerrit2.example.com for load-test:refs/changes/42/1194742/meta

That’s expected.

Another question I have with pull-replication is that if you have a cluster of 3 Gerrit Nodes with multi-site enabled and pull-replication. Do I have the pull-replication remotes configured for the other 3 nodes on each of the nodes? So for example, would I have the following?

GerritNode1:

  GERRIT_SITE/etc/replication/node2.config

  GERRIT_SITE/etc/replication/node3.config

GerritNode2:

  GERRIT_SITE/etc/replication/node1.config

  GERRIT_SITE/etc/replication/node3.config

GerritNode3:

  GERRIT_SITE/etc/replication/node1.config

  GERRIT_SITE/etc/replication/node2.config

Yes, that’s correct.

Thanks for confirming!

 

HTH

Luca.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/3a505fc0-e8af-4c45-9e8c-00fa68140bedn%40googlegroups.com.