On 3 Dec 2021, at 14:36, Thomas Dräbing <thomas....@gmail.com> wrote:Hi all,some time ago I opened issue 15194 [1]. The issue is that for huge, busy projects it can happen that a replication push contains the change ref (refs/changes/01/1/1), but not the meta ref of the change (refs/changes/01/1/meta). If the replication delay is significant, it can be minutes until the meta ref reaches the replica. Since the meta ref is required to fetch a change due to permission checks, clients fail to fetch the change even after the ref-replicated event was fired for the change.I pushed a change [2] that fixes that by always adding the meta ref to the push task, if a change ref is being pushed. This solution however is not very nice. Often enough the meta ref does not have to be replicated. While jgit won't push it, if the remote ref is up-to-date, adding it to the task already might impact performance.Matthias had the proposal that instead of having separate ref-updated events for the change ref and the meta-ref, which in turn lead to them being scheduled for replication, to have a single event listing both refs, since they are created with a batch ref update anyway. This would solve the duplicate addition of the meta ref in my change, but would require a change in Gerrit core that will likely affect other areas.
What do you think? Reviews and ideas are welcome!Thanks,Thomas
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAG7bb4AekueTeECXxD2ZQBB0LYXh9uab9SYEyXPnk_U6ARNLOg%40mail.gmail.com.
On Dec 3, 2021, at 2:30 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:On 3 Dec 2021, at 14:36, Thomas Dräbing <thomas....@gmail.com> wrote:Hi all,some time ago I opened issue 15194 [1]. The issue is that for huge, busy projects it can happen that a replication push contains the change ref (refs/changes/01/1/1), but not the meta ref of the change (refs/changes/01/1/meta). If the replication delay is significant, it can be minutes until the meta ref reaches the replica. Since the meta ref is required to fetch a change due to permission checks, clients fail to fetch the change even after the ref-replicated event was fired for the change.I pushed a change [2] that fixes that by always adding the meta ref to the push task, if a change ref is being pushed. This solution however is not very nice. Often enough the meta ref does not have to be replicated. While jgit won't push it, if the remote ref is up-to-date, adding it to the task already might impact performance.Matthias had the proposal that instead of having separate ref-updated events for the change ref and the meta-ref, which in turn lead to them being scheduled for replication, to have a single event listing both refs, since they are created with a batch ref update anyway. This would solve the duplicate addition of the meta ref in my change, but would require a change in Gerrit core that will likely affect other areas.I agree with Matthias’s proposal, however, we should formulate it in a forward-compatible way.One idea could be listing the full set of refs included in the batch ref-update in the ref-updated event but as a separate field.For those who are still looking at the old field, they’ll get two separate events. For those looking at the new field, they’ll get both events in one go.
The drawback is that you may receive one event twice, which is a minor issue IMHO.WDYT?Luca.What do you think? Reviews and ideas are welcome!Thanks,Thomas--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAG7bb4AekueTeECXxD2ZQBB0LYXh9uab9SYEyXPnk_U6ARNLOg%40mail.gmail.com.--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/E54404AE-1245-4C06-86CE-E9C9B1C22A7F%40gmail.com.
On Dec 3, 2021, at 2:30 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:On 3 Dec 2021, at 14:36, Thomas Dräbing <thomas....@gmail.com> wrote:Hi all,some time ago I opened issue 15194 [1]. The issue is that for huge, busy projects it can happen that a replication push contains the change ref (refs/changes/01/1/1), but not the meta ref of the change (refs/changes/01/1/meta). If the replication delay is significant, it can be minutes until the meta ref reaches the replica. Since the meta ref is required to fetch a change due to permission checks, clients fail to fetch the change even after the ref-replicated event was fired for the change.I pushed a change [2] that fixes that by always adding the meta ref to the push task, if a change ref is being pushed. This solution however is not very nice. Often enough the meta ref does not have to be replicated. While jgit won't push it, if the remote ref is up-to-date, adding it to the task already might impact performance.Matthias had the proposal that instead of having separate ref-updated events for the change ref and the meta-ref, which in turn lead to them being scheduled for replication, to have a single event listing both refs, since they are created with a batch ref update anyway. This would solve the duplicate addition of the meta ref in my change, but would require a change in Gerrit core that will likely affect other areas.I agree with Matthias’s proposal, however, we should formulate it in a forward-compatible way.One idea could be listing the full set of refs included in the batch ref-update in the ref-updated event but as a separate field.For those who are still looking at the old field, they’ll get two separate events. For those looking at the new field, they’ll get both events in one go.I don’t think an event/class that’s singular (ref-updated) should have information about multiple refs. That’s going to lead to confusion.I do think it would be great to have a new refs-updated event with a list of all updated refs and I think that particularly makes sense for the batch update use cases.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/84B85664-9967-4554-9F3A-0B75311D6327%40codeaurora.org.
On Fri, Dec 3, 2021 at 10:54 PM Nasser Grainawi <nas...@codeaurora.org> wrote:On Dec 3, 2021, at 2:30 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:On 3 Dec 2021, at 14:36, Thomas Dräbing <thomas....@gmail.com> wrote:Hi all,some time ago I opened issue 15194 [1]. The issue is that for huge, busy projects it can happen that a replication push contains the change ref (refs/changes/01/1/1), but not the meta ref of the change (refs/changes/01/1/meta). If the replication delay is significant, it can be minutes until the meta ref reaches the replica. Since the meta ref is required to fetch a change due to permission checks, clients fail to fetch the change even after the ref-replicated event was fired for the change.I pushed a change [2] that fixes that by always adding the meta ref to the push task, if a change ref is being pushed. This solution however is not very nice. Often enough the meta ref does not have to be replicated. While jgit won't push it, if the remote ref is up-to-date, adding it to the task already might impact performance.Matthias had the proposal that instead of having separate ref-updated events for the change ref and the meta-ref, which in turn lead to them being scheduled for replication, to have a single event listing both refs, since they are created with a batch ref update anyway. This would solve the duplicate addition of the meta ref in my change, but would require a change in Gerrit core that will likely affect other areas.I agree with Matthias’s proposal, however, we should formulate it in a forward-compatible way.One idea could be listing the full set of refs included in the batch ref-update in the ref-updated event but as a separate field.For those who are still looking at the old field, they’ll get two separate events. For those looking at the new field, they’ll get both events in one go.I don’t think an event/class that’s singular (ref-updated) should have information about multiple refs. That’s going to lead to confusion.I do think it would be great to have a new refs-updated event with a list of all updated refs and I think that particularly makes sense for the batch update use cases.+1, we could introduce a new refs-updated event and deprecate the old ref-updated event and remove it in a future release.This way existing consumers of the old event have time to migrate.Sending all refs in the old ref-updated event would be confusing (didn't check if this could be done in a compatible way)and send the same information multiple times. If e.g. a batch ref update updates 5 refs we would send 5 events eachcontaining all the 5 updated refs. If we introduce a new refs-updated event we send 5 ref-updated events each containinga single ref and 1 refs-updated event containing all 5.I think that using the same transaction boundaries for (batch) ref update and for replication has the potential to eliminatea class of consistency problems like the one Thomas wants to fix.
On Fri, 3 Dec 2021 at 23:45, Matthias Sohn <matthi...@gmail.com> wrote:On Fri, Dec 3, 2021 at 10:54 PM Nasser Grainawi <nas...@codeaurora.org> wrote:On Dec 3, 2021, at 2:30 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:On 3 Dec 2021, at 14:36, Thomas Dräbing <thomas....@gmail.com> wrote:Hi all,some time ago I opened issue 15194 [1]. The issue is that for huge, busy projects it can happen that a replication push contains the change ref (refs/changes/01/1/1), but not the meta ref of the change (refs/changes/01/1/meta). If the replication delay is significant, it can be minutes until the meta ref reaches the replica. Since the meta ref is required to fetch a change due to permission checks, clients fail to fetch the change even after the ref-replicated event was fired for the change.I pushed a change [2] that fixes that by always adding the meta ref to the push task, if a change ref is being pushed. This solution however is not very nice. Often enough the meta ref does not have to be replicated. While jgit won't push it, if the remote ref is up-to-date, adding it to the task already might impact performance.Matthias had the proposal that instead of having separate ref-updated events for the change ref and the meta-ref, which in turn lead to them being scheduled for replication, to have a single event listing both refs, since they are created with a batch ref update anyway. This would solve the duplicate addition of the meta ref in my change, but would require a change in Gerrit core that will likely affect other areas.I agree with Matthias’s proposal, however, we should formulate it in a forward-compatible way.One idea could be listing the full set of refs included in the batch ref-update in the ref-updated event but as a separate field.For those who are still looking at the old field, they’ll get two separate events. For those looking at the new field, they’ll get both events in one go.I don’t think an event/class that’s singular (ref-updated) should have information about multiple refs. That’s going to lead to confusion.I do think it would be great to have a new refs-updated event with a list of all updated refs and I think that particularly makes sense for the batch update use cases.+1, we could introduce a new refs-updated event and deprecate the old ref-updated event and remove it in a future release.This way existing consumers of the old event have time to migrate.Sending all refs in the old ref-updated event would be confusing (didn't check if this could be done in a compatible way)and send the same information multiple times. If e.g. a batch ref update updates 5 refs we would send 5 events eachcontaining all the 5 updated refs. If we introduce a new refs-updated event we send 5 ref-updated events each containinga single ref and 1 refs-updated event containing all 5.I think that using the same transaction boundaries for (batch) ref update and for replication has the potential to eliminatea class of consistency problems like the one Thomas wants to fix.I agree with Nasser and Matthias on this. There should be options to enable either or both of the events to support components that still use the ref-updated event and components that use refs-updated events at the same time, e.g. Jenkins with gerrit-trigger plugin, which still listens to ref-updated, and the replication plugin, which might already use the refs-updated event.I will have a look at that and push a change adding the refs-updated event as soon as I have something working.
To cater to this case and keep the old ref-updated event backwards compatible (for all but meta refs, which there shouldn't be that many "external" consumers of) how about
1. Update the ref-update events with field metaRefUpdate.
or2. To cater to the case where only the meta-ref is updated but not the patchset, we could add metaRefUpdate to ref-update event but also introduce a new type "meta-ref-update" for the "only meta-ref" case.That way existing consumers, who are already filtering for type "ref-update", wont have to deal with the "only meta-ref update" case.
An additional benefit is that we could prioritize meta-ref-updates in replication so that f.i. ACL changes are more likely to be replicated before other other ref-updates.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/6e542a7a-8ea0-4e9b-bb59-92ac192c4729n%40googlegroups.com.
or2. To cater to the case where only the meta-ref is updated but not the patchset, we could add metaRefUpdate to ref-update event but also introduce a new type "meta-ref-update" for the "only meta-ref" case.That way existing consumers, who are already filtering for type "ref-update", wont have to deal with the "only meta-ref update" case.This would also require the adaptation of quite some event listeners. Also logically there should then be a meta-ref-update event, even if a change ref is updated at the same time, because maybe a listener is only interested in meta-refs. So we would fire additional events with duplicated data.
An additional benefit is that we could prioritize meta-ref-updates in replication so that f.i. ACL changes are more likely to be replicated before other other ref-updates.We would rather have to prioritise the firing of the events, meta-ref before change-ref, since the replication plugin will collect refs for a push for a configured amount of time after the first ref-updated event. What happened in the issue we saw was, that on a busy project, if the ref-updated for the change ref was received first and the ref-updated event of the meta ref only later after the push request was sent out, they were not replicated at the same time. It might be a solution to order the event firing to fire ref-updated for the meta-ref first. I am not sure if that wouldn't cause different issues.
I still think it would make sense to inform listeners of the complete connected change that was done over multiple refs in a single event. Basically, the event would not have
public String getRefName()butpublic Set<String> getRefNames()And as discussed, we should stay backward compatible at least for a while by allowing to still use ref-updated either alone or at the same time as refs-updated.On the other hand, users voiced concerns about big changes in existing behaviour during the user summit and I would include replacing the ref-updated event with a refs-updated event, even if gradually, into this category. So finding a different solution would certainly be preferable for a considerable fraction of Gerrit's user base.
We could potentially also solve this problem by switching to refs-updated internally but for outward facing events (stream-events and events-.* plugins) split them into ref-updated events for backwards compatibility.
public String getRefName()butpublic Set<String> getRefNames()And as discussed, we should stay backward compatible at least for a while by allowing to still use ref-updated either alone or at the same time as refs-updated.On the other hand, users voiced concerns about big changes in existing behaviour during the user summit and I would include replacing the ref-updated event with a refs-updated event, even if gradually, into this category. So finding a different solution would certainly be preferable for a considerable fraction of Gerrit's user base.This is my main concern also, if possible keep backwards compatibility in outward facing APIs (such as REST, events).
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/bcace29d-2431-4e66-8525-75e89070196fn%40googlegroups.com.
Just a thought.
The drawback is that you may receive one event twice, which is a minor issue IMHO.WDYT?Luca.What do you think? Reviews and ideas are welcome!Thanks,Thomas--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAG7bb4AekueTeECXxD2ZQBB0LYXh9uab9SYEyXPnk_U6ARNLOg%40mail.gmail.com.
(i.e. the plugin splits before publishing to event-bus)
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/19726b54-17fa-4010-9396-b51d38206f3dn%40googlegroups.com.