[3.5.4] Re-running copy-approvals constantly updates same changesets

[Ian Wienand]

Hello,

We are running 3.5.4 and preparing to update to 3.6, and as part of that am running copy-approvals on our instance.

Out of caution I did this separately for each project in ls-projects, just so I could separate out logs for each project in case one failed. I then re-ran copy-approvals, expecting it to do nothing on the second run, but found one of our projects constantly has 24 changes that are reported as "updated".

I see three warnings/errors when running copy-approvals for this project:

ERROR com.google.gerrit.server.approval.RecursiveApprovalCopier : Error in a slice of project openstack/neutron, will retry and skip corrupt meta-refs [CONTEXT request="SSH" ] com.google.gerrit.server.notedb.LimitExceededException: Change 84223 may not exceed 1000 updates. (complete log [1])

WARN  com.google.gerrit.server.change.ChangeKindCacheImpl : Cannot check change kind of new patch set f24a40fa5e727f176b63233c6b75476213ffc506 in openstack/neutron [CONTEXT request="SSH" ]
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown f24a40fa5e727f176b63233c6b75476213ffc506 (complete log [2])

WARN  com.google.gerrit.server.change.ChangeKindCacheImpl : Cannot check change kind of new patch set c095827ea8f82a286c84da1ad3d1aab38a2e1328 in openstack/neutron [CONTEXT request="SSH" ]
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown f24a40fa5e727f176b63233c6b75476213ffc506 (complete log [3])

Then, every time I run copy-approvals against this project, I get the same set of 24 changes that have been "updated" [4] (any of these can be checked on the instance at https://review.opendev.org, I'm not really sure what to look for to see if they really are updated).

I guess there are three questions/problems:

1) Change 84223 has more than a thousand updates, which seems to make the process stop.  Is there some sort of workaround for this?

2) Both the warnings refer to different patchsets, but reference the same missing object f24a40fa5e727f176b63233c6b75476213ffc506.  I really have no idea how this is missing; it's the first occurrence I can find in our logs.  Any suggestions on what to do about this?

3) If I re-run copy-approvals on the project, I get the same set of updated changes reported; but I suspect they're not really updated given the errors?  It feels suspiciously like the same "slice" of changes isn't being processed each time?  Should I worry about this?

Thanks for any help!

-i

[1] https://paste.opendev.org/show/b1g6qz7pWz3IkH1Sd7Ad/

[2] https://paste.opendev.org/show/bQNP3ZIbik6S3T82JIZA/

[3] https://paste.opendev.org/show/bcLQ0jNNkPZQ5WEQMXKm/

[4] https://paste.opendev.org/show/bynUtJEe4BYfzGVGM4I7/

[Sven Selberg]

On Wednesday, November 30, 2022 at 4:16:50 AM UTC+1 iwie...@redhat.com wrote:

Hello,

We are running 3.5.4 and preparing to update to 3.6, and as part of that am running copy-approvals on our instance.

Out of caution I did this separately for each project in ls-projects, just so I could separate out logs for each project in case one failed. I then re-ran copy-approvals, expecting it to do nothing on the second run, but found one of our projects constantly has 24 changes that are reported as "updated".

I see three warnings/errors when running copy-approvals for this project:

ERROR com.google.gerrit.server.approval.RecursiveApprovalCopier : Error in a slice of project openstack/neutron, will retry and skip corrupt meta-refs [CONTEXT request="SSH" ] com.google.gerrit.server.notedb.LimitExceededException: Change 84223 may not exceed 1000 updates. (complete log [1]) 

WARN  com.google.gerrit.server.change.ChangeKindCacheImpl : Cannot check change kind of new patch set f24a40fa5e727f176b63233c6b75476213ffc506 in openstack/neutron [CONTEXT request="SSH" ]
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown f24a40fa5e727f176b63233c6b75476213ffc506 (complete log [2])

WARN  com.google.gerrit.server.change.ChangeKindCacheImpl : Cannot check change kind of new patch set c095827ea8f82a286c84da1ad3d1aab38a2e1328 in openstack/neutron [CONTEXT request="SSH" ]
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown f24a40fa5e727f176b63233c6b75476213ffc506 (complete log [3])

Then, every time I run copy-approvals against this project, I get the same set of 24 changes that have been "updated" [4] (any of these can be checked on the instance at https://review.opendev.org, I'm not really sure what to look for to see if they really are updated).

I guess there are three questions/problems:

1) Change 84223 has more than a thousand updates, which seems to make the process stop.  Is there some sort of workaround for this?

You can configure the maximum number of patch-sets and updates, default is 1000.

https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#change.maxPatchSets

 

2) Both the warnings refer to different patchsets, but reference the same missing object f24a40fa5e727f176b63233c6b75476213ffc506.  I really have no idea how this is missing; it's the first occurrence I can find in our logs.  Any suggestions on what to do about this?

We have found that we have a small set of old patch-sets that are missing objects.
You can restore the objects in some cases through some git-surgery but it's generally not worth the effort if it's not a change that anyone cares about (i.e. if no-one is complaining that their change is broken).

[Clark Boylan]

On Wed, Nov 30, 2022 at 12:03 AM Sven Selberg <sven.s...@axis.com> wrote:
>
>
>
> On Wednesday, November 30, 2022 at 4:16:50 AM UTC+1 iwie...@redhat.com wrote:
>>
>> Hello,
>>
>> We are running 3.5.4 and preparing to update to 3.6, and as part of that am running copy-approvals on our instance.
>>
>> Out of caution I did this separately for each project in ls-projects, just so I could separate out logs for each project in case one failed. I then re-ran copy-approvals, expecting it to do nothing on the second run, but found one of our projects constantly has 24 changes that are reported as "updated".
>>
>> I see three warnings/errors when running copy-approvals for this project:
>>
>> ERROR com.google.gerrit.server.approval.RecursiveApprovalCopier : Error in a slice of project openstack/neutron, will retry and skip corrupt meta-refs [CONTEXT request="SSH" ] com.google.gerrit.server.notedb.LimitExceededException: Change 84223 may not exceed 1000 updates. (complete log [1])
>>
>>
>> WARN com.google.gerrit.server.change.ChangeKindCacheImpl : Cannot check change kind of new patch set f24a40fa5e727f176b63233c6b75476213ffc506 in openstack/neutron [CONTEXT request="SSH" ]
>> java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown f24a40fa5e727f176b63233c6b75476213ffc506 (complete log [2])
>>
>>
>> WARN com.google.gerrit.server.change.ChangeKindCacheImpl : Cannot check change kind of new patch set c095827ea8f82a286c84da1ad3d1aab38a2e1328 in openstack/neutron [CONTEXT request="SSH" ]
>> java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown f24a40fa5e727f176b63233c6b75476213ffc506 (complete log [3])
>>
>> Then, every time I run copy-approvals against this project, I get the same set of 24 changes that have been "updated" [4] (any of these can be checked on the instance at https://review.opendev.org, I'm not really sure what to look for to see if they really are updated).
>>
>> I guess there are three questions/problems:
>>
>> 1) Change 84223 has more than a thousand updates, which seems to make the process stop. Is there some sort of workaround for this?
>
>
> You can configure the maximum number of patch-sets and updates, default is 1000.
> https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#change.maxPatchSets

I don't think modern Gerrit installations should need to modify global
settings for new changes in order to properly perform database
migration activities against old (ancient even) changes. Would it be
possible to have migration tooling like copy-approvals process changes
as they exist and not try to enforce limits that should be enforced
for new changes?

>
>>
>>
>> 2) Both the warnings refer to different patchsets, but reference the same missing object f24a40fa5e727f176b63233c6b75476213ffc506. I really have no idea how this is missing; it's the first occurrence I can find in our logs. Any suggestions on what to do about this?
>
>
> We have found that we have a small set of old patch-sets that are missing objects.
> You can restore the objects in some cases through some git-surgery but it's generally not worth the effort if it's not a change that anyone cares about (i.e. if no-one is complaining that their change is broken).
>
>>
>>
>> 3) If I re-run copy-approvals on the project, I get the same set of updated changes reported; but I suspect they're not really updated given the errors? It feels suspiciously like the same "slice" of changes isn't being processed each time? Should I worry about this?

To expand on this, what are the long term implications of not running
copy-approvals properly against all changes? Does the copy-approvals
process modify changes in a way that isn't transparent to 3.6 and
beyond? Do we run the risk of future notedb modifications failing
because the copy-approval step wasn't completed fully? Or do we just
end up with historical changes whose carry over votes due to rebases
are no longer properly reflected? I think we can probably live with
carry over votes no longer being properly reflected in old changes,
but don't want to risk breaking future migrations.

[Luca Milanesio]

On 30 Nov 2022, at 17:12, Clark Boylan <clark....@gmail.com> wrote:

On Wed, Nov 30, 2022 at 12:03 AM Sven Selberg <sven.s...@axis.com> wrote:

On Wednesday, November 30, 2022 at 4:16:50 AM UTC+1 iwie...@redhat.com wrote:

Hello,

We are running 3.5.4 and preparing to update to 3.6, and as part of that am running copy-approvals on our instance.

Out of caution I did this separately for each project in ls-projects, just so I could separate out logs for each project in case one failed. I then re-ran copy-approvals, expecting it to do nothing on the second run, but found one of our projects constantly has 24 changes that are reported as "updated".

I see three warnings/errors when running copy-approvals for this project:

ERROR com.google.gerrit.server.approval.RecursiveApprovalCopier : Error in a slice of project openstack/neutron, will retry and skip corrupt meta-refs [CONTEXT request="SSH" ] com.google.gerrit.server.notedb.LimitExceededException: Change 84223 may not exceed 1000 updates. (complete log [1])


WARN  com.google.gerrit.server.change.ChangeKindCacheImpl : Cannot check change kind of new patch set f24a40fa5e727f176b63233c6b75476213ffc506 in openstack/neutron [CONTEXT request="SSH" ]
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown f24a40fa5e727f176b63233c6b75476213ffc506 (complete log [2])


WARN  com.google.gerrit.server.change.ChangeKindCacheImpl : Cannot check change kind of new patch set c095827ea8f82a286c84da1ad3d1aab38a2e1328 in openstack/neutron [CONTEXT request="SSH" ]
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown f24a40fa5e727f176b63233c6b75476213ffc506 (complete log [3])

Then, every time I run copy-approvals against this project, I get the same set of 24 changes that have been "updated" [4] (any of these can be checked on the instance at https://review.opendev.org, I'm not really sure what to look for to see if they really are updated).

I guess there are three questions/problems:

1) Change 84223 has more than a thousand updates, which seems to make the process stop.  Is there some sort of workaround for this?

You can configure the maximum number of patch-sets and updates, default is 1000.
https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#change.maxPatchSets

I don't think modern Gerrit installations should need to modify global
settings for new changes in order to properly perform database
migration activities against old (ancient even) changes.

Was the change open or closed?

Would it be
possible to have migration tooling like copy-approvals process changes
as they exist and not try to enforce limits that should be enforced
for new changes?

Good point: because this is a migration activity it should not count against the limit IMHO.

2) Both the warnings refer to different patchsets, but reference the same missing object f24a40fa5e727f176b63233c6b75476213ffc506.  I really have no idea how this is missing; it's the first occurrence I can find in our logs.  Any suggestions on what to do about this?

We have found that we have a small set of old patch-sets that are missing objects.
You can restore the objects in some cases through some git-surgery but it's generally not worth the effort if it's not a change that anyone cares about (i.e. if no-one is complaining that their change is broken).

3) If I re-run copy-approvals on the project, I get the same set of updated changes reported; but I suspect they're not really updated given the errors?  It feels suspiciously like the same "slice" of changes isn't being processed each time?  Should I worry about this?

To expand on this, what are the long term implications of not running
copy-approvals properly against all changes?

If you don’t run the copy-approvals, the approval status after the migration could be different from the one before the migration.

That is due to the different way Gerrit v3.6 evaluates the approval status compared to the v3.5 or earlier.

Does the copy-approvals
process modify changes in a way that isn't transparent to 3.6 and
beyond?

It does not modify the change, just calculates the approval status and add one extra comment to the change notes with it.

Do we run the risk of future notedb modifications failing
because the copy-approval step wasn't completed fully?

Nope.

Or do we just
end up with historical changes whose carry over votes due to rebases
are no longer properly reflected? I think we can probably live with
carry over votes no longer being properly reflected in old changes,
but don't want to risk breaking future migrations.

Sure, there are no implications on future upgrades.

HTH

Luca.

Thanks for any help!

-i

[1] https://paste.opendev.org/show/b1g6qz7pWz3IkH1Sd7Ad/
[2] https://paste.opendev.org/show/bQNP3ZIbik6S3T82JIZA/
[3] https://paste.opendev.org/show/bcLQ0jNNkPZQ5WEQMXKm/
[4] https://paste.opendev.org/show/bynUtJEe4BYfzGVGM4I7/

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAAcH74izsZkXG9nu_WW3WKJWO2wv_40itVnjR5G1o4nbnkRu5A%40mail.gmail.com.

[Ian Wienand]

On Thursday, December 1, 2022 at 6:33:23 AM UTC+11 lucamilanesio wrote:

1) Change 84223 has more than a thousand updates, which seems to make the process stop.  Is there some sort of workaround for this?

You can configure the maximum number of patch-sets and updates, default is 1000.
https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#change.maxPatchSets

I don't think modern Gerrit installations should need to modify global
settings for new changes in order to properly perform database
migration activities against old (ancient even) changes.

Was the change open or closed?

This is https://review.opendev.org/c/openstack/neutron/+/84223 which is closed

Would it be
possible to have migration tooling like copy-approvals process changes
as they exist and not try to enforce limits that should be enforced
for new changes?

Good point: because this is a migration activity it should not count against the limit IMHO.

Since we have some agreement on that, I can file a issue specifically about this for further consideration.

I'm wondering though if this would cause the repeated application of copy-approvals to keep reporting that the same 24 changesets have been updated?  How would I tell if those changes have *really* been updated (I can't see anything in the UI, right?)

-i

 

[Luca Milanesio]

On 30 Nov 2022, at 19:53, Ian Wienand <iwie...@redhat.com> wrote:

On Thursday, December 1, 2022 at 6:33:23 AM UTC+11 lucamilanesio wrote:

1) Change 84223 has more than a thousand updates, which seems to make the process stop.  Is there some sort of workaround for this?

You can configure the maximum number of patch-sets and updates, default is 1000.
https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#change.maxPatchSets

I don't think modern Gerrit installations should need to modify global
settings for new changes in order to properly perform database
migration activities against old (ancient even) changes.

Was the change open or closed?

This is https://review.opendev.org/c/openstack/neutron/+/84223 which is closed

I see, so we possibly need an option to run copy-approvals on open changes only.

Would it be
possible to have migration tooling like copy-approvals process changes
as they exist and not try to enforce limits that should be enforced
for new changes?

Good point: because this is a migration activity it should not count against the limit IMHO.

Since we have some agreement on that, I can file a issue specifically about this for further consideration.

+1

I'm wondering though if this would cause the repeated application of copy-approvals to keep reporting that the same 24 changesets have been updated?  How would I tell if those changes have *really* been updated (I can't see anything in the UI, right?)

There is no UI difference in v3.5. In v3.6, you would notice only if comparing the approval status with the v3.5.

Luca.

-i

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/1e846a1b-6528-4dbf-85e7-180d694cb47an%40googlegroups.com.