How to cleanup gerrit database from corrupt entries?

[Matthias Keller]

Hi

We too had the problem, that we could not update to 3.6 because of some corrupted entries where the migration always failed until this was fixed in 3.5.4.

However I'm wondering if there's any chance I can get rid of the corrupted entries?

During reindex, i get a lot of entries like

AllChangesIndexer : Failed to read change 47103 for indexing

java.io.IOException: Error loading refs/changes/03/47103/meta in my.project at 35ca5339f50fdd34ee07b0a16ba53900ea646fc3

org.eclipse.jgit.errors.ConfigInvalidException: Change 47103: missing or multiple Patch-set: []

or

com.google.gerrit.index.Schema : error getting field added of ChangeData{Change{27669 (I998a9e12af3944c6eeb3c812671372837c4ca854), dest=my.project,refs/heads/master, status=A}}
com.google.gerrit.exceptions.StorageException: Loading commit AnyObjectId[36f5b04d1d3e9794593dc5de64e50e3f85d6e088] for ps 1 of change 27669 failed.

MissingObjectException: Missing unknown 36f5b04d1d3e9794593dc5de64e50e3f85d6e088

Since the corruption happened in changes a while ago, I'd prefer to just get rid of the whole change in gerrit just as long as the git history is untouched.

Is there any way to do this? I've already tried calling the REST endpoint /a/changes/47103/abandon

for which I just get the message "Not found: 47103"

I guess I'd have to dig in the DB to clean those up - any hints on how to do this?

I've found the https://gerrit.googlesource.com/gerrit/+/refs/heads/master/contrib/remove-notedb-refs.sh script - could I just do a

git update-ref -d /refs/changes/69/27669/meta    ?

What exactly would this remove? The whole review information but not the commits in the concerned branch?

Thanks Matt

[Matthias Sohn]

Don't use this script for this purpose, it deletes all noteDB data,

you don't want that to get rid of some corrupted entries.

See the script's description:

This util script can be used in case of rollback to ReviewDB during an unsuccessful
migration to NoteDB or simply while testing the migration process.
It will remove all the refs used by NoteDB added during the migration (i.e.: change meta refs and sequence ref)

Check which refs exist for the affected changes. Do that by grepping the output of git show-ref
run on the respective git repository. E.g. for change 121: 

$ git show-ref | grep refs/changes/21/121
cd623d447ed125d004369e96e96141da189342d9 refs/changes/21/121/1
6c1de686f91e5b26b22d999339a6b9c23999524e refs/changes/21/121/2
f320b3b6a8450acae43732dab59c944c4b4367d7 refs/changes/21/121/meta

A Gerrit change consists of 1..n patchsets, here patchset 1 (refs/changes/21/121/1) and 2 (refs/changes/21/121/2),

and the corresponding meta ref (refs/changes/21/121/meta).

If the meta ref of a change went missing it's impossible to fully reconstruct if you don't find it in a backup or replica.

In that case first check if the commits the patchsets refer to are reachable from any branch or tag:

e.g. for the patchset refs/changes/21/121/1:

$ git branch --contains cd623d447ed125d004369e96e96141da189342d9

$ git tag --contains cd623d447ed125d004369e96e96141da189342d9

If the result of both these commands is empty this commit will become unreachable

if you delete the patchset's ref (here refs/changes/21/121/1). This implies that

git gc will delete it when the prune interval has passed.

In that case look at the commits diff to decide if you want to keep it or not:

$ git show 6c1de686f91e5b26b22d999339a6b9c23999524e
commit 6c1de686f91e5b26b22d999339a6b9c23999524e
Author: Matthias Sohn <matthi...@sap.com>
Date:   Wed Nov 30 16:13:31 2022 +0100

    a7

    Change-Id: I8f1483bbe16abd7181e7eb732eb47b9708d00cc7

diff --git a/a.txt b/a.txt
index 5b521d0..f1914bb 100644
--- a/a.txt
+++ b/a.txt
@@ -1 +1 @@
-a6
+a8

If the commit is reachable from a branch or tag I think you can delete the patchset refs
using update-ref -d (see https://git-scm.com/docs/git-update-ref).

For the other error:

com.google.gerrit.index.Schema : error getting field added of ChangeData{Change{27669 (I998a9e12af3944c6eeb3c812671372837c4ca854), dest=my.project,refs/heads/master, status=A}}
com.google.gerrit.exceptions.StorageException: Loading commit AnyObjectId[36f5b04d1d3e9794593dc5de64e50e3f85d6e088] for ps 1 of change 27669 failed.

MissingObjectException: Missing unknown 36f5b04d1d3e9794593dc5de64e50e3f85d6e088

check if this commit is present in this repository

$ git show 36f5b04d1d3e9794593dc5de64e50e3f85d6e088

if this commit isn't present and you don't find it in a backup or replica I'd delete the corresponding patchset ref

here refs/changes/6927669/1.

Try any manual surgery first on a copy and take a backup before you do it on productive data. 

-Matthias