git gc error:"bad refs/chages/xxx"

Yingchun Li

unread,

Jan 7, 2021, 2:18:49 AM1/7/21

to Repo and Gerrit Discussion

Hi, All

My Gerrit version is 3.3.0, when I run `git gc` in ther server side, it

failed with following message:

git gc

fatal: bad object refs/changes/54/754/meta

fatal: failed to run repack

the following messages were reported by

git fsck --full

Checking object directories: 100% (256/256), done.

Checking objects: 100% (96104/96104), done.

error: refs/changes/54/754/meta: invalid sha1 pointer ab6cd47eaf4cd2fee0b565ce55380a0ce0749345

error: refs/changes/62/762/2: invalid sha1 pointer 1ca0211a41f0d205aae4faf012e5a7e4df6701dc

dangling commit cd0640e7c50f06d1489b1d24ef842e0e953c94c3

dangling commit 7c07280af730b6abc4aaffa93112352952dc7e65

dangling commit e40e98ab5fd31bbdff27e979dab3f8f0f1d74d69

dangling commit f417e0c55dafd66ce6b86b5c648dc1150593e1af

dangling commit 46197862a20ce93fdae5742b21e01398af7a429f

So what I should do to fix this issue?

Br,
Yingchun

Luca Milanesio

unread,

Jan 7, 2021, 2:50:40 AM1/7/21

to Yingchun Li, Luca Milanesio, Repo and Gerrit Discussion

On 7 Jan 2021, at 07:18, Yingchun Li <sword.l...@gmail.com> wrote:

Hi, All
My Gerrit version is 3.3.0, when I run `git gc` in ther server side, it
failed with following message:

git gc
fatal: bad object refs/changes/54/754/meta
fatal: failed to run repack

Are you running ‘git gc’ while Gerrit is serving read/write traffic?

If yes, you may have corrupted the repository in that way.

the following messages were reported by

git fsck --full

Checking object directories: 100% (256/256), done.
Checking objects: 100% (96104/96104), done.
error: refs/changes/54/754/meta: invalid sha1 pointer ab6cd47eaf4cd2fee0b565ce55380a0ce0749345
error: refs/changes/62/762/2: invalid sha1 pointer 1ca0211a41f0d205aae4faf012e5a7e4df6701dc

It looks like your change #762 and #754 are corrupted.

Do you have any backups? Any replicas to recover the git refs?

If not, you would have to remove the refs and run an offline reindex.

HTH

Luca.

dangling commit cd0640e7c50f06d1489b1d24ef842e0e953c94c3
dangling commit 7c07280af730b6abc4aaffa93112352952dc7e65
dangling commit e40e98ab5fd31bbdff27e979dab3f8f0f1d74d69
dangling commit f417e0c55dafd66ce6b86b5c648dc1150593e1af
dangling commit 46197862a20ce93fdae5742b21e01398af7a429f

So what I should do to fix this issue?
Br,
Yingchun

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/473f1b5b-f190-4aa3-bbab-5358cfed939fn%40googlegroups.com.

Yingchun Li

unread,

Jan 7, 2021, 3:15:21 AM1/7/21

to Repo and Gerrit Discussion

Thank you Luca!

On 7 Jan 2021, at 07:18, Yingchun Li <sword.l...@gmail.com> wrote:

Hi, All
My Gerrit version is 3.3.0, when I run `git gc` in ther server side, it
failed with following message:

git gc
fatal: bad object refs/changes/54/754/meta
fatal: failed to run repack

Are you running ‘git gc’ while Gerrit is serving read/write traffic?
If yes, you may have corrupted the repository in that way.

That's maybe the cause, my server was still running when I did `git gc`,

So that means when do `git gc`, I would always stop gerrit? Is the command

`ssp -p 29418 myserver gerrit gc --all1` more safe?

the following messages were reported by

git fsck --full

Checking object directories: 100% (256/256), done.
Checking objects: 100% (96104/96104), done.
error: refs/changes/54/754/meta: invalid sha1 pointer ab6cd47eaf4cd2fee0b565ce55380a0ce0749345
error: refs/changes/62/762/2: invalid sha1 pointer 1ca0211a41f0d205aae4faf012e5a7e4df6701dc

It looks like your change #762 and #754 are corrupted.

It is strange, checked the #762 and #754 changes, and I can still read the

content, the diff..., was I wrong?(I just find #754 and #762 in the gerrit search box)

Do you have any backups? Any replicas to recover the git refs?
If not, you would have to remove the refs and run an offline reindex.

HTH

Luca.

Ok, How to remove the refs, just `rm refs/changes/54/754/`, and refs/changes/62/762?

luca.mi...@gmail.com

unread,

Jan 7, 2021, 3:29:10 AM1/7/21

to Yingchun Li, Repo and Gerrit Discussion

Sent from my iPhone

On 7 Jan 2021, at 08:15, Yingchun Li <sword.l...@gmail.com> wrote:

Thank you Luca!
On 7 Jan 2021, at 07:18, Yingchun Li <sword.l...@gmail.com> wrote:

Hi, All
My Gerrit version is 3.3.0, when I run `git gc` in ther server side, it
failed with following message:

git gc
fatal: bad object refs/changes/54/754/meta
fatal: failed to run repack

Are you running ‘git gc’ while Gerrit is serving read/write traffic?
If yes, you may have corrupted the repository in that way.

That's maybe the cause, my server was still running when I did `git gc`,
So that means when do `git gc`, I would always stop gerrit?

Yes, or at least read-only

Is the command
`ssp -p 29418 myserver gerrit gc --all1` more safe?

Yep.

HTH

Luca

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/8aa6147e-d6c5-4420-9b98-93f872e61a22n%40googlegroups.com.

Sven Selberg

unread,

Jan 7, 2021, 4:17:35 AM1/7/21

to Repo and Gerrit Discussion

On Thursday, January 7, 2021 at 9:29:10 AM UTC+1 lucamilanesio wrote:

Sent from my iPhone

On 7 Jan 2021, at 08:15, Yingchun Li <sword.l...@gmail.com> wrote:

Thank you Luca!
On 7 Jan 2021, at 07:18, Yingchun Li <sword.l...@gmail.com> wrote:

Hi, All
My Gerrit version is 3.3.0, when I run `git gc` in ther server side, it
failed with following message:

git gc
fatal: bad object refs/changes/54/754/meta
fatal: failed to run repack

Are you running ‘git gc’ while Gerrit is serving read/write traffic?
If yes, you may have corrupted the repository in that way.

That's maybe the cause, my server was still running when I did `git gc`,
So that means when do `git gc`, I would always stop gerrit?

What happens is that:

1. Gerrit writes objects to disk

2. gc removes the parent-less objects

3. Gerrit writes the ref that points to objects to disk

You can fix this by only disallowing the gc to remove objects that have just been created.

We have a quite cautious limit, I'm sure you should be able to get away with seconds:

git prune --expire=1.hours.ago

Yes, or at least read-only

Is the command
`ssp -p 29418 myserver gerrit gc --all1` more safe?

Yep.

HTH
Luca

the following messages were reported by

git fsck --full

Checking object directories: 100% (256/256), done.
Checking objects: 100% (96104/96104), done.
error: refs/changes/54/754/meta: invalid sha1 pointer ab6cd47eaf4cd2fee0b565ce55380a0ce0749345
error: refs/changes/62/762/2: invalid sha1 pointer 1ca0211a41f0d205aae4faf012e5a7e4df6701dc

It looks like your change #762 and #754 are corrupted.

It is strange, checked the #762 and #754 changes, and I can still read the
content, the diff..., was I wrong?(I just find #754 and #762 in the gerrit search box)

Do you have any backups? Any replicas to recover the git refs?
If not, you would have to remove the refs and run an offline reindex.

HTH
Luca.

Ok, How to remove the refs, just `rm refs/changes/54/754/`, and refs/changes/62/762?

git update-ref -d refs/changes/62/762

Matthias Sohn

unread,

Jan 7, 2021, 5:49:48 AM1/7/21

to Sven Selberg, Repo and Gerrit Discussion

On Thu, Jan 7, 2021 at 10:17 AM Sven Selberg <sven.s...@axis.com> wrote:

On Thursday, January 7, 2021 at 9:29:10 AM UTC+1 lucamilanesio wrote:

Sent from my iPhone

On 7 Jan 2021, at 08:15, Yingchun Li <sword.l...@gmail.com> wrote:

Thank you Luca!
On 7 Jan 2021, at 07:18, Yingchun Li <sword.l...@gmail.com> wrote:

Hi, All
My Gerrit version is 3.3.0, when I run `git gc` in ther server side, it
failed with following message:

git gc
fatal: bad object refs/changes/54/754/meta
fatal: failed to run repack

Are you running ‘git gc’ while Gerrit is serving read/write traffic?
If yes, you may have corrupted the repository in that way.

That's maybe the cause, my server was still running when I did `git gc`,
So that means when do `git gc`, I would always stop gerrit?

What happens is that:
1. Gerrit writes objects to disk
2. gc removes the parent-less objects
3. Gerrit writes the ref that points to objects to disk

You can fix this by only disallowing the gc to remove objects that have just been created.
We have a quite cautious limit, I'm sure you should be able to get away with seconds:

git prune --expire=1.hours.ago

the default is 2 weeks [1]

We configure all repositories to use this configuration

# enable reflog
core.logallrefupdates = true

# switch off auto gc
gc.auto = 0
gc.autodetach = false
gc.autopacklimit = 0
receive.autogc = false

# do not remove old reflogs, we need them for auditing e.g. force pushes
gc.reflogexpire = never
gc.reflogexpireunreachable = never

# pack configuration
gc.packrefs = true
pack.compression = 9
pack.depth = 50
pack.indexversion = 2
pack.window = 250
repack.usedeltabaseoffset = true

# create bitmap index for performance
repack.writebitmaps = true

with this configuration we never saw corruption caused by running git gc within the 10 years we use Gerrit

and we do not shutdown or make read-only Gerrit while git gc is running

[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-gcpruneExpire

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/45b15ebf-07b2-43bb-9c9d-9a4475877de8n%40googlegroups.com.

Luca Milanesio

unread,

Jan 7, 2021, 9:41:36 AM1/7/21

to Repo and Gerrit Discussion, Luca Milanesio, Sven Selberg, Matthias Sohn

If the default is 2 weeks, how could have caused the removal of parent-less objects just created?

We configure all repositories to use this configuration

# enable reflog
core.logallrefupdates = true

# switch off auto gc
gc.auto = 0
gc.autodetach = false
gc.autopacklimit = 0
receive.autogc = false

# do not remove old reflogs, we need them for auditing e.g. force pushes
gc.reflogexpire = never
gc.reflogexpireunreachable = never

Don’t you have problems in not being able to get rid of unwanted commits in this way?

Example:

Someone pushes a 1GB BLOB in a commit, realises the problem, does a force push making the first commit unreachable.

If you have reflogexpireunreachable = never, you’ll never get rid of the 1GB BLOB, isn’t it?

# pack configuration
gc.packrefs = true
pack.compression = 9
pack.depth = 50
pack.indexversion = 2
pack.window = 250
repack.usedeltabaseoffset = true

# create bitmap index for performance
repack.writebitmaps = true

with this configuration we never saw corruption caused by running git gc within the 10 years we use Gerrit
and we do not shutdown or make read-only Gerrit while git gc is running

We unfortunately did see and a few times, that’s the reason behind my suggestion.

We’ve fixed lot of problems, but I am unsure if we got all of them sorted :-(

P.S. You also worked on one of them, the racy-JGit problem, and the last was discovered during the test of Gerrit v3.3.

I always prefer to have a cautious approach, rather than having to deal with repositories corruptions :-(

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3Q%3DVxGTWcZWgxHOeMv5MiziCEiEnACK%3DKRZb%3DVmq%2BFteg%40mail.gmail.com.

Matthias Sohn

unread,

Jan 7, 2021, 10:23:53 AM1/7/21

to Luca Milanesio, Repo and Gerrit Discussion, Sven Selberg

we disallow pushing objects larger than 20MB via receive.maxObjectSizeLimit [2]

[2] https://gerrit-documentation.storage.googleapis.com/Documentation/3.3.1/config-gerrit.html#receive.maxObjectSizeLimit

If you have reflogexpireunreachable = never, you’ll never get rid of the 1GB BLOB, isn’t it?

# pack configuration
gc.packrefs = true
pack.compression = 9
pack.depth = 50
pack.indexversion = 2
pack.window = 250
repack.usedeltabaseoffset = true

# create bitmap index for performance
repack.writebitmaps = true

with this configuration we never saw corruption caused by running git gc within the 10 years we use Gerrit
and we do not shutdown or make read-only Gerrit while git gc is running

We unfortunately did see and a few times, that’s the reason behind my suggestion.
We’ve fixed lot of problems, but I am unsure if we got all of them sorted :-(

P.S. You also worked on one of them, the racy-JGit problem, and the last was discovered during the test of Gerrit v3.3.

we don't use JGit to run gc but run git gc from a cronjob

Yingchun Li

unread,

Jan 7, 2021, 11:21:34 PM1/7/21

to Repo and Gerrit Discussion

Thank you Matthias, where you run `git gc`? if you run on server side, is there still race condition?

Reply all

Reply to author

Forward