Post upgrade to 3.6.8 : java.lang.IndexOutOfBoundsException

tech....@gmail.com

unread,

Apr 1, 2024, 2:47:01 AMApr 1

to Repo and Gerrit Discussion

Hello Experts,

We are observing issues after upgrading from Gerrit version 3.6.5 to 3.6.8 .

Normal git clone failed with below error and attaching error log

Cloning into 'manifest'...
remote: internal server error
fatal: early EOF
fatal: internal server error
fatal: fetch-pack: invalid index-pack output

Error log:

ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user sperwez account 1006354) during git-upload-pack '/android/manifest'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:804)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:101)
at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:109)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:74)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:492)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: java.lang.IndexOutOfBoundsException: 0
at org.eclipse.jgit.internal.storage.file.BitmapIndexImpl$MutableBitmapIndex.getObject(BitmapIndexImpl.java:427)
at org.eclipse.jgit.internal.storage.file.BitmapIndexImpl$CompressedBitmap$1.next(BitmapIndexImpl.java:366)
at org.eclipse.jgit.internal.storage.file.BitmapIndexImpl$CompressedBitmap$1.next(BitmapIndexImpl.java:328)
at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPackUsingBitmaps(PackWriter.java:2206)
at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1986)
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:1026)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2416)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2304)
at org.eclipse.jgit.transport.UploadPack.fetchV2(UploadPack.java:1296)
at org.eclipse.jgit.transport.UploadPack.serveOneCommandV2(UploadPack.java:1359)
at org.eclipse.jgit.transport.UploadPack.serviceV2(UploadPack.java:1413)
at org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:871)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:781)
... 12 more
Caused by: java.lang.IndexOutOfBoundsException: 0
at org.eclipse.jgit.util.BlockList.get(BlockList.java:101)
at org.eclipse.jgit.internal.storage.file.BitmapIndexImpl$MutableBitmapIndex.getObject(BitmapIndexImpl.java:420)
... 24 more

Please suggest work around or a fix

Aaron Smith

unread,

Apr 1, 2024, 3:14:54 PMApr 1

to Repo and Gerrit Discussion

Have you tried running 'git gc' and/or 'git fsck' on the Gerrit server in the affected repository?

Aside from that, you could upgrade to a later version -- 3.6 is end-of-life. 3.7 will be end-of-life when 3.10 is released in May.

tech....@gmail.com

unread,

Apr 17, 2024, 11:39:01 AMApr 17

to Repo and Gerrit Discussion

This looks temporary solution.

After some days issues are coming back for those reposities.

Daniele Sassoli

unread,

Apr 17, 2024, 1:18:56 PMApr 17

to Repo and Gerrit Discussion

On Wednesday 17 April 2024 at 16:39:01 UTC+1 tech....@gmail.com wrote:

This looks temporary solution.

After some days issues are coming back for those reposities.

This makes it sound like the problem might be with the repositories themselves, looks like they are becoming corrupted for some reason and a failed Clone is the result, rather than the cause.

Git GC should be run regularly on the repo, depending on how busy the repo is, so I wouldn't call it a temporary solution.

Aaron Smith

unread,

Apr 17, 2024, 1:55:30 PMApr 17

to Repo and Gerrit Discussion

On Wednesday, April 17, 2024 at 10:18:56 AM UTC-7 Daniele Sassoli wrote:

On Wednesday 17 April 2024 at 16:39:01 UTC+1 tech....@gmail.com wrote:
This looks temporary solution.

After some days issues are coming back for those reposities.
This makes it sound like the problem might be with the repositories themselves, looks like they are becoming corrupted for some reason and a failed Clone is the result, rather than the cause.

Git GC should be run regularly on the repo, depending on how busy the repo is, so I wouldn't call it a temporary solution.

One of my repositories can't be deep-cloned after a scheduled jgit gc operation -- it requires manual `git fsck` and `git gc` to recover it. I presume something in jgit is causing the issue, but I didn't report it here. We disabled scheduled gc, instead.

Matthias Sohn

unread,

Apr 17, 2024, 4:10:13 PMApr 17

to Aaron Smith, Repo and Gerrit Discussion

On Wed, Apr 17, 2024 at 7:55 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

On Wednesday, April 17, 2024 at 10:18:56 AM UTC-7 Daniele Sassoli wrote:
On Wednesday 17 April 2024 at 16:39:01 UTC+1 tech....@gmail.com wrote:
This looks temporary solution.

After some days issues are coming back for those reposities.
This makes it sound like the problem might be with the repositories themselves, looks like they are becoming corrupted for some reason and a failed Clone is the result, rather than the cause.

Git GC should be run regularly on the repo, depending on how busy the repo is, so I wouldn't call it a temporary solution.

One of my repositories can't be deep-cloned after a scheduled jgit gc operation -- it requires manual `git fsck` and `git gc` to recover it. I presume something in jgit is causing the issue, but I didn't report it here. We disabled scheduled gc, instead.

Did git fsck show any errors or corruptions ?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d341dc3d-de2c-416d-8cee-f45639bf7503n%40googlegroups.com.

Aaron Smith

unread,

Apr 17, 2024, 7:02:00 PMApr 17

to Repo and Gerrit Discussion

On Wednesday, April 17, 2024 at 1:10:13 PM UTC-7 Matthias Sohn wrote:

On Wed, Apr 17, 2024 at 7:55 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
On Wednesday, April 17, 2024 at 10:18:56 AM UTC-7 Daniele Sassoli wrote:
On Wednesday 17 April 2024 at 16:39:01 UTC+1 tech....@gmail.com wrote:
This looks temporary solution.

After some days issues are coming back for those reposities.
This makes it sound like the problem might be with the repositories themselves, looks like they are becoming corrupted for some reason and a failed Clone is the result, rather than the cause.

Git GC should be run regularly on the repo, depending on how busy the repo is, so I wouldn't call it a temporary solution.

One of my repositories can't be deep-cloned after a scheduled jgit gc operation -- it requires manual `git fsck` and `git gc` to recover it. I presume something in jgit is causing the issue, but I didn't report it here. We disabled scheduled gc, instead.

Did git fsck show any errors or corruptions ?

I ran into this while running Gerit 3.9.1. It started after I enabled scheduled gc on a weekly basis. Attempts to clone gave this message:

Cloning into 'XXXX-XXXXXX'...
fatal: internal server error

remote: internal server error
fatal: early EOF

fatal: fetch-pack: invalid index-pack output

If I did a shallow clone it worked, then we could `git fetch --unshallow origin` -- but this wasn't good for our workflow.

Here's what git fsck said:

Checking object directories: 100% (256/256), done.
Checking objects: 100% (206765/206765), done.
dangling tree 865ef05485f307e178e11ad2e11d6f3dab83f9bb
dangling blob e94cb6471e690215a1c3d86b25b3582584ad3e7c
dangling commit d2b71e1ba021bac5bc417f9eeb10a78b34cdd4c2

That dangling commit was the HEAD of our main development branch. After running `fit fsck`, I ran `git gc` (from the shell, not from Gerrit UI):

Counting objects: 205576, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (80098/80098), done.
Writing objects: 100% (205576/205576), done.
Total 205576 (delta 126873), reused 202217 (delta 123572)

... and this fixed full cloning. I "fixed" the error several times before disabling the scheduled gc. We haven't seen the issue since.

The problem is reproducible on my server if I enable scheduled gc. I haven't tried it since upgrading to Gerrit 3.9.2, but I could set up a sandbox environment in which to try it.

tech....@gmail.com

unread,

Apr 17, 2024, 11:30:52 PMApr 17

to Repo and Gerrit Discussion

Should we disable scheduled gc. Is that causing the issue?

If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this

Aaron Smith

unread,

Apr 18, 2024, 12:21:46 AMApr 18

to Repo and Gerrit Discussion

On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:

Should we disable scheduled gc. Is that causing the issue?
If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this

I don't know about your case, I only know that it was a problem in my installation. If the issue returns shortly after a scheduled gc has completed then it may be the same issue I saw.

Sven Selberg

unread,

Apr 18, 2024, 2:45:40 AMApr 18

to Repo and Gerrit Discussion

On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:

On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
Should we disable scheduled gc. Is that causing the issue?
If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this

It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.

Matthias Sohn

unread,

Apr 18, 2024, 4:12:48 AMApr 18

to Aaron Smith, Repo and Gerrit Discussion

On Thu, Apr 18, 2024 at 1:02 AM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

On Wednesday, April 17, 2024 at 1:10:13 PM UTC-7 Matthias Sohn wrote:
On Wed, Apr 17, 2024 at 7:55 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
On Wednesday, April 17, 2024 at 10:18:56 AM UTC-7 Daniele Sassoli wrote:
On Wednesday 17 April 2024 at 16:39:01 UTC+1 tech....@gmail.com wrote:
This looks temporary solution.

After some days issues are coming back for those reposities.
This makes it sound like the problem might be with the repositories themselves, looks like they are becoming corrupted for some reason and a failed Clone is the result, rather than the cause.

Git GC should be run regularly on the repo, depending on how busy the repo is, so I wouldn't call it a temporary solution.

One of my repositories can't be deep-cloned after a scheduled jgit gc operation -- it requires manual `git fsck` and `git gc` to recover it. I presume something in jgit is causing the issue, but I didn't report it here. We disabled scheduled gc, instead.

Did git fsck show any errors or corruptions ?

I ran into this while running Gerit 3.9.1. It started after I enabled scheduled gc on a weekly basis. Attempts to clone gave this message:

Do you mean gc scheduled in Gerrit using JGit or git gc scheduled by e.g. a cronjob ?

Cloning into 'XXXX-XXXXXX'...
fatal: internal server error
remote: internal server error
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

If I did a shallow clone it worked, then we could `git fetch --unshallow origin` -- but this wasn't good for our workflow.

Here's what git fsck said:
Checking object directories: 100% (256/256), done.
Checking objects: 100% (206765/206765), done.
dangling tree 865ef05485f307e178e11ad2e11d6f3dab83f9bb
dangling blob e94cb6471e690215a1c3d86b25b3582584ad3e7c
dangling commit d2b71e1ba021bac5bc417f9eeb10a78b34cdd4c2

That dangling commit was the HEAD of our main development branch. After running `fit fsck`, I ran `git gc` (from the shell, not from Gerrit UI):
Counting objects: 205576, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (80098/80098), done.
Writing objects: 100% (205576/205576), done.
Total 205576 (delta 126873), reused 202217 (delta 123572)

Does that mean you lost the ref of the main development branch ?

Are you using NFS for storing git repositories ?

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a53449bc-5159-4614-9e10-ca9b891d9d73n%40googlegroups.com.

Luca Milanesio

unread,

Apr 18, 2024, 5:22:17 AMApr 18

to Repo and Gerrit Discussion, Luca Milanesio, Sven Selberg

On 18 Apr 2024, at 07:45, Sven Selberg <sven.s...@axis.com> wrote:

On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:
On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
Should we disable scheduled gc. Is that causing the issue?
If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this

It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.

There was an issue with JGit GC which was fixed months ago: it was an index out of bound on the Bitmap, which matches what it was reported.

Gertit v3.6.8 is EOL, so I doubt it contains the fixes. Can you reproduce it with the latest released version of Gerrit? (e.g. v3.9.4)

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/c97ae8ba-5091-42d8-b646-253559c2eb3bn%40googlegroups.com.

tech....@gmail.com

unread,

Apr 18, 2024, 11:01:33 AMApr 18

to Repo and Gerrit Discussion

On Thursday 18 April 2024 at 14:52:17 UTC+5:30 Luca Milanesio wrote:

On 18 Apr 2024, at 07:45, Sven Selberg <sven.s...@axis.com> wrote:

On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:
On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
Should we disable scheduled gc. Is that causing the issue?
If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this

It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.

There was an issue with JGit GC which was fixed months ago: it was an index out of bound on the Bitmap, which matches what it was reported.
Gertit v3.6.8 is EOL, so I doubt it contains the fixes. Can you reproduce it with the latest released version of Gerrit? (e.g. v3.9.4)

Luca.

We have a replica sandbox instance , which is also having same issue. we can try upgrading from 3.6.8 to 3.9.4 step by step.

So as per your input it should remediate the current situation on Gerrit 3.9.4 ?

Matthias Sohn

unread,

Apr 18, 2024, 11:25:33 AMApr 18

to Luca Milanesio, Repo and Gerrit Discussion, Sven Selberg

On Thu, Apr 18, 2024 at 11:22 AM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 18 Apr 2024, at 07:45, Sven Selberg <sven.s...@axis.com> wrote:

On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:
On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
Should we disable scheduled gc. Is that causing the issue?
If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this

It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.

There was an issue with JGit GC which was fixed months ago: it was an index out of bound on the Bitmap, which matches what it was reported.

Which JGit change fixing this are you referring to ?

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/04E1A2EF-CFB7-49FC-A93F-2B1BA1C07047%40gmail.com.

Luca Milanesio

unread,

Apr 18, 2024, 12:16:54 PMApr 18

to Repo and Gerrit Discussion, Luca Milanesio, Sven Selberg, Matthias Sohn

On 18 Apr 2024, at 16:25, Matthias Sohn <matthi...@gmail.com> wrote:

On Thu, Apr 18, 2024 at 11:22 AM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 18 Apr 2024, at 07:45, Sven Selberg <sven.s...@axis.com> wrote:

On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:
On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
Should we disable scheduled gc. Is that causing the issue?
If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this

It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.

There was an issue with JGit GC which was fixed months ago: it was an index out of bound on the Bitmap, which matches what it was reported.

Which JGit change fixing this are you referring to ?

The repacking with concurrency with incoming traffic:

https://eclipse.gerrithub.io/c/eclipse-jgit/jgit/+/203625

The issue happens when the GC happens at the same time of incoming receive-packs.

Luca.

Aaron Smith

unread,

Apr 18, 2024, 1:06:31 PMApr 18

to Repo and Gerrit Discussion

On Thursday, April 18, 2024 at 1:12:48 AM UTC-7 Matthias Sohn wrote:

On Thu, Apr 18, 2024 at 1:02 AM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
On Wednesday, April 17, 2024 at 1:10:13 PM UTC-7 Matthias Sohn wrote:
On Wed, Apr 17, 2024 at 7:55 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
On Wednesday, April 17, 2024 at 10:18:56 AM UTC-7 Daniele Sassoli wrote:
On Wednesday 17 April 2024 at 16:39:01 UTC+1 tech....@gmail.com wrote:
This looks temporary solution.

After some days issues are coming back for those reposities.
This makes it sound like the problem might be with the repositories themselves, looks like they are becoming corrupted for some reason and a failed Clone is the result, rather than the cause.

Git GC should be run regularly on the repo, depending on how busy the repo is, so I wouldn't call it a temporary solution.

One of my repositories can't be deep-cloned after a scheduled jgit gc operation -- it requires manual `git fsck` and `git gc` to recover it. I presume something in jgit is causing the issue, but I didn't report it here. We disabled scheduled gc, instead.

Did git fsck show any errors or corruptions ?

I ran into this while running Gerit 3.9.1. It started after I enabled scheduled gc on a weekly basis. Attempts to clone gave this message:

Do you mean gc scheduled in Gerrit using JGit or git gc scheduled by e.g. a cronjob ?

I mean JGit gc, scheduled via gerrit.config.

Aaron Smith

unread,

Apr 18, 2024, 1:07:54 PMApr 18

to Repo and Gerrit Discussion

On Thursday, April 18, 2024 at 8:01:33 AM UTC-7 tech....@gmail.com wrote:

On Thursday 18 April 2024 at 14:52:17 UTC+5:30 Luca Milanesio wrote:

On 18 Apr 2024, at 07:45, Sven Selberg <sven.s...@axis.com> wrote:

On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:
On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
Should we disable scheduled gc. Is that causing the issue?
If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this

It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.

There was an issue with JGit GC which was fixed months ago: it was an index out of bound on the Bitmap, which matches what it was reported.
Gertit v3.6.8 is EOL, so I doubt it contains the fixes. Can you reproduce it with the latest released version of Gerrit? (e.g. v3.9.4)

Luca.

We have a replica sandbox instance , which is also having same issue. we can try upgrading from 3.6.8 to 3.9.4 step by step.

I'll also try replicating this with 3.9.4 in my sandbox env.

Luca Milanesio

unread,

Apr 18, 2024, 1:11:02 PMApr 18

to Repo and Gerrit Discussion, Luca Milanesio, Aaron Smith

On 18 Apr 2024, at 18:07, 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

On Thursday, April 18, 2024 at 8:01:33 AM UTC-7 tech....@gmail.com wrote:
On Thursday 18 April 2024 at 14:52:17 UTC+5:30 Luca Milanesio wrote:

On 18 Apr 2024, at 07:45, Sven Selberg <sven.s...@axis.com> wrote:

On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:
On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
Should we disable scheduled gc. Is that causing the issue?
If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this

It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.

There was an issue with JGit GC which was fixed months ago: it was an index out of bound on the Bitmap, which matches what it was reported.
Gertit v3.6.8 is EOL, so I doubt it contains the fixes. Can you reproduce it with the latest released version of Gerrit? (e.g. v3.9.4)

Luca.

We have a replica sandbox instance , which is also having same issue. we can try upgrading from 3.6.8 to 3.9.4 step by step.

I'll also try replicating this with 3.9.4 in my sandbox env.

The problem I was facing with JGit was due to concurrency of git-receive-packs and GCs at the same time.

Try to do that and see if you can reproduce the issue with v3.6.8 and then again with v3.9.4.

If that still happens with v3.9.4, then it could be a different new issue that we haven’t seen or fixed yet.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2005e76a-bb78-4b1f-a592-89a93343f701n%40googlegroups.com.

Luca Milanesio

unread,

Apr 18, 2024, 4:19:21 PMApr 18

to Repo and Gerrit Discussion, Luca Milanesio, Aaron Smith

> On 18 Apr 2024, at 18:10, Luca Milanesio <luca.mi...@gmail.com> wrote:
>
>
>
>> On 18 Apr 2024, at 18:07, 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>>
>>
>> On Thursday, April 18, 2024 at 8:01:33 AM UTC-7 tech....@gmail.com wrote:
>> On Thursday 18 April 2024 at 14:52:17 UTC+5:30 Luca Milanesio wrote:
>>
>>> On 18 Apr 2024, at 07:45, Sven Selberg <sven.s...@axis.com> wrote:
>>>
>>>
>>>
>>> On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:
>>> On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
>>> Should we disable scheduled gc. Is that causing the issue?
>>> If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this
>>>
>>> It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.
>>
>> There was an issue with JGit GC which was fixed months ago: it was an index out of bound on the Bitmap, which matches what it was reported.
>> Gertit v3.6.8 is EOL, so I doubt it contains the fixes. Can you reproduce it with the latest released version of Gerrit? (e.g. v3.9.4)
>>
>> Luca.
>>
>> We have a replica sandbox instance , which is also having same issue. we can try upgrading from 3.6.8 to 3.9.4 step by step.
>>
>> I'll also try replicating this with 3.9.4 in my sandbox env.
>
> The problem I was facing with JGit was due to concurrency of git-receive-packs and GCs at the same time.
> Try to do that and see if you can reproduce the issue with v3.6.8 and then again with v3.9.4.

I checked and my JGit fix (f5f4bf0ad97f67ff56db18033c0a0795b722e96e) *IS NOT* included in v3.6.8 but *IS* included in v3.9.4.

Another option for you is to upgrade to the latest stable-3.6 (currently v3.6.8-36-g27847d3bbf) which contains the jgit bump that includes my fix:

commit 1a2b7ff50f14c767943f795f1fd56b399d8f5174
Author: Luca Milanesio <luca.mi...@gmail.com>
Date: Wed Nov 1 22:32:37 2023 +0000

Update JGit to acf21c0bc

$ git log --oneline --no-merges 74fa245b3..acf21c0bc
acf21c0bc RefDirectory: Do not unlock until after deleting loose ref
86ef2d531 Add missing javadoc description for declared exception
29c89d1f0 SnapshottingRefDirectory: Invalidate snapshot after locking ref for update
8197cab33 SnapshottingRefDir: Replace lambas with method refs
8b49e01ab SnapshottingRefDir: Reduce casts with overrides
747618358 Improve handling of NFS stale handle errors
ca54c5176 Fix handling of missing pack index file
60af389b4 Add tests for handling pack files removal during fetch
b4c66104f Introduce a PriorityQueue sorting RevCommits by commit timestamp
2d52df154 Remove org.eclipse.jgit.benchmark/.factorypath
10c5c17eb Update jmh to 1.37 for org.eclipse.jgit.benchmark
2177bed9a Silence API warnings
e712b4716 Make sure ref to prune is in packed refs
170244d05 Checkout: better directory handling
4d6671b4c (origin/stable-5.13) PackConfig: fix @since tags
244165fc5 Remove unused API problem filters
f103a1d5c Add support for git config repack.packKeptObjects
f5f4bf0ad Do not exclude objects in locked packs from bitmap processing <<<<==== This is my JGit fix

HTH

Luca.

tech....@gmail.com

unread,

Apr 21, 2024, 3:43:26 AMApr 21

to Repo and Gerrit Discussion

On Friday 19 April 2024 at 01:49:21 UTC+5:30 Luca Milanesio wrote:

> On 18 Apr 2024, at 18:10, Luca Milanesio <luca.mi...@gmail.com> wrote:
>
>
>
>> On 18 Apr 2024, at 18:07, 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>>
>>
>> On Thursday, April 18, 2024 at 8:01:33 AM UTC-7 tech....@gmail.com wrote:
>> On Thursday 18 April 2024 at 14:52:17 UTC+5:30 Luca Milanesio wrote:
>>
>>> On 18 Apr 2024, at 07:45, Sven Selberg <sven.s...@axis.com> wrote:
>>>
>>>
>>>
>>> On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:
>>> On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
>>> Should we disable scheduled gc. Is that causing the issue?
>>> If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this
>>>
>>> It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.

We have stopped scheduled gc from gerrit.config. should we run only "git gc " inside all bare repositories in the backend using cron for 6000+ repos ?

Or, we need to run ssh -p 29418 gerrit gc projectname , command to do manual gc ?

Sven Selberg

unread,

Apr 22, 2024, 3:07:14 AMApr 22

to Repo and Gerrit Discussion

On Sunday, April 21, 2024 at 9:43:26 AM UTC+2 tech....@gmail.com wrote:

On Friday 19 April 2024 at 01:49:21 UTC+5:30 Luca Milanesio wrote:

> On 18 Apr 2024, at 18:10, Luca Milanesio <luca.mi...@gmail.com> wrote:
>
>
>
>> On 18 Apr 2024, at 18:07, 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>>
>>
>> On Thursday, April 18, 2024 at 8:01:33 AM UTC-7 tech....@gmail.com wrote:
>> On Thursday 18 April 2024 at 14:52:17 UTC+5:30 Luca Milanesio wrote:
>>
>>> On 18 Apr 2024, at 07:45, Sven Selberg <sven.s...@axis.com> wrote:
>>>
>>>
>>>
>>> On Thursday, April 18, 2024 at 6:21:46 AM UTC+2 Aaron Smith wrote:
>>> On Wednesday, April 17, 2024 at 8:30:52 PM UTC-7 tech....@gmail.com wrote:
>>> Should we disable scheduled gc. Is that causing the issue?
>>> If we disable schedules gc, how we are going to do git garbage collection ? Repository size will grow wihtout this
>>>
>>> It seems like there's an issue with jgit gc, but you can still schedule `git gc` in a cron-job or simlar until the issue is fixed.
We have stopped scheduled gc from gerrit.config. should we run only "git gc " inside all bare repositories in the backend using cron for 6000+ repos ?

We use cron to, for each of our ~ 12.000 projects, run:
```

git repack --write-midx --write-bitmap-index -d --geometric=2

Sun - Fri: git pack-refs

Sat: git pack-refs --all

git prune --expire=1.hours.ago

```

Or, we need to run ssh -p 29418 gerrit gc projectname , command to do manual gc ?

That command, `gerrit gc`, also uses jgit gc so you should hit the same issues.

>>
>> There was an issue with JGit GC which was fixed months ago: it was an index out of bound on the Bitmap, which matches what it was reported.
>> Gertit v3.6.8 is EOL, so I doubt it contains the fixes. Can you reproduce it with the latest released version of Gerrit? (e.g. v3.9.4)
>>
>> Luca.
>>
>> We have a replica sandbox instance , which is also having same issue. we can try upgrading from 3.6.8 to 3.9.4 step by step.
>>
>> I'll also try replicating this with 3.9.4 in my sandbox env.
>
> The problem I was facing with JGit was due to concurrency of git-receive-packs and GCs at the same time.
> Try to do that and see if you can reproduce the issue with v3.6.8 and then again with v3.9.4.

I checked and my JGit fix (f5f4bf0ad97f67ff56db18033c0a0795b722e96e) *IS NOT* included in v3.6.8 but *IS* included in v3.9.4.

Another option for you is to upgrade to the latest stable-3.6 (currently v3.6.8-36-g27847d3bbf) which contains the jgit bump that includes my fix:

As Luca says another option should be to upgrade to latest stable-3.6 that contains the fix.

tech....@gmail.com

unread,

May 17, 2024, 3:44:24 AMMay 17

to Repo and Gerrit Discussion

Please guide me , where to find v3.6.8-36-g27847d3bbf Gerrit version

Fabio Ponciroli

unread,

May 17, 2024, 4:10:35 AMMay 17

to tech....@gmail.com, Repo and Gerrit Discussion

Hi,

You will need to build it yourself, getting the tip of stable-3.6.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/bd1cb187-0dba-4f76-ab17-8d9f3bc24a01n%40googlegroups.com.

Aaron Smith

unread,

Jun 12, 2024, 5:48:58 PMJun 12

to Repo and Gerrit Discussion

I upgraded to 3.10.0 this month and I'm happy to report that scheduled JGit gc now leaves the repositories in good shape, no issues found.

Reply all

Reply to author

Forward