Slow loading of detail.json

[phil]

After upgrading from 3.8.9 to 3.10.8 ("a reindex of all indexes" also done per release notes) some changes load for 30+ seconds, might be even up to a minute. In web inspector—network I see this request taking longest:

GET https://gerrit/changes/repo_name~7973/detail?O=9996394

It is a new issue, according to server users. What could possibly cause this? The Change Log has 55 entries plus 32 hidden (not visible by default). Is it just too large?

[phil]

On Friday, October 10, 2025 at 5:44:22 PM UTC+3 phil wrote:

After upgrading from 3.8.9 to 3.10.8 ("a reindex of all indexes" also done per release notes) some changes load for 30+ seconds, might be even up to a minute. In web inspector—network I see this request taking longest:

GET https://gerrit/changes/repo_name~7973/detail?O=9996394

Resolved with 3.11.5 upgrade, probably not worth pursuing further.

[phil]

On Wednesday, October 15, 2025 at 8:08:46 PM UTC+3 phil wrote:

On Friday, October 10, 2025 at 5:44:22 PM UTC+3 phil wrote:

After upgrading from 3.8.9 to 3.10.8 ("a reindex of all indexes" also done per release notes) some changes load for 30+ seconds, might be even up to a minute. In web inspector—network I see this request taking longest:

GET https://gerrit/changes/repo_name~7973/detail?O=9996394

Resolved with 3.11.5 upgrade, probably not worth pursuing further.

Actually scratch that. Even on 3.12.3 the reply takes 10-12 seconds to submit on this one repository. In other repos on the same server reply submits happen near instantaneously, and it used to be instant on the repo in question too. Would be happy to hear any hints how to investigate this further.

[Luca Milanesio]

> On 15 Jan 2026, at 14:19, phil <phil.s...@gmail.com> wrote:
>
>
>
> On Wednesday, October 15, 2025 at 8:08:46 PM UTC+3 phil wrote:
> On Friday, October 10, 2025 at 5:44:22 PM UTC+3 phil wrote:
> After upgrading from 3.8.9 to 3.10.8 ("a reindex of all indexes" also done per release notes) some changes load for 30+ seconds, might be even up to a minute. In web inspector—network I see this request taking longest:
>
> GET https://gerrit/changes/repo_name~7973/detail?O=9996394
>
> Resolved with 3.11.5 upgrade, probably not worth pursuing further.
> Actually scratch that. Even on 3.12.3 the reply takes 10-12 seconds to submit on this one repository. In other repos on the same server reply submits happen near instantaneously, and it used to be instant on the repo in question too. Would be happy to hear any hints how to investigate this further.

Because the call takes a long time (over 30s), you should take multiple Java thread-dump of the Gerrit JVM pid (e.g. 60 dumps, with a 0.5s interval) and see if you recognise some “hot points” where the code is stuck.

HTH

Luca.

[Matthias Sohn]

Did you check if this repository is well packed ?

Maybe git garbage collection isn't scheduled to run for this repository or fails with an error.

You can run git count-objects -vH on this repository on the Gerrit server to check if it has

an excessive number of loose objects or pack files.

HTH

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/ED32DA70-7F0A-421A-B8B3-8BAB362C6721%40gmail.com.

[phil]

OK, it is more like 10-12 sec now, but I might investigate this Java thread dump thing anyway, and report what I find, thanks. Not familiar with the process.

HTH

Luca.

[phil]

On Thursday, January 15, 2026 at 5:08:39 PM UTC+2 Matthias Sohn wrote:

On Thu, Jan 15, 2026 at 3:36 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

> On 15 Jan 2026, at 14:19, phil <phil.s...@gmail.com> wrote:
>
>
>
> On Wednesday, October 15, 2025 at 8:08:46 PM UTC+3 phil wrote:
> On Friday, October 10, 2025 at 5:44:22 PM UTC+3 phil wrote:
> After upgrading from 3.8.9 to 3.10.8 ("a reindex of all indexes" also done per release notes) some changes load for 30+ seconds, might be even up to a minute. In web inspector—network I see this request taking longest:
>
> GET https://gerrit/changes/repo_name~7973/detail?O=9996394
>
> Resolved with 3.11.5 upgrade, probably not worth pursuing further.
> Actually scratch that. Even on 3.12.3 the reply takes 10-12 seconds to submit on this one repository. In other repos on the same server reply submits happen near instantaneously, and it used to be instant on the repo in question too. Would be happy to hear any hints how to investigate this further.


Because the call takes a long time (over 30s), you should take multiple Java thread-dump of the Gerrit JVM pid (e.g. 60 dumps, with a 0.5s interval) and see if you recognise some “hot points” where the code is stuck.

Did you check if this repository is well packed ?

Maybe git garbage collection isn't scheduled to run for this repository or fails with an error.

You can run git count-objects -vH on this repository on the Gerrit server to check if it has

an excessive number of loose objects or pack files.

OK, I'm not well-familiar with git internals, but I can see the following output (running directly on the gerrit server VM in the repo directory ):

gerrit@gerrit:/var/gerrit/git/repo_name.git$ git count-objects -vH
count: 145283
size: 577.32 MiB
in-pack: 818015
packs: 10740
size-pack: 1.49 GiB
prune-packable: 108
garbage: 0
size-garbage: 0 bytes

Does zero garbage indicate that GC is running as it should? Or is "count" high enough to be as you said excessive? Worth running GC manually now, and maybe scheduling it with cron after that? Is there a recommendation for frequency? I could not find any settings in GUI except for the actual "Run GC" button.

I also dry-ran "git prune-packed -n" and got "Removing duplicate objects: 100% (256/256), done." as a result. Worth running this command for real?

[Daniele Sassoli]

Very confusingly, the garbage and size-garbage fields of the git count-objects -v output have nothing to do with garbage collection. Those indicate really garbage files (like partial packfiles caused by temporary operations), see [1]. From what I understand git garbage collection will not actually affect the amount of garbage printed in those fields(or at least not always)

You've definitely got quite a few objects, although not a stupid crazy amount. But yeah, I'd say worth running GC.

recommendation for frequency? I could not find any settings in GUI except for the actual "Run GC" button.

 

I'd recommend scheduling jgit gc to run separately as a cron-job.

I also dry-ran "git prune-packed -n" and got "Removing duplicate objects: 100% (256/256), done." as a result. Worth running this command for real?

I'd imagine GC will take care of this too for you. 

[1] https://lore.kernel.org/git/CADR1ei41b6oNONF1GPiWMm+LdwbV7_2K=vBy1b30O...@mail.gmail.com/

[Matthias Sohn]

On Fri, Jan 16, 2026 at 4:09 PM phil <phil.s...@gmail.com> wrote:

On Thursday, January 15, 2026 at 5:08:39 PM UTC+2 Matthias Sohn wrote:

On Thu, Jan 15, 2026 at 3:36 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

> On 15 Jan 2026, at 14:19, phil <phil.s...@gmail.com> wrote:
>
>
>
> On Wednesday, October 15, 2025 at 8:08:46 PM UTC+3 phil wrote:
> On Friday, October 10, 2025 at 5:44:22 PM UTC+3 phil wrote:
> After upgrading from 3.8.9 to 3.10.8 ("a reindex of all indexes" also done per release notes) some changes load for 30+ seconds, might be even up to a minute. In web inspector—network I see this request taking longest:
>
> GET https://gerrit/changes/repo_name~7973/detail?O=9996394
>
> Resolved with 3.11.5 upgrade, probably not worth pursuing further.
> Actually scratch that. Even on 3.12.3 the reply takes 10-12 seconds to submit on this one repository. In other repos on the same server reply submits happen near instantaneously, and it used to be instant on the repo in question too. Would be happy to hear any hints how to investigate this further.


Because the call takes a long time (over 30s), you should take multiple Java thread-dump of the Gerrit JVM pid (e.g. 60 dumps, with a 0.5s interval) and see if you recognise some “hot points” where the code is stuck.

Did you check if this repository is well packed ?

Maybe git garbage collection isn't scheduled to run for this repository or fails with an error.

You can run git count-objects -vH on this repository on the Gerrit server to check if it has

an excessive number of loose objects or pack files.

OK, I'm not well-familiar with git internals, but I can see the following output (running directly on the gerrit server VM in the repo directory ):

gerrit@gerrit:/var/gerrit/git/repo_name.git$ git count-objects -vH
count: 145283

This repo has 145k loose objects, depending on the type of filesystem you are using this can cause performance degradation.

On local file storage I would try to keep this <10k, when using NFS <1k.

 

size: 577.32 MiB
in-pack: 818015
packs: 10740

This is the smoking gun, with >10k pack files jgit has to search through >10k pack indexes to find an object.

This causes massive performance degradation. As a rule of thumb try keeping this metric <100.

 

size-pack: 1.49 GiB
prune-packable: 108
garbage: 0
size-garbage: 0 bytes

Does zero garbage indicate that GC is running as it should? Or is "count" high enough to be as you said excessive? Worth running GC manually now, and maybe scheduling it with cron after that? Is there a recommendation for frequency? I could not find any settings in GUI except for the actual "Run GC" button.

I also dry-ran "git prune-packed -n" and got "Removing duplicate objects: 100% (256/256), done." as a result. Worth running this command for real?

HTH

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/ED32DA70-7F0A-421A-B8B3-8BAB362C6721%40gmail.com.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/2160e43b-ef17-455c-9044-b5bc70ab3d90n%40googlegroups.com.

[luca.mi...@gmail.com]

Sent from my iPhone

On 16 Jan 2026, at 17:18, Matthias Sohn <matthi...@gmail.com> wrote:



You may want to install the git-repo-metrics plugin that exposes those values as real time metrics and then putting an alarm on it.

HTH

Luca

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3QAFVdtzEE1dZO1qxLYmXXEBT8UuLAwLJ-YOiTDTDZoEA%40mail.gmail.com.

[phil]

Thanks for the pointers. GC took about 22 min to run and indeed made it "tiny" again:

gerrit@gerrit:/var/gerrit/git/repo_name.git$ git count-objects -vH

count: 2535
size: 11.75 MiB
in-pack: 389660
packs: 3
size-pack: 135.49 MiB
prune-packable: 0

garbage: 0
size-garbage: 0 bytes

Will schedule it to run regularly.

You may want to install the git-repo-metrics plugin that exposes those values as real time metrics and then putting an alarm on it.

Thanks, will see about that too.

[phil]

...of course. Thanks, TIL.

 

You've definitely got quite a few objects, although not a stupid crazy amount. But yeah, I'd say worth running GC.

recommendation for frequency? I could not find any settings in GUI except for the actual "Run GC" button.

 

I'd recommend scheduling jgit gc to run separately as a cron-job.

I also dry-ran "git prune-packed -n" and got "Removing duplicate objects: 100% (256/256), done." as a result. Worth running this command for real?

I'd imagine GC will take care of this too for you. 

Yes, can confirm it did.