git review performance on large monorepo — reftable attempt and ClosedChannelException

[Denis Fortin]

Hi,

We're running Gerrit 3.12 at leboncoin and facing performance issues on our main
Go monorepo for quite some time now. We have ~120 developers working on it. It
has accumulated ~1M refs in refs/changes/ over the years.

On this repo, `git review` takes ~24 seconds on average, with ~6s spent on the
SSH handshake/ref advertisement and the rest on server-side processing.
Using `git review --no-rebase` or the `git push refs/for/master` helps, but still
takes several seconds to execute.

We identified the ref count as the main culprit using `GIT_TRACE=1 git push` and
decided to try migrating to reftable.

This issue is specific to this monorepo. Our other repositories, including other
large ones, do not have the same performance problem.

First issue: After converting the repo with gerrit convert-ref-storage, every push failed with:
ClassCastException: FileReftableDatabase cannot be cast to RefDirectory
  at PerThreadRefDbCache.getRefDatabase (PerThreadRefDbCache.java:33)
We found that setting core.usePerRequestRefCache = false in gerrit.config works around this.

Second issue: With usePerRequestRefCache = false and reftable enabled, we started seeing errors in production after a few hours:
ClosedChannelException
  at FileChannelImpl.position
  at BlockSource$2.read
  at ReftableReader.readBlock
  at BanCommit.loadRejectCommitsMap
This caused both push failures and UI errors (500 on change pages). We had to roll back to refdir.

Server config (relevant parts):
- core.packedGitLimit = 12g
- core.packedGitWindowSize = 32m
- core.packedGitOpenFiles = 32768
- protocol.version = 2
- gc.aggressive = true, interval = 1 day

Our questions:
- Is reftable actually usable in production for our context?

- Are there other approaches to handle 1M refs performance that we might have missed?

Thanks

Denis

[Daniele Sassoli]

Hi Denis

On Wednesday, 3 June 2026 at 14:04:33 UTC+1 denis....@adevinta.com wrote:

Hi,

We're running Gerrit 3.12 at leboncoin and facing performance issues on our main
Go monorepo for quite some time now. We have ~120 developers working on it. It
has accumulated ~1M refs in refs/changes/ over the years.  

On this repo, `git review` takes ~24 seconds on average, with ~6s spent on the
SSH handshake/ref advertisement and the rest on server-side processing.
Using `git review --no-rebase` or the `git push refs/for/master` helps, but still
takes several seconds to execute.

What's git review? Are you referring to [1]? In that case I'm not sure this is the best

forum to report issues, but I'm also unaware of a better one. I know some mediawiki
maintainers are on the list so hopefully they can help.

We identified the ref count as the main culprit using `GIT_TRACE=1 git push` and
decided to try migrating to reftable.

This issue is specific to this monorepo. Our other repositories, including other
large ones, do not have the same performance problem.

First issue: After converting the repo with gerrit convert-ref-storage, every push failed with:
ClassCastException: FileReftableDatabase cannot be cast to RefDirectory
  at PerThreadRefDbCache.getRefDatabase (PerThreadRefDbCache.java:33)
We found that setting core.usePerRequestRefCache = false in gerrit.config works around this.

Yeah, this is a known issue, we never got round to merging the documentation update at [2]. 

Second issue: With usePerRequestRefCache = false and reftable enabled, we started seeing errors in production after a few hours:
ClosedChannelException
  at FileChannelImpl.position
  at BlockSource$2.read
  at ReftableReader.readBlock
  at BanCommit.loadRejectCommitsMap
This caused both push failures and UI errors (500 on change pages). We had to roll back to refdir.

Server config (relevant parts):
- core.packedGitLimit = 12g
- core.packedGitWindowSize = 32m
- core.packedGitOpenFiles = 32768
- protocol.version = 2
- gc.aggressive = true, interval = 1 day

Our questions:
- Is reftable actually usable in production for our context?

 

Should be, would be good to know more about the issues you where facing.
 

- Are there other approaches to handle 1M refs performance that we might have missed?

 

Have you looked at [3]? NOTE: This was open source up to 3.12, but requires a licence from 3.13 onwards.
This allows you to hide refs from advertising which should speed up that part of the process.

[1] https://www.mediawiki.org/wiki/Gerrit/git-review
[2] https://gerrit-review.googlesource.com/c/gerrit/+/512283
[3] https://github.com/GerritForge/git-refs-filter

Thanks

Denis

[Michael Nazzareno Trimarchi]

Hi

Was a bit curios and I have done some investigation and this is what I found coming from some brainstorming idea with AI. I don't test it but I will do. For jgit I think you can tweak in jgit.config. 

Michael 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/acd91c1c-6746-497c-bdad-df4baef44a6an%40googlegroups.com.

[Daniele Sassoli]

Hi Michael

On Wednesday, 3 June 2026 at 14:50:26 UTC+1 mic...@amarulasolutions.com wrote:

Hi

Was a bit curios and I have done some investigation and this is what I found coming from some brainstorming idea with AI. I don't test it but I will do. For jgit I think you can tweak in jgit.config. 

Please avoid top-posting on this list and instead use interleaved posting, as per [4].

For what concerns the AI generated document, it's clearly just a random assortment of fairly basic suggestions.
If most of those where not in place already the where be bigger issues on this Gerrit installations than the ones described so far.
Also, none of them address the ref advertisement phase that Denis mentions.

Please let's refrain from just shooting AI generated documents with untested and unvetted suggestions onto the mailing list.

[4] https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
 

[Michael Nazzareno Trimarchi]

Hi

Il mer 3 giu 2026, 16:01 Daniele Sassoli <daniele...@gmail.com> ha scritto:

Hi Michael

On Wednesday, 3 June 2026 at 14:50:26 UTC+1 mic...@amarulasolutions.com wrote:

Hi

Was a bit curios and I have done some investigation and this is what I found coming from some brainstorming idea with AI. I don't test it but I will do. For jgit I think you can tweak in jgit.config. 

Please avoid top-posting on this list and instead use interleaved posting, as per [4].

Sorry not always easy with mobile phone. Anyway ...

For what concerns the AI generated document, it's clearly just a random assortment of fairly basic suggestions.
If most of those where not in place already the where be bigger issues on this Gerrit installations than the ones described so far.
Also, none of them address the ref advertisement phase that Denis mentions.

Ok 

Please let's refrain from just shooting AI generated documents with untested and unvetted suggestions onto the mailing list.

No problem, I was just misread the topic. I will pay more attention.

Michael 

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/d35fe9ea-2c85-4d03-a109-0fd517a6ce8dn%40googlegroups.com.

Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
mic...@amarulasolutions.com
__________________________________

Amarula Solutions BV

Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
in...@amarulasolutions.com
www.amarulasolutions.com

[mfick]

On Wednesday, June 3, 2026 at 7:04:33 AM UTC-6 denis....@adevinta.com wrote:

We're running Gerrit 3.12 at leboncoin and facing performance issues on our main
Go monorepo for quite some time now. We have ~120 developers working on it. It
has accumulated ~1M refs in refs/changes/ over the years.

On this repo, `git review` takes ~24 seconds on average, with ~6s spent on the
SSH handshake/ref advertisement and the rest on server-side processing.
Using `git review --no-rebase` or the `git push refs/for/master` helps, but still
takes several seconds to execute.

While I am not familiar with git review, I can confirm that 6s is not unreasonable for 1M ref

advertisement. Generally, I measure this with a git ls-remote, and our server can perform an

ls-remote on a ~4M ref repo in about 10s. There might be some room for improvement on 

yours, but you are in the right ballpark.

 

First issue: After converting the repo with gerrit convert-ref-storage, every push failed with:

FYI, I would not expect the ref advertisement to get any faster with RefTables. While RefTables

are a great improvement for many things, I think a pure dump of all the refs should actually be 

slower with it compared to a well packed ref directory as it likely has more table (files), and longer

histories to read. 

That being said, slow ref advertisements in Gerrit and a well packed repo (yours is well packed 

right?) are generally due to ACLs, as Gerrit ACLs are rather computationally expensive. If you 

have not already, you show be sure to size the changes_by_project cache large enough. I try to 

fit all the changes from the entire server into it. I use the following guideline for the 

changes_by_project cache, I advise not making it bigger than 1/10 the heap, and no bigger than 

10K * # of changes on the entire server.  

Of course, network speed matters a bit also for advertisements.

 

- gc.aggressive = true, interval = 1 day

Aggressive is not generally advised on servers where most people would gladly trade space for 

fetch speed.

 

Our questions:
- Is reftable actually usable in production for our context?

If you are using this in production (aside from the google version) please add some anecdotes

for the rest of us. :)

-Martin

[Daniele Sassoli]

On Wednesday, 3 June 2026 at 15:28:32 UTC+1 mfick wrote:

On Wednesday, June 3, 2026 at 7:04:33 AM UTC-6 denis....@adevinta.com wrote:

We're running Gerrit 3.12 at leboncoin and facing performance issues on our main
Go monorepo for quite some time now. We have ~120 developers working on it. It
has accumulated ~1M refs in refs/changes/ over the years.

On this repo, `git review` takes ~24 seconds on average, with ~6s spent on the
SSH handshake/ref advertisement and the rest on server-side processing.
Using `git review --no-rebase` or the `git push refs/for/master` helps, but still
takes several seconds to execute.

While I am not familiar with git review, I can confirm that 6s is not unreasonable for 1M ref

advertisement. Generally, I measure this with a git ls-remote, and our server can perform an

ls-remote on a ~4M ref repo in about 10s. There might be some room for improvement on 

yours, but you are in the right ballpark.

I guess the question is, do you really need to advertise them all? Can some be hidden with
JGit configs? (or Gerrit ACLs, or git-refs-filter, depending on the use case).
 

[Luca Milanesio]

My 2c inline

> On 3 Jun 2026, at 15:40, Daniele Sassoli <daniele...@gmail.com> wrote:
>
>
>
> On Wednesday, 3 June 2026 at 15:28:32 UTC+1 mfick wrote:
> On Wednesday, June 3, 2026 at 7:04:33 AM UTC-6 denis....@adevinta.com wrote:
> We're running Gerrit 3.12 at leboncoin and facing performance issues on our main
> Go monorepo for quite some time now. We have ~120 developers working on it. It
> has accumulated ~1M refs in refs/changes/ over the years.
>
> On this repo, `git review` takes ~24 seconds on average, with ~6s spent on the
> SSH handshake/ref advertisement and the rest on server-side processing.
> Using `git review --no-rebase` or the `git push refs/for/master` helps, but still
> takes several seconds to execute.

With 1M refs, the above times are expected, mainly because of the need to transfer 1M refs in text format over the wire, and that means all the ref names and their SHA1s in ASCII.

>
> While I am not familiar with git review, I can confirm that 6s is not unreasonable for 1M ref
> advertisement. Generally, I measure this with a git ls-remote, and our server can perform an
> ls-remote on a ~4M ref repo in about 10s. There might be some room for improvement on
> yours, but you are in the right ballpark.
>
> I guess the question is, do you really need to advertise them all? Can some be hidden with
> JGit configs? (or Gerrit ACLs, or git-refs-filter, depending on the use case).

You cannot do that with Gerrit ACLs, because most of the refs are changes and patch-sets. Gerrit ACLs are aren’t specific for individual changes / patch-sets.
The only solution AFAIK is git-refs-filter as Dani mentioned.

See the GerritMeets talk:
https://youtu.be/U-fqHzmh8Eo

HTH

Luca.

>
> First issue: After converting the repo with gerrit convert-ref-storage, every push failed with:
>
> FYI, I would not expect the ref advertisement to get any faster with RefTables. While RefTables
> are a great improvement for many things, I think a pure dump of all the refs should actually be
> slower with it compared to a well packed ref directory as it likely has more table (files), and longer
> histories to read.
>
> That being said, slow ref advertisements in Gerrit and a well packed repo (yours is well packed
> right?) are generally due to ACLs, as Gerrit ACLs are rather computationally expensive. If you
> have not already, you show be sure to size the changes_by_project cache large enough. I try to
> fit all the changes from the entire server into it. I use the following guideline for the
> changes_by_project cache, I advise not making it bigger than 1/10 the heap, and no bigger than
> 10K * # of changes on the entire server.
>
> Of course, network speed matters a bit also for advertisements.
> - gc.aggressive = true, interval = 1 day
>
> Aggressive is not generally advised on servers where most people would gladly trade space for
> fetch speed.
> Our questions:
> - Is reftable actually usable in production for our context?
>
> If you are using this in production (aside from the google version) please add some anecdotes
> for the rest of us. :)
>
> -Martin
>
>

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/45e0b541-e700-4faf-beb3-e27e28ba357fn%40googlegroups.com.

[Denis Fortin]

Thanks for all the feedback, it's very helpful.

> With 1M refs, the above times are expected, mainly because of the need to transfer 1M refs in text format over the wire, and that means all the ref names and their SHA1s in ASCII.

It's good to know. 

I'm wondering if there could be a mechanism to reduce the number of advertised refs without losing history. 

For instance, by archiving old refs/changes somewhere that Gerrit would still use for the UI, but that wouldn't be advertised to clients. 

Not sure if that's feasible or if something similar has already been discussed?

> What's git review? Are you referring to [1]? In that case I'm not sure this is the best

> forum to report issues, but I'm also unaware of a better one. I know some mediawiki
> maintainers are on the list so hopefully they can help.

Yes I was referring to the tool mentioned in mediawiki [1]. I thought it was a common tool in the Gerrit ecosystem.

Sorry, I was not intending to report an issue with the tool, but rather with what's behind the tool in the git part.

I understand better now.

> Have you looked at [3]? NOTE: This was open source up to 3.12, but requires a licence from 3.13 onwards.
> This allows you to hide refs from advertising which should speed up that part of the process.

I didn't know git-refs-filter. But as I understood, if we hide the refs/changes to everyone, the developers won't be able to check out their reviews/patchsets anymore. Right?

It's still very interesting for the CI process.

> That being said, slow ref advertisements in Gerrit and a well packed repo (yours is well packed right?)

I'll have to check that.

> If you have not already, you show be sure to size the changes_by_project cache large enough. I try to 

> fit all the changes from the entire server into it. I use the following guideline for the 

> changes_by_project cache, I advise not making it bigger than 1/10 the heap, and no bigger than 

> 10K * # of changes on the entire server.  

Very interesting. I'll have to try to fine tune this on our instance.

> Aggressive is not generally advised on servers where most people would gladly trade space for 

> fetch speed.

Didn't know that either, thanks for the tip. I'll try that also.

> If you are using this in production (aside from the google version) please add some anecdotes

> for the rest of us. :)

We tried, but had to rollback because of a ClosedChannelException in JGit's ReftableReader. 

It appeared to be a race condition related to usePerRequestRefCache = false, which was required to work around the ClassCastException. 

I didn't investigate further as it was too impactful for our developers.

[1] https://docs.opendev.org/opendev/git-review/latest/ 

[mfick]

On Thursday, June 4, 2026 at 3:52:30 AM UTC-6 denis....@adevinta.com wrote:

Thanks for all the feedback, it's very helpful.

> With 1M refs, the above times are expected, mainly because of the need to transfer 1M refs in text format over the wire, and that means all the ref names and their SHA1s in ASCII.

It's good to know. 

I'm wondering if there could be a mechanism to reduce the number of advertised refs without losing history. 

Advertised refs have a purpose, they allow the world to know what they could potentially fetch from a server, so reducing these impacts functionality. That being said, I think it is worth asking the question, are their use cases which do not require advertised refs, and if so, how can be make sure that ref counts no longer impact these uses cases in bad ways. So, for example git protocol v2 simply does not advertise refs for fetches any more, unless the client requests them. In this case, faking in reducing the advertised refs should be seen as a regression, some functionality is lost. However, git p v2 is only available for upload-packs (fetches) currently, not for receive-pack. Making protocol v2 available for push would be very beneficial here, and it likely is not that difficult to do as upload-pack has already set an example of how to do things.

This brings us back to your use case, and the question becomes, why is your git review slow? When it is making use of advertised refs, is it pushing to Gerrit, or only fetching? If fetching, then are you running an old git client or Gerrit server w/o protocol V2 support, or do you have it turned off for some reason?

> If you are using this in production (aside from the google version) please add some anecdotes

> for the rest of us. :)

We tried, but had to rollback because of a ClosedChannelException in JGit's ReftableReader. 

It appeared to be a race condition related to usePerRequestRefCache = false, which was required to work around the ClassCastException. 

I didn't investigate further as it was too impactful for our developers.

I was not aware of a race condition, could it be that it is just too slow and that clients are timing out (and hanging up) due to this switch? Are you using NFS (turning off the perRequest cache can be pretty bad on NFS, but unfortunately, the RefTables implementation does not support turning this on yet)?

-Martin