Improving Fetch and Push Latencies

[Joel Thompson]

Hi,

I manage a Gerrit server with a moderately sized repository, and Gerrit usage for this repository has become increasingly slow for our users. I have a couple ideas on how the performance can be improved (summarized in the last paragraph) and would like your feedback.

Specifically, we have about 100K total distinct changes, 500K total `refs/changes/*` refs (including the `refs/changes/*/meta` refs), and about 1.5K "other" refs (mostly branches, some tags). Doing a `get fetch` or a `git push` will always take about 11 seconds, even if the amount of data to transfer is relatively small. Both the Gerrit server and the git clients have plenty of resources (CPU/memory/etc.) and are in the same AWS region so there's low latency between them.

I believe what's happening in both cases is JGit is calling the `PermissionAwareReadOnlyRefDatabase` object's `getRefs` method, which is then iterating serially  over all 500K+ refs and doing a permission check on each and every single one of them, which is what is causing this observed slowness.

When performing a git fetch, with `GIT_TRACE_PACKET` enabled, I can see that my client is sending a `command=ls-refs` with `ref-prefix refs/heads/` and `ref-prefix refs/tags/`, and Gerrit is responding with the 1.5K or so matching refs pretty quickly (about 150ms). My client then sends over `command=fetch` specifying the object IDs wanted via `want` arguments followed by `have` arguments, then followed by the flush-pkt. It then takes Gerrit 10.9 seconds to respond with `acknowledgments`, and then the fetch proceeds fine from there. Reading through the JGit code, I believe that this code path is calling `PermissionAwareReadOnlyRefDatabase.getRefs`.

I believe there's a very simple solution to this -- setting `uploadpack.allowRefInWant` via `etc/jgit.config`. I've implemented this in production, and now `git fetch` operations start similarly (except the server now advertises the `ref-in-want` feature), up until my client issues the `fetch` command, when it now sends `want-ref` arguments instead of `want` arguments. Gerrit then sends back the `acknowledgements` in around 360ms rather than 10.9s. By sending over only `want-ref` arguments, the `PermissionAwareReadOnlyRefDatabase` only has to permission check the individual refs requested, rather than every single ref in the database, and thus it's much, much quicker.

I think that Gerrit should set this as a default -- in the same way `receive.autogc` is set `false` upon Gerrit startup unless it's explicitly set to `true`, Gerrit could set `uploadpack.allowRefInWant` to `true` unless it's explicitly set to false. Alternatively, this could be documented in the Gerrit documentation to make users more aware of its existence.

Unfortunately, it seems there isn't as easy of a fix for the slow git pushes. Again enabling `GIT_TRACE_PACKET`, I can see about 10.9 seconds between when `git-receive-pack` is started on the remote server until the server sends the first ref back. Reading through the code, I believe it's calling the same `PermissionAwareReadOnlyRefDatabase.getRefs` method without any filtering, which causes Gerrit to permission check all 500K+ refs, which is causing the 10.9 slowdown. In theory, setting `receive.hideRefs` to `refs/changes/` should speed this up, but JGit doesn't appear to respect `receive.hideRefs` (specifically, it only looks for `uploadpack.hideRefs` but then setting this causes it to behave the way `transfer.hideRefs` should).

Gerrit already does filter out the `refs/changes` refs before responding to the `receive-pack`, but this filtering is done AFTER all permissions checks, not before, and as a result, Gerrit is performing 500K permission checks and then discarding the results of 99.5% of them. I believe a more efficient way to handle this would be to filter out `refs/changes` before doing the permissions check. I think this could be implemented in `AsyncReceiveCommits` -- where it calls `PermissionAwareRepositoryManager.wrap` around the bare repository, a similar wrapper around the bare repository object could be created which filters out all `refs/changes` first, and then this wrapper could then be wrapped with `PermissionAwareRepositoryManager.wrap`.

Thoughts on these two changes, specifically defaulting `uploadpack.allowRefInWant` to `true` and filtering out `refs/changes` before the permission check in `AsyncReceiveCommits`? Do they seem valuable to the Gerrit project? Would you be willing to accept contributions implementing them?

Thanks!

--Joel

[Daniele Sassoli]

Hi Joel,

Thanks for your in-depth analysis.
I'll start by saying that there's people here that know more than me, but I'll give my 2 cents below.

On Tuesday, 19 November 2024 at 07:06:48 UTC Joel Thompson wrote:

Hi,

I manage a Gerrit server with a moderately sized repository, and Gerrit usage for this repository has become increasingly slow for our users. I have a couple ideas on how the performance can be improved (summarized in the last paragraph) and would like your feedback.

Specifically, we have about 100K total distinct changes, 500K total `refs/changes/*` refs (including the `refs/changes/*/meta` refs), and about 1.5K "other" refs (mostly branches, some tags). Doing a `get fetch` or a `git push` will always take about 11 seconds, even if the amount of data to transfer is relatively small. Both the Gerrit server and the git clients have plenty of resources (CPU/memory/etc.) and are in the same AWS region so there's low latency between them.

I believe what's happening in both cases is JGit is calling the `PermissionAwareReadOnlyRefDatabase` object's `getRefs` method, which is then iterating serially  over all 500K+ refs and doing a permission check on each and every single one of them, which is what is causing this observed slowness.

When performing a git fetch, with `GIT_TRACE_PACKET` enabled, I can see that my client is sending a `command=ls-refs` with `ref-prefix refs/heads/` and `ref-prefix refs/tags/`, and Gerrit is responding with the 1.5K or so matching refs pretty quickly (about 150ms). My client then sends over `command=fetch` specifying the object IDs wanted via `want` arguments followed by `have` arguments, then followed by the flush-pkt. It then takes Gerrit 10.9 seconds to respond with `acknowledgments`, and then the fetch proceeds fine from there. Reading through the JGit code, I believe that this code path is calling `PermissionAwareReadOnlyRefDatabase.getRefs`.

I believe there's a very simple solution to this -- setting `uploadpack.allowRefInWant` via `etc/jgit.config`. I've implemented this in production, and now `git fetch` operations start similarly (except the server now advertises the `ref-in-want` feature), up until my client issues the `fetch` command, when it now sends `want-ref` arguments instead of `want` arguments. Gerrit then sends back the `acknowledgements` in around 360ms rather than 10.9s. By sending over only `want-ref` arguments, the `PermissionAwareReadOnlyRefDatabase` only has to permission check the individual refs requested, rather than every single ref in the database, and thus it's much, much quicker.

I think that Gerrit should set this as a default -- in the same way `receive.autogc` is set `false` upon Gerrit startup unless it's explicitly set to `true`, Gerrit could set `uploadpack.allowRefInWant` to `true` unless it's explicitly set to false. Alternatively, this could be documented in the Gerrit documentation to make users more aware of its existence.

 

I believe there are security reasons for which these setting are disabled by default, but document contributions are always welcome.
Have you tried setting "allowAnySHA1InWant"? I suspect it might give you an even better performance boost.

Unfortunately, it seems there isn't as easy of a fix for the slow git pushes. Again enabling `GIT_TRACE_PACKET`, I can see about 10.9 seconds between when `git-receive-pack` is started on the remote server until the server sends the first ref back. Reading through the code, I believe it's calling the same `PermissionAwareReadOnlyRefDatabase.getRefs` method without any filtering, which causes Gerrit to permission check all 500K+ refs, which is causing the 10.9 slowdown. In theory, setting `receive.hideRefs` to `refs/changes/` should speed this up, but JGit doesn't appear to respect `receive.hideRefs` (specifically, it only looks for `uploadpack.hideRefs` but then setting this causes it to behave the way `transfer.hideRefs` should).

Gerrit already does filter out the `refs/changes` refs before responding to the `receive-pack`, but this filtering is done AFTER all permissions checks, not before, and as a result, Gerrit is performing 500K permission checks and then discarding the results of 99.5% of them. I believe a more efficient way to handle this would be to filter out `refs/changes` before doing the permissions check. I think this could be implemented in `AsyncReceiveCommits` -- where it calls `PermissionAwareRepositoryManager.wrap` around the bare repository, a similar wrapper around the bare repository object could be created which filters out all `refs/changes` first, and then this wrapper could then be wrapped with `PermissionAwareRepositoryManager.wrap`.

Have you experimented with the git-refs-filter plugin[1]? 
It should reduce the amount of refs advertised and hopefully speed up your operations.

Thoughts on these two changes, specifically defaulting `uploadpack.allowRefInWant` to `true` and filtering out `refs/changes` before the permission check in `AsyncReceiveCommits`? Do they seem valuable to the Gerrit project? Would you be willing to accept contributions implementing them?

Thanks!

--Joel

Thanks,
Dani

[1] https://gerrit.googlesource.com/modules/git-refs-filter/ 

[Matthias Sohn]

On Tue, Nov 19, 2024 at 8:06 AM Joel Thompson <jatho...@gmail.com> wrote:

Hi,

I manage a Gerrit server with a moderately sized repository, and Gerrit usage for this repository has become increasingly slow for our users. I have a couple ideas on how the performance can be improved (summarized in the last paragraph) and would like your feedback.

Specifically, we have about 100K total distinct changes, 500K total `refs/changes/*` refs (including the `refs/changes/*/meta` refs), and about 1.5K "other" refs (mostly branches, some tags). Doing a `get fetch` or a `git push` will always take about 11 seconds, even if the amount of data to transfer is relatively small. Both the Gerrit server and the git clients have plenty of resources (CPU/memory/etc.) and are in the same AWS region so there's low latency between them.

I believe what's happening in both cases is JGit is calling the `PermissionAwareReadOnlyRefDatabase` object's `getRefs` method, which is then iterating serially  over all 500K+ refs and doing a permission check on each and every single one of them, which is what is causing this observed slowness.

run the git command with tracing enabled [1], this should give you timing details in the error_log 

[1] https://gerrit-review.googlesource.com/Documentation/user-request-tracing.html

When performing a git fetch, with `GIT_TRACE_PACKET` enabled, I can see that my client is sending a `command=ls-refs` with `ref-prefix refs/heads/` and `ref-prefix refs/tags/`, and Gerrit is responding with the 1.5K or so matching refs pretty quickly (about 150ms). My client then sends over `command=fetch` specifying the object IDs wanted via `want` arguments followed by `have` arguments, then followed by the flush-pkt. It then takes Gerrit 10.9 seconds to respond with `acknowledgments`, and then the fetch proceeds fine from there. Reading through the JGit code, I believe that this code path is calling `PermissionAwareReadOnlyRefDatabase.getRefs`.

I believe there's a very simple solution to this -- setting `uploadpack.allowRefInWant` via `etc/jgit.config`. I've implemented this in production, and now `git fetch` operations start similarly (except the server now advertises the `ref-in-want` feature), up until my client issues the `fetch` command, when it now sends `want-ref` arguments instead of `want` arguments. Gerrit then sends back the `acknowledgements` in around 360ms rather than 10.9s. By sending over only `want-ref` arguments, the `PermissionAwareReadOnlyRefDatabase` only has to permission check the individual refs requested, rather than every single ref in the database, and thus it's much, much quicker.

I think that Gerrit should set this as a default -- in the same way `receive.autogc` is set `false` upon Gerrit startup unless it's explicitly set to `true`, Gerrit could set `uploadpack.allowRefInWant` to `true` unless it's explicitly set to false. Alternatively, this could be documented in the Gerrit documentation to make users more aware of its existence.

Unfortunately, it seems there isn't as easy of a fix for the slow git pushes. Again enabling `GIT_TRACE_PACKET`, I can see about 10.9 seconds between when `git-receive-pack` is started on the remote server until the server sends the first ref back. Reading through the code, I believe it's calling the same `PermissionAwareReadOnlyRefDatabase.getRefs` method without any filtering, which causes Gerrit to permission check all 500K+ refs, which is causing the 10.9 slowdown. In theory, setting `receive.hideRefs` to `refs/changes/` should speed this up, but JGit doesn't appear to respect `receive.hideRefs` (specifically, it only looks for `uploadpack.hideRefs` but then setting this causes it to behave the way `transfer.hideRefs` should).

there is no protocol v2 for push (yet)

 

Gerrit already does filter out the `refs/changes` refs before responding to the `receive-pack`, but this filtering is done AFTER all permissions checks, not before, and as a result, Gerrit is performing 500K permission checks and then discarding the results of 99.5% of them. I believe a more efficient way to handle this would be to filter out `refs/changes` before doing the permissions check. I think this could be implemented in `AsyncReceiveCommits` -- where it calls `PermissionAwareRepositoryManager.wrap` around the bare repository, a similar wrapper around the bare repository object could be created which filters out all `refs/changes` first, and then this wrapper could then be wrapped with `PermissionAwareRepositoryManager.wrap`.

Thoughts on these two changes, specifically defaulting `uploadpack.allowRefInWant` to `true` and filtering out `refs/changes` before the permission check in `AsyncReceiveCommits`? Do they seem valuable to the Gerrit project? Would you be willing to accept contributions implementing them?

Thanks!

--Joel

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/f903f86c-efaf-4cd7-9d05-e27bf9bc5486n%40googlegroups.com.

[Joel Thompson]

On Wed, Nov 20, 2024 at 4:42 AM Matthias Sohn <matthi...@gmail.com> wrote:

On Tue, Nov 19, 2024 at 8:06 AM Joel Thompson <jatho...@gmail.com> wrote:

Hi,

I manage a Gerrit server with a moderately sized repository, and Gerrit usage for this repository has become increasingly slow for our users. I have a couple ideas on how the performance can be improved (summarized in the last paragraph) and would like your feedback.

Specifically, we have about 100K total distinct changes, 500K total `refs/changes/*` refs (including the `refs/changes/*/meta` refs), and about 1.5K "other" refs (mostly branches, some tags). Doing a `get fetch` or a `git push` will always take about 11 seconds, even if the amount of data to transfer is relatively small. Both the Gerrit server and the git clients have plenty of resources (CPU/memory/etc.) and are in the same AWS region so there's low latency between them.

I believe what's happening in both cases is JGit is calling the `PermissionAwareReadOnlyRefDatabase` object's `getRefs` method, which is then iterating serially  over all 500K+ refs and doing a permission check on each and every single one of them, which is what is causing this observed slowness.

run the git command with tracing enabled [1], this should give you timing details in the error_log 

I just tried this on a git push (git fetch is performing well now that I've set uploadpack.allowRefInWant on the server), and it's not giving me detailed timing information on what's slow. I believe the reason is related to the way the trace ID is sent over -- my client connects to Gerrit, runs `git-receive-pack`, and then waits for Gerrit to send back all the refs, and only once that is done does my client send the trace ID to Gerrit The slow part is waiting for Gerrit to send over all the refs, i.e., before Gerrit starts the tracing. As you noted, there's no protocol v2 for push, and so there's no way for my client to communicate the trace ID to Gerrit before Gerrit returns all the refs.

 

[1] https://gerrit-review.googlesource.com/Documentation/user-request-tracing.html

When performing a git fetch, with `GIT_TRACE_PACKET` enabled, I can see that my client is sending a `command=ls-refs` with `ref-prefix refs/heads/` and `ref-prefix refs/tags/`, and Gerrit is responding with the 1.5K or so matching refs pretty quickly (about 150ms). My client then sends over `command=fetch` specifying the object IDs wanted via `want` arguments followed by `have` arguments, then followed by the flush-pkt. It then takes Gerrit 10.9 seconds to respond with `acknowledgments`, and then the fetch proceeds fine from there. Reading through the JGit code, I believe that this code path is calling `PermissionAwareReadOnlyRefDatabase.getRefs`.

I believe there's a very simple solution to this -- setting `uploadpack.allowRefInWant` via `etc/jgit.config`. I've implemented this in production, and now `git fetch` operations start similarly (except the server now advertises the `ref-in-want` feature), up until my client issues the `fetch` command, when it now sends `want-ref` arguments instead of `want` arguments. Gerrit then sends back the `acknowledgements` in around 360ms rather than 10.9s. By sending over only `want-ref` arguments, the `PermissionAwareReadOnlyRefDatabase` only has to permission check the individual refs requested, rather than every single ref in the database, and thus it's much, much quicker.

I think that Gerrit should set this as a default -- in the same way `receive.autogc` is set `false` upon Gerrit startup unless it's explicitly set to `true`, Gerrit could set `uploadpack.allowRefInWant` to `true` unless it's explicitly set to false. Alternatively, this could be documented in the Gerrit documentation to make users more aware of its existence.

Unfortunately, it seems there isn't as easy of a fix for the slow git pushes. Again enabling `GIT_TRACE_PACKET`, I can see about 10.9 seconds between when `git-receive-pack` is started on the remote server until the server sends the first ref back. Reading through the code, I believe it's calling the same `PermissionAwareReadOnlyRefDatabase.getRefs` method without any filtering, which causes Gerrit to permission check all 500K+ refs, which is causing the 10.9 slowdown. In theory, setting `receive.hideRefs` to `refs/changes/` should speed this up, but JGit doesn't appear to respect `receive.hideRefs` (specifically, it only looks for `uploadpack.hideRefs` but then setting this causes it to behave the way `transfer.hideRefs` should).

there is no protocol v2 for push (yet)

Understood; I'm talking about setting `receive.hideRefs` on the Gerrit server, in the etc/jgit.config file. The git-config documentation [0] explicitly describes a `receive.hideRefs` setting, and a protocol v1 server could honor that setting on a `git-receive-pack` call, right? Am I missing something here?

[0] https://git-scm.com/docs/git-config#Documentation/git-config.txt-receivehideRefs

[Joel Thompson]

Hi Dani,

Thanks for the reply!

On Tue, Nov 19, 2024 at 4:42 PM Daniele Sassoli <daniele...@gmail.com> wrote:

Hi Joel,

Thanks for your in-depth analysis.
I'll start by saying that there's people here that know more than me, but I'll give my 2 cents below.

On Tuesday, 19 November 2024 at 07:06:48 UTC Joel Thompson wrote:

Hi,

I manage a Gerrit server with a moderately sized repository, and Gerrit usage for this repository has become increasingly slow for our users. I have a couple ideas on how the performance can be improved (summarized in the last paragraph) and would like your feedback.

Specifically, we have about 100K total distinct changes, 500K total `refs/changes/*` refs (including the `refs/changes/*/meta` refs), and about 1.5K "other" refs (mostly branches, some tags). Doing a `get fetch` or a `git push` will always take about 11 seconds, even if the amount of data to transfer is relatively small. Both the Gerrit server and the git clients have plenty of resources (CPU/memory/etc.) and are in the same AWS region so there's low latency between them.

I believe what's happening in both cases is JGit is calling the `PermissionAwareReadOnlyRefDatabase` object's `getRefs` method, which is then iterating serially  over all 500K+ refs and doing a permission check on each and every single one of them, which is what is causing this observed slowness.

When performing a git fetch, with `GIT_TRACE_PACKET` enabled, I can see that my client is sending a `command=ls-refs` with `ref-prefix refs/heads/` and `ref-prefix refs/tags/`, and Gerrit is responding with the 1.5K or so matching refs pretty quickly (about 150ms). My client then sends over `command=fetch` specifying the object IDs wanted via `want` arguments followed by `have` arguments, then followed by the flush-pkt. It then takes Gerrit 10.9 seconds to respond with `acknowledgments`, and then the fetch proceeds fine from there. Reading through the JGit code, I believe that this code path is calling `PermissionAwareReadOnlyRefDatabase.getRefs`.

I believe there's a very simple solution to this -- setting `uploadpack.allowRefInWant` via `etc/jgit.config`. I've implemented this in production, and now `git fetch` operations start similarly (except the server now advertises the `ref-in-want` feature), up until my client issues the `fetch` command, when it now sends `want-ref` arguments instead of `want` arguments. Gerrit then sends back the `acknowledgements` in around 360ms rather than 10.9s. By sending over only `want-ref` arguments, the `PermissionAwareReadOnlyRefDatabase` only has to permission check the individual refs requested, rather than every single ref in the database, and thus it's much, much quicker.

I think that Gerrit should set this as a default -- in the same way `receive.autogc` is set `false` upon Gerrit startup unless it's explicitly set to `true`, Gerrit could set `uploadpack.allowRefInWant` to `true` unless it's explicitly set to false. Alternatively, this could be documented in the Gerrit documentation to make users more aware of its existence.

 

I believe there are security reasons for which these setting are disabled by default, but document contributions are always welcome.

I have thought about security considerations for `allowRefInWant` but haven't been able to think of any, unless there's some sort of security bug in Git itself -- whether I'm asking for a branch head by ref or by object ID, it should still perform the same access checks, right?

 

Have you tried setting "allowAnySHA1InWant"? I suspect it might give you an even better performance boost.

I haven't tried this, but it seems like this option would have security concerns, bypassing Gerrit's checks on object reachability?

 

Unfortunately, it seems there isn't as easy of a fix for the slow git pushes. Again enabling `GIT_TRACE_PACKET`, I can see about 10.9 seconds between when `git-receive-pack` is started on the remote server until the server sends the first ref back. Reading through the code, I believe it's calling the same `PermissionAwareReadOnlyRefDatabase.getRefs` method without any filtering, which causes Gerrit to permission check all 500K+ refs, which is causing the 10.9 slowdown. In theory, setting `receive.hideRefs` to `refs/changes/` should speed this up, but JGit doesn't appear to respect `receive.hideRefs` (specifically, it only looks for `uploadpack.hideRefs` but then setting this causes it to behave the way `transfer.hideRefs` should).

Gerrit already does filter out the `refs/changes` refs before responding to the `receive-pack`, but this filtering is done AFTER all permissions checks, not before, and as a result, Gerrit is performing 500K permission checks and then discarding the results of 99.5% of them. I believe a more efficient way to handle this would be to filter out `refs/changes` before doing the permissions check. I think this could be implemented in `AsyncReceiveCommits` -- where it calls `PermissionAwareRepositoryManager.wrap` around the bare repository, a similar wrapper around the bare repository object could be created which filters out all `refs/changes` first, and then this wrapper could then be wrapped with `PermissionAwareRepositoryManager.wrap`.

Have you experimented with the git-refs-filter plugin[1]? 
It should reduce the amount of refs advertised and hopefully speed up your operations.

Thanks for the suggestion! I haven't experimented with it; however, as the documentation notes, Git's `hideRefs` feature is the fastest, and that's really what I'd like to implement here. I also only want to filter out the refs/changes refs on `git push` (i.e., calling `git-receive-pack`); filtering out closed changes for all users would prevent being able to download old patch sets of old changes -- not terrible, but not quite what I'd like to do.

 

Thoughts on these two changes, specifically defaulting `uploadpack.allowRefInWant` to `true` and filtering out `refs/changes` before the permission check in `AsyncReceiveCommits`? Do they seem valuable to the Gerrit project? Would you be willing to accept contributions implementing them?

Thanks!

--Joel

Thanks,
Dani

[1] https://gerrit.googlesource.com/modules/git-refs-filter/ 

--

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---

You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/J5Q3RqCw3WY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/8e679205-13f2-436a-97c5-c224f0d5812bn%40googlegroups.com.