Gerrit 3.8 push slow with big files, always stuck in writing objects

[張宇誠]

Hi Sir,

I have stucking with git push problem, when I git push about 6.8 GB commit to Gerrit，writing objects always fail from Mb/s to Kb/s 

Here is the error:

$  git push "ssh://x...@xxx.com:29418/test" HEAD:release/common/
Enumerating objects: 6434007, done.
Counting objects: 100% (6434007/6434007), done.
Delta compression using up to 16 threads
Compressing objects: 100% (1735617/1735617), done.
Writing objects:  42% (2760422/6434007), 1.63 GiB | 132.00 KiB/s

Git Server with gerrit: 8 core、32 GB RAM

And below is my gerrit.config:

[gerrit]
        basePath = /var/gerrit/git
        serverId = xxx
        canonicalWebUrl = xx
[index]
        type = LUCENE
[auth]
        type = LDAP
        gitBasicAuthPolicy = HTTP_LDAP
[receive]
        enableSignedPush = false
        enableSignedPushMerge = false
        requireSignedPush = false
        requireSignedPushMerge = false
        maxBatchCommits = 1000000
        maxObjectSizeLimit = 10g

[container]
        user = gerrit
        javaHome = /usr/lib/jvm/java-11-openjdk-amd64
        heapLimit = 28g 
        javaOptions = "-Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance"
        javaOptions = "-Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance"
[sshd]
        threads = 32
        batchThreads = 8
        listenAddress = *:29418
        waitTimeout = 2h
        idleTimeout = 2h
        rekeyTimeLimit = 0
        enableCompression = true
        TCPKeepAlive = yes
[httpd]
        listenURL = proxy-https://*:8080/gerrit/

[cache]
        directory = /var/gerrit/gerrit_cache
[cache "sshkeys"]
        maxAge = 10m

[pack]
       deltacompression = true
       threads = 0
[download]
   command = checkout
   command = cherry_pick
   command = pull
   command = format_patch
   scheme = ssh
   scheme = repo_download

[gitweb]
        type = custom
        linkname = source
        url = /gerrit/plugins/gitiles/
        roottree = ${project}
        revision = ${project}/+/${commit}
        project = ${project}
        branch = ${project}/+/${branch}
        filehistory = ${project}/+log/${branch}/${file}
        file = ${project}+${commit}/${file}
[commitmessage]
        maxSubjectLength = 1024
        maxLineLength = 1024
[gc]
        aggressive = true
        startTime = 03:00
        interval = 1d

[cache "diff"]
        timeout = 200ms
[lfs]
        plugin = lfs

[core]
        packedGitOpenFiles = 32768
        packedGitLimit = 10g
        packedGitWindowSize = 16k
        deltaBaseCacheLimit = 1g
        streamFileThreshold = 20g

Can anyone help me to solve this problem or how to tune the gerrit.config let me to push the code to gerrit server ? Really thanks for help. 

[Matthias Sohn]

On Wed, Apr 24, 2024 at 4:24 AM 張宇誠 <aa98...@gmail.com> wrote:

Hi Sir,

I have stucking with git push problem, when I git push about 6.8 GB commit to Gerrit，writing objects always fail from Mb/s to Kb/s 

Why are you trying to push such a huge commit ?
Why is it so large ?
Does it contain large binary files ?

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f7d64c05-aec8-483c-9a33-4d2fd7848b8fn%40googlegroups.com.

[張宇誠]

Hi, Matthias

Thanks for your reply, 

Why is it so large ? => This project is AOSP (Android Open Source Project), I can't separate to small project

Does it contain large binary files ? => Correct

So I want know how can I tune my gerrit.config or git.config for me to push my code in normal speed ? Really thanks for your reply.

Matthias Sohn 在 2024年4月25日 星期四凌晨3:44:33 [UTC+8] 的信中寫道：

[張宇誠]

Hi, Matthias

Is there any difference between our usage and the actual official recommended usage?
Our repo will only get bigger and bigger in the future. Should we refer to the official recommendations to make adjustments? Thank you.

Thank you for the official advice. Your advice is very valuable.

張宇誠 在 2024年4月25日 星期四上午8:52:27 [UTC+8] 的信中寫道：

[Nasser Grainawi]

On Thu, Apr 25, 2024 at 12:06 AM 張宇誠 <aa98...@gmail.com> wrote:

Hi, Matthias

Is there any difference between our usage and the actual official recommended usage?
Our repo will only get bigger and bigger in the future. Should we refer to the official recommendations to make adjustments? Thank you.

Thank you for the official advice. Your advice is very valuable.

Please avoid top posting on this list, we prefer interleaved posting which is easier to follow in long mail threads

https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

I think it's possible for Gerrit to handle repos with commits like that (though you didn't mention how you measured the 6.8GB, so maybe when you say "commit" it's not what we're thinking). As a few ideas for next steps based on your gerrit.config below (I recommend testing all of these in a non-production environment first):

1. I would see if you have a way to add more RAM to the host and then increase the heapLimit above 28g. I wouldn't increase the heapLimit if you only have 32g of RAM.
2. You can capture 'jstack' output a few times (wait 10-15 seconds between captures) from the Gerrit process during the push when you see it slow down and share the thread stacks for the ReceiveCommits thread as a reply here. Someone on the list might be able to help more based on that.
3. Your values in the 'core' section might need adjusting. The deltaBaseCacheLimit and streamFileThreshold values are much higher than the default and your packedGitLimit might be too small to support a repository with commits as large as you seem to have.
   
4. See if adding other java options recommended on this list in the past will help (i.e. -XX:+UseParallelOldGC -XX:ParallelGCThreads=<some number>)
5. It could be that other activity on the server during this push is competing for resources. To isolate that as the issue, you could try this same push on a backup non-production environment and see if it succeeds.

Other suggestions for the config that aren't likely to help this exact problem, but would be good to do anyway:

1. Increase the maxAge you have set for your sshkeys cache. There's no reason to have such a short time there
2. I don't think you want sshd.compression enabled since that rarely helps
3. The "pack" section in your $site_dir/etc/gerrit.config isn't doing anything. Nothing in Gerrit reads those values from that file. You could add those values to an individual <projectname>.git/config or to the $site_dir/etc/jgit.config if you really wanted them to apply globally to all Gerrit operations.

Nasser 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/e2297a74-3828-4ec6-9226-c6e809e6dbd7n%40googlegroups.com.

[張宇誠]

share the thread stacks for the ReceiveCommits thread as a reply here. => I have changing my gerrit.config, but seems there is no help with the situation.

[index]
        type = LUCENE
[auth]
        type = LDAP
        gitBasicAuthPolicy = HTTP_LDAP
[receive]
        enableSignedPush = false
        enableSignedPushMerge = false
        requireSignedPush = false
        requireSignedPushMerge = false
        maxBatchCommits = 1000000
        maxObjectSizeLimit = 10g
[container]
        user = gerrit
        javaHome = /usr/lib/jvm/java-11-openjdk-amd64

        heapLimit = 50g

        javaOptions = "-Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance"
        javaOptions = "-Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance"

        javaOptions = "-XX:+UseParallelOldGC -XX:ParallelGCThreads=16"
[sshd]
        threads = 128
        batchThreads = 32
        listenAddress = *:29418
        idleTimeout = 1d
        waitTimeout = 1d
        rekeyTimeLimit = 0
        enableCompression = false
        TCPKeepAlive = yes
[http]
        postBuffer = 524288000
[cache]
        directory = /var/gerrit/gerrit_cache
[cache "sshkeys"]
        maxAge = 4h

[pack]
       deltacompression = true
       threads = 0
[download]
   command = checkout
   command = cherry_pick
   command = pull
   command = format_patch
   scheme = ssh
   scheme = repo_download

[gc]
        aggressive = true
        startTime = 03:00
        interval = 1d

[lfs]
        plugin = lfs

[core]
        packedGitOpenFiles = 32768
        packedGitLimit = 10g
        packedGitWindowSize = 16k
        deltaBaseCacheLimit = 1g

        streamFileThreshold = 4096m

Attachment is my jstack output, if there is any idea for gerrit.config, please tell me, and really thanks for your kind assistance.

Best Regards,

Andy

Nasser Grainawi 在 2024年4月25日 星期四晚上10:55:18 [UTC+8] 的信中寫道：

[Matthias Sohn]

The work related to your push is happening in the receive-pack threads.

In the first thread dump it's the thread

"SSH git-receive-pack /git/DHC_SDK/android14/image_file/dailybuild/hifi-fw (devops_jenkins)"

In the second it's

"SSH git-receive-pack /git/DHC_SDK/AN14/phoenix/platform/frameworks/base (yunchi_hsieh)"

which is for a different repository. Don't know which one is your request.
As long as runnable receive-pack requests related to your push are moving it should make progress.
You can check this by taking several thread dumps and comparing the respective thread's stack traces.

 

In the first thread dump there is one more runnable request which is doing git gc scheduled in gerrit which can also

eat considerable resources especially if it is run in the same Java process.

From the second thread dump I can see that you are using the (default) Java implementation of SHA1.

Using the native implementation in the JDK can buy you some performance. Though this implementation

doesn't have protection against (the very unlikely) SHA1 collisions.
If you want to try that set this in $gerrit_site/jgit.config:

[core]
	sha1Implementation = jdkNative

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/fc1fc6cb-9105-46b2-ac01-a1f21f398e01n%40googlegroups.com.

[卞姗姗]

I encountered a similar issue where with version 2.14, the git push operation worked fine at around 30MiB/s, but after upgrading to version 3.5.6, for the same repository, performing an identical git push, the speed initially jumps to 50MiB, but then quickly drops to a very slow speed of only tens of KiB. 

I am using the same server for testing, but only the Gerrit version has been upgraded. The server remains the same, and the related configurations for Gerrit sshd and core, such as those for pushing, are kept consistent. I have tested this on several Gerrit instances, and the issue occurs consistently and reproducibly. 

[Luca Milanesio]

Hi 卞姗姗,

On 5 May 2024, at 09:54, 卞姗姗 <shansha...@gmail.com> wrote:

I encountered a similar issue where with version 2.14, the git push operation worked fine at around 30MiB/s, but after upgrading to version 3.5.6, for the same repository, performing an identical git push, the speed initially jumps to 50MiB, but then quickly drops to a very slow speed of only tens of KiB. 

You cannot really have the *same repository* on v2.14 and v3.5.6: the upgrade process adds a lot of extra Git objects and refs to it.

I am not surprised you have a different performance, which is expected.

Did you watch my presentation on how to manage large mono-repos? I was describing in details the characteristics of a repository and how they influence the performance of the Git operations.

See my talk recording at:

https://youtu.be/eANday2TsOk

HTH

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/9e8a53d7-f8b3-4175-b37a-6ff94f51933an%40googlegroups.com.

[卞姗姗]

Thank you for your reply. I will go check out your presentation  

[張宇誠]

Hi All,

Updated situation. After I upgrade gerrit version fron 3.8.2 to 3.8.5, the git push performance error is fixed, thanks for everyone kindness help.

卞姗姗 在 2024年5月5日 星期日下午5:34:01 [UTC+8] 的信中寫道：

[Matthias Sohn]

On Mon, May 6, 2024 at 5:07 AM 張宇誠 <aa98...@gmail.com> wrote:

Hi All,

Updated situation. After I upgrade gerrit version fron 3.8.2 to 3.8.5, the git push performance error is fixed, thanks for everyone kindness help.

Between 3.8.2 and 3.8.5 jgit was updated from v6.5.0.202303070854-r-30-g5ae8d28fa to v6.9.0.202403050737-r-23-gc0b415fb0.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/baa1c414-18a2-43fe-9162-392e5fcfce87n%40googlegroups.com.

[卞姗姗]

Upgrading to 3.8.5 also solved my problem. Thank you all.  

[卞姗姗]

hi Mattias,

i have  tested Gerrit 3.8.4 and found that push operations were very fast, but in Gerrit 3.8.3, push operations were slow. I cherry-picked the commit 1a2b7ff50f14c767943f795f1fd56b399d8f5174 from Gerrit 3.8.4 to gerrit 3.8.3, which upgrades the JGit version, and updated the module/jgit repository to acf21c0bc, However, after compiling the WAR package locally, I still found push operations to be slow.

Are there any other changes in the commits between Gerrit 3.8.3 and 3.8.4 that might affect push speed? 

I am looking forward to your response. 

[卞姗姗]

I kept the jgit version of Gerrit 3.8.3 unchanged, without upgrading jgit, and only upgraded the sshd version, specifically from 2.9.2 to 2.12.0. The issue of slow push speed using the SSH protocol on Gerrit 3.8.3 was resolved, so I suspect it was a problem with the SSH version