Some of my users suffered from low performence with push patch

Yingchun Li

unread,

Mar 11, 2024, 7:31:05 AM3/11/24

to Repo and Gerrit Discussion

Hi,

I have a gerrit server(3.8.3) hosting a big project(linux kernel), some users had a very low

speed when pushing up a patch, even a very simple patch.

And I found the "remote: counting objects" took a lot of time, like below:

"

GIT_TRACE=1 git push origin HEAD:refs/for/master
16:15:41.029637 git.c:455 trace: built-in: git push origin HEAD:refs/for/master
16:15:41.039893 run-command.c:668 trace: run_command: unset GIT_PREFIX; ssh -p 29418 172.16.10.40 'git-receive-pack '\''/linux/kernel'\'''
16:15:41.169691 run-command.c:668 trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
16:15:41.172849 git.c:455 trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 80 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 323 bytes | 323.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2)
remote: Counting objects: 1991385, done
remote: Processing changes: refs: 1, new: 1, done
remote: hint: commit 670d98d: [code-owners] skipping validation of code owner config files
remote: hint: commit 670d98d: [code-owners] code-owners functionality is disabled
remote:
remote: SUCCESS
remote:
remote: http://172.16.10.40/c/linux/kernel/+/2867 tst [NEW]
remote:
To ssh://172.16.10.40:29418/linux/kernel
* [new reference] HEAD -> refs/for/master

"

There is a very big objects number:1991385, and the most strange thing is when I copyed the whole project(in same machine), and generated a simple patch with another user, and pushing it again, it is very fast, and almost shows no "remote: counting objects xxx" message.

I did the gc periodically, and some of the configuations are :

[container]
...
heapLimit = 56g

[core]
#cache for the repo. should be the max size of one repo
packedGitLimit = 16g
packedGitWindowSize = 16k
packedGitOpenFiles = 2048

...

[gc]
startTime = Sun 02:30
interval = 3 days

and the gc_log:

"

[2024-03-10 02:58:05,988] INFO : [linux/kernel] gc config: gc.aggressive=false;
[2024-03-10 02:58:05,989] INFO : [linux/kernel] pack config: maxDeltaDepth=50, deltaSearchWindowSize=10, deltaSearchMemoryLimit=0, deltaCacheSize=52428800, deltaCacheLimit=100, compressionLevel=-1, indexVersion=2, bigFileThreshold=52428800, threads=0, reuseDeltas=true, reuseObjects=true, deltaCompress=true, buildBitmaps=true, bitmapContiguousCommitCount=100, bitmapRecentCommitCount=20000, bitmapRecentCommitSpan=100, bitmapDistantCommitSpan=5000, bitmapExcessiveBranchCount=100, bitmapInactiveBranchAge=90, searchForReuseTimeoutPT596523H14M7S, singlePack=false, minBytesForObjSizeIndex=-1
[2024-03-10 02:58:06,172] INFO : [linux/kernel] before: numberOfPackedRefs=5604, numberOfPackFiles=32, sizeOfLooseObjects=43184, numberOfLooseObjects=62, numberOfPackedObjects=11482820, numberOfBitmaps=0, numberOfLooseRefs=6, sizeOfPackedObjects=4848003783
[2024-03-10 03:14:55,690] INFO : [linux/kernel] after: numberOfPackedRefs=5609, numberOfPackFiles=2, sizeOfLooseObjects=16316, numberOfLooseObjects=19, numberOfPackedObjects=11477050, numberOfBitmaps=2091, numberOfLooseRefs=1, sizeOfPackedObjects=4838679568

"

and the related sshd_log:

"

2024-03-11T17:44:59.257+08:00] 0f3e9ad5 [SSH git-receive-pack /linux/kernel (user)] user a/1000097 git-receive-pack./linux/kernel 4ms 51749ms - 0 git/2.34.1 51000ms 51000ms 19538841112

"

the git-sizer shows:

Processing blobs: 2900232
Processing trees: 7035339
Processing commits: 1522584
Matching commits to trees: 1522584
Processing annotated tags: 4612
Processing references: 4721
| Name | Value | Level of concern |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size | | |
| * Commits | | |
| * Count | 1.52 M | *** |
| * Total size | 1.35 GiB | ***** |
| * Trees | | |
| * Count | 7.04 M | **** |
| * Total size | 19.8 GiB | ********** |
| * Total tree entries | 573 M | *********** |
| * Blobs | | |
| * Count | 2.90 M | * |
| * Total size | 116 GiB | ************ |
| | | |
| Biggest objects | | |
| * Commits | | |
| * Maximum size [1] | 72.7 KiB | * |
| * Maximum parents [2] | 66 | ****** |
| * Trees | | |
| * Maximum entries [3] | 2.59 k | ** |
| * Blobs | | |
| * Maximum size [4] | 22.8 MiB | ** |
| | | |
| Biggest checkouts | | |
| * Number of directories [5] | 5.19 k | ** |
| * Maximum path depth [6] | 13 | * |
| * Maximum path length [7] | 134 B | * |
| * Number of files [5] | 80.3 k | * |
| * Total size of files [5] | 1.24 GiB | * |

[1] 91cc53b0c78596a73fa708cceb7313e7168bb146
[2] 2cde51fbd0f310c8a2c5f977e665c0ac3945b46d
[3] d17756098208af5d31186b054f985d9ac19e465e (refs/heads/master:arch/arm/boot/dts)
[4] 0691e328d0f023b16bffee67d7db9771fc7a09c1 (refs/heads/master:drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_3_2_0_sh_mask.h)
[5] 216667c45120a199c4a2a4c54bc14eab4d8fcf03 (refs/heads/master^{tree})
[6] 78a269635e76ed927e17d7883f2d90313570fdbc (dae09011115133666e47c35673c0564b0a702db7^{tree})
[7] 07db354d8abc4b25fb980f8443848f4cb11dc4c2 (refs/remotes/origin/linux-4.14.y^{tree})

Please give some light, thanks.

Br,

Yingchun

Matthias Sohn

unread,

Mar 11, 2024, 8:13:47 AM3/11/24

to Yingchun Li, Repo and Gerrit Discussion

Maybe these users need to run git gc on their clone.

They can run git count-objects -vH on their clone to find out if there's an excessive number of loose objects or pack files

which may cause slowness.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2b34970d-cd93-4a19-a3cd-57abb4cea9e6n%40googlegroups.com.

Yingchun Li

unread,

Mar 11, 2024, 8:43:09 AM3/11/24

to Repo and Gerrit Discussion

Thank you, seems no extra info.(and also I copyed the repo to my own location,and made a patch and send again, it is very fast),seems only

related the user's specific environment , also I have checked the git config, also no meaningful info.

git count-objects -vH

"
count: 9
size: 96.00 KiB
in-pack: 11462446
packs: 1
size-pack: 4.79 GiB
prune-packable: 0
garbage: 0

size-garbage: 0 bytes

"

git config -l

''

color.ui=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true

Matthias Sohn

unread,

Mar 11, 2024, 9:22:24 AM3/11/24

to Yingchun Li, Repo and Gerrit Discussion

This looks good. Maybe they use an old git version ?

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a9769309-a146-4dda-8e55-a476b58bd1b7n%40googlegroups.com.

Yingchun Li

unread,

Mar 12, 2024, 2:41:26 AM3/12/24

to Repo and Gerrit Discussion

git version is 2.34.1, not so old.

Now I am sure it is related with the user group, if I (the admin) was removed from the adminiatrator group, then

pushed a patch, it is also very slow.the difference is the "remote: couting objects" step, normal users have a very

big number(as the top, about 1991453), but as admin, this step is very fast, I even can not see the log.

Kai Lei

unread,

Mar 12, 2024, 3:38:42 AM3/12/24

to Yingchun Li, Repo and Gerrit Discussion

Can you try to grant read access under refs/* to the user then try again?

On Mar 11, 2024, at 23:41, Yingchun Li <sword.l...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/9865e286-bd88-4fd1-bc6b-e2629e7f5f17n%40googlegroups.com.

Kai Lei

unread,

Mar 12, 2024, 3:42:11 AM3/12/24

to Yingchun Li, Repo and Gerrit Discussion

I cans similar issue but with fetch, seems it cause by checking if user has access on refs, it grants read to refs/* , issue is gone

On Mar 12, 2024, at 00:38, Kai Lei <tiger...@gmail.com> wrote:

Yingchun Li

unread,

Mar 12, 2024, 5:01:25 AM3/12/24

to Repo and Gerrit Discussion

On Tuesday, March 12, 2024 at 3:42:11 PM UTC+8 Kai Lei wrote:

I cans similar issue but with fetch, seems it cause by checking if user has access on refs, it grants read to refs/* , issue is gone

You are right, after give all read permission under refs/*, now git push very fast. Thanks a lot.

I tried to narrow down which directory under refs/ impact the behave, and give read permission to all these

dirs like refs/for/*, refs/changes/* ,refs/notes/*, refs/meta/, refs/users/*, but with no lucky,

only the refs/* works.

So don't know why, if the permission is not right, I perfer it give me a more direct error, but

now just the speed is very slow.

Yingchun Li

unread,

Mar 16, 2024, 1:41:12 AM3/16/24

to Repo and Gerrit Discussion

Still confused,

Thats the access in my All-projects(Here I only grant the read asscess to admin, and expect

the sub project to grant to specific groups.)

[access "refs/*"]
read = group Administrators
[access "refs/for/*"]
addPatchSet = group Registered Users
label-Owners-Override = +0..+1 group sdk_developer
[access "refs/for/refs/*"]
push = group Registered Users
pushMerge = group Registered Users label-Owners-Override = +0..+1 group Administrators

and that's access in my sub-project:

[access "refs/*"]
owner = group software_admin
read = group software_developers exclusiveGroupPermissions = read

Now if I dont set exclusive read permisson to software_devleopers,

the push speed is still slow.

My problems is there must be something wrong on my settings, but
I expect there is access error, not a slow perfomance.

Reply all

Reply to author

Forward