Repo sync become slow in slave

ashspencil

unread,

Feb 16, 2022, 11:35:53 PM2/16/22

to Repo and Gerrit Discussion

Hi,

We have one master and two slaves in different region, the network speed between both of them is 100Mbps(for replication).
In office network is 1Gbps.

Region A: Master and A slaves
Region B: B slaves

The version is 2.15.18 and we are planning and running test for upgrade to version 3.5.0.1.

In last half year, we notice the B slave become slower and slower when repo sync and A slave wasn't. They were same fast in begin.
For example, from B slave
```
git-upload-pack./kernel/aosp/platform/development 1ms 32507ms (our man repo sync in Region B)
```
-
from A slave
```
git-upload-pack./kernel/aosp/platform/development 1ms 430ms (our man repo sync in Region A)
```

When run repo sync with same manifest, B slave always takes many times longer than A slave.

Both of two slaves have same hardware and setting

CPU: 16 Core with Xeon(R) Gold 5218R CPU @ 2.10GHz
RAM: 48G
OS: Centos 6.5
JAVA: jre1.8.0_162
DB: mysql 5.1.71

Here is gerrit.config (A and B slave with same config)

==========
[database]
type = mysql
hostname = xxx
database = reviewdb
username = gerrit
[auth]
type = LDAP
gitBasicAuthPolicy = HTTP_LDAP
[container]
user = gerrit
javaHome = /SDLC/jre1.8.0_162
slave = true
heapLimit = 42g
[core]
packedGitLimit = 4g
[sshd]
threads = 64
batchThreads = 16
listenAddress = *:29418
maxConnectionsPerUser = 0
tcpKeepAlive = true
idleTimeout = 1d
waitTimeout = 1d
[cache]
directory = /dev/shm/gerrit-cache
[pack]
deltacompression = true
threads = 0
[gc]
aggressive = true
startTime = 3:0
interval = 1d
==========

Here is show-caches status in regular

A:
==========
Gerrit Code Review 2.15.18-3-g71dfa727e3 now 15:52:08 CST
uptime 23 days 7 hrs

Name |Entries | AvgGet |Hit Ratio|
| Mem Disk Space| |Mem Disk|
--------------------------------+---------------------+---------+---------+
accounts | 16 | 7.3ms | 99% |
adv_bases | | | |
change_notes | | | |
changeid_project | | | |
groups | | | |
groups_bymember | | | |
groups_byname | | | |
groups_bysubgroup | | | |
groups_byuuid | 83 | 6.1ms | 98% |
groups_external | | | |
groups_subgroups | | | |
ldap_group_existence | | | |
ldap_groups | 1 | | |
ldap_groups_byinclude | | | |
ldap_usernames | | | |
permission_sort | 1024 | | 80% |
plugin_resources | | | |
project_list | | 1.7s | 0% |
projects | 1024 | 9.0ms | 98% |
sshkeys | 6 | 1.4ms | 89% |
static_content | | | |
D change_kind | 0.00k| | |
D conflicts | 0.00k| | |
D diff | 0.00k| | |
D diff_intraline | 0.00k| | |
D diff_summary | 0.00k| | |
D git_tags | 787 787 7.88m| | 74% 100%|
D mergeability | 0.00k| | |
D oauth_tokens | 0.00k| | |
D web_sessions | 0.00k| | |

SSH: 9 users, oldest session started 0 ms ago
Tasks: 8 total = 7 running + 1 ready + 0 sleeping
Mem: 23.38g total = 7.21g used + 13.16g free + 3.01g buffers
37.33g max
128 open files
Threads: 16 CPUs available, 130 threads
==========

B:
==========
Gerrit Code Review 2.15.18-3-g71dfa727e3 now 15:53:10 CST
uptime 4 days 11 hrs

Name |Entries | AvgGet |Hit Ratio|
| Mem Disk Space| |Mem Disk|
--------------------------------+---------------------+---------+---------+
accounts | 23 | 8.3ms | 99% |
adv_bases | | | |
change_notes | | | |
changeid_project | | | |
groups | | | |
groups_bymember | | | |
groups_byname | | | |
groups_bysubgroup | | | |
groups_byuuid | 65 | 883.5ms | 56% |
groups_external | | | |
groups_subgroups | | | |
ldap_group_existence | | | |
ldap_groups | 1 | | |
ldap_groups_byinclude | | | |
ldap_usernames | | | |
permission_sort | 1024 | | 85% |
plugin_resources | | | |
project_list | 1 | 349.8ms | 0% |
projects | 1024 | 6.1ms | 99% |
sshkeys | 4 | 911.4us | 1% |
static_content | | | |
D change_kind | 0.00k| | |
D conflicts | 0.00k| | |
D diff | 0.00k| | |
D diff_intraline | 0.00k| | |
D diff_summary | 0.00k| | |
D git_tags | 914 1133 9.40m| | 82% 100%|
D mergeability | 0.00k| | |
D oauth_tokens | 0.00k| | |
D web_sessions | 0.00k| | |

SSH: 8 users, oldest session started 0 ms ago
Tasks: 36 total = 35 running + 1 ready + 0 sleeping
Mem: 20.84g total = 9.63g used + 7.21g free + 4.00g buffers
37.33g max
29 open files

Threads: 16 CPUs available, 125 threads

We have tried adjust parameter such as core.packedGitWindowSize、pack.threads ...and etc, also try to run flush-caches in crontab every few hours, but still can not reslove it.
Does anyone has some suggest for check this situation? Thanks a lot!

Sven Selberg

unread,

Feb 17, 2022, 2:06:54 AM2/17/22

to Repo and Gerrit Discussion

I'd check if git gc actually is performed successfully on Replica B.
Check the gc-log and run "git fsck" in some active repositories.

You can also check creation date of the pack-files in said repositories.

Matthias Sohn

unread,

Feb 17, 2022, 6:02:43 AM2/17/22

to Sven Selberg, Repo and Gerrit Discussion

Run

$ git count-objects -vH

on some busy repositories (also check All-Users).

If the count of loose objects / pack files is large, check how you run git gc on the replica and if it really runs on a regular basis.

-Matthias

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/62d87140-2876-4af6-91b7-8d6da79bd397n%40googlegroups.com.

Reply all

Reply to author

Forward