Repo sync is excruciatingly slow even for 15 ssh connections
praveenm
Hi,
We are facing performance issue in our Gerrit 2.5.2. and having trouble figuring out where the bottleneck is.
Repo
sync is awfully slow when more than 15 ssh connections are established to the server.
ex : netstat -n | grep ':2**' | grep ESTABLISHED | wc –l
15
Server configuration:
Processor : 10 core @ 2.40GHz X 2 processor
RAM: 157 GB
Network : 8 ports bonded with bond-mode balance-rr (1Gbps x 8)
Git version : 1.9.0
OS : Ubuntu 10.04.4
openssh-server : 1:5.3p1-3ubuntu7.1
------------------------------------------------------------------------
gerrit.config settings
[container]
javaHome = /usr/lib/jvm/java-6-openjdk/jre
javaOptions = -XX:+DoEscapeAnalysis -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -Xms100g -Xmx130g -XX:NewRatio=3
[sshd]
listenAddress = *:****
threads = 150
batchThreads = 10
idleTimeout = 5m
commandStartThreads = 5
streamThreads = 10
[core]
streamFileThreshold = 3g
packedGitLimit = 50g
packedGitWindowSize = 64k
packedGitOpenFiles = 4096
deltaBaseCacheLimit = 1g
vmstat –s output
165378288 K total memory
146797216 K used memory
134967088 K active memory
7264932 K inactive memory
18581076 K free memory
180868 K buffer memory
3162840 K swap cache
97655800 K total swap
597232 K used swap
97058568 K free swap
114589545 non-nice user cpu ticks
97 nice user cpu ticks
7923063 system cpu ticks
1299816118 idle cpu ticks
6090046 IO-wait cpu ticks
30915 IRQ cpu ticks
4066078 softirq cpu ticks
0 stolen cpu ticks
8520574387 pages paged in
2577258648 pages paged out
3654330 pages swapped in
3253832 pages swapped out
254349785 interrupts
2333283560 CPU context switches
1403858828 boot time
3126296 forks
--------------------------------------------------------------------------------------------------------------------------------------------
We have 3000 + repos, & regularly running git gc use repo sync -j8 -q -c option
Am I missing something here?
Please help us in improving performance of the gerrit server.
Thanks
Doug Kelly
...
praveenm
now we have added
poolMaxIdle = 32
poolLimit = 128
poolMinIdle = 16
...