Why most of my git-upload-pack tasks are waiting in show-queue command?

528 views
Skip to first unread message

Yinyin Xiao

unread,
Dec 2, 2016, 7:10:22 AM12/2/16
to Repo and Gerrit Discussion
Hi,

I use 'repo sync -j8' to sync code from my slave gerrit, but the show-queue command show that only 2 concurrent tasks are processed, the others just waiting, then the sync is not fast enough.
Task     State        StartTime         Command
------------------------------------------------------------------------------
8da15f4f              18:48:57.885      git-upload-pack '/platform/vendor/xxx/proprietary' (xxx)
82bfeed9              18:49:21.363      git-upload-pack '/platform/packages/apps/SpareParts' (xxx)
02d3fe30 waiting .... 18:49:22.845      git-upload-pack '/platform/external/strace' (xxx)
62ff5a9c waiting .... 18:49:24.930      git-upload-pack '/platform/vendor/nxp/opensource/packages/apps/Nfc' (xxx)
e2ea6add waiting .... 18:49:26.713      git-upload-pack '/device/xxx/xxx' (xxx)
62167ae6 waiting .... 18:49:27.925      git-upload-pack '/device/xxx/xxx' (xxx)
e2010a9f waiting .... 18:49:29.000      git-upload-pack '/platform/packages/apps/Launcher2' (xxx)
62519aab waiting .... 18:49:30.509      git-upload-pack '/platform/frameworks/multidex' (xxx)

If I do the same to sync code from my master gerrit, all 8 task are processed concurrently.
my slave gerrit PC have 32 processers and 64G RAM, so the HW should be good enought, but why it have the limited concurrent tasks?
here is my gerrit.config on slave side:
[gerrit]
        basePath = git
        serverId = 7x378640-59b3-470d-aa58-f4b3462bxc91
        canonicalWebUrl = http://x.x.x.x
[database]
        type = mysql
        hostname = localhost
        database = gerritdb
        username = gerrit
[auth]
        type = HTTP
[sendemail]
        smtpServer = localhost
[container]
        user = gerrit
        javaHome = /usr/lib/jvm/java-8-openjdk-amd64/jre
[sshd]
        listenAddress = *:29418
[httpd]
        listenUrl = proxy-http://*:8081/
[cache]
        directory = cache

----
Yinyin

Saša Živkov

unread,
Dec 2, 2016, 8:48:10 AM12/2/16
to Yinyin Xiao, Repo and Gerrit Discussion
As documented in [1], "waiting ..." means that task is ready to execute but is
waiting for an idle thread in its associated thread pool.
So you need to increase a thread pool.
In this case you are interested into the sshd.threads and sshd.batchThreads [2].

The show-queue command (for a newer Gerrit version) also provides the "-q" option
which will print the tasks grouped by the queue name. This should help you
to figure out which queue you should look at and which thread pool is processing that queue.
You will need to dig a bit in Gerrit's source code to determine the thread pool settings
but that should be too hard. Just grep the queue name from the output of the "-q"
option and it is very likely that the related thread pool size setting is in the same file.


An example related to exactly your question here:

$ ssh ... gerrit show-queue -w -q
Task     State        StartTime         Command
------------------------------------------------------------------------------
...
Queue: SSH-Interactive-Worker
8de464a8              Dec-01 14:22      git-upload-pack '...'
...



$ git grep 'SSH-Interactive'
...
gerrit-sshd/src/main/java/com/google/gerrit/sshd/CommandExecutorQueueProvider.java:        "SSH-Interactive-Worker");

$ vim gerrit-sshd/src/main/java/com/google/gerrit/sshd/CommandExecutorQueueProvider.java
...
    final int cores = Runtime.getRuntime().availableProcessors();
    poolSize = config.getInt("sshd", "threads", 3 * cores / 2);
    batchThreads = config.getInt("sshd", "batchThreads", cores == 1 ? 1 : 2);
    if (batchThreads > poolSize) {
      poolSize += batchThreads;
    }
    int interactiveThreads = Math.max(1, poolSize - batchThreads);
    interactiveExecutor = queues.createQueue(interactiveThreads,
        "SSH-Interactive-Worker");
...

After that just read the documentation for these two settings [2]

:-)



--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matthias Sohn

unread,
Dec 2, 2016, 8:57:53 AM12/2/16
to Yinyin Xiao, Repo and Gerrit Discussion
On Fri, Dec 2, 2016 at 12:19 PM, Yinyin Xiao <yin8...@gmail.com> wrote:

Yinyin Xiao

unread,
Dec 6, 2016, 1:08:54 AM12/6/16
to Repo and Gerrit Discussion, yin8...@gmail.com
Many thanks to @zivkov, @Matthias

My issue is caused by the default value changed for different Gerrit versions:
--my master Gerrit (version 2.11)  have the default value  sshd.batchThreads=0 (reuse the queue defined by sshd.threads=1.5x the number of CPUs available to the JVM.)
--my slave Gerrit (version 2.13.3) have the default value sshd.batchThreads=2 (By default is 1 on single core node, 2 otherwise.)

As I add "sshd.batchThreads=32" in my slave Gerrit config, it works! 


BR/Yinyin
在 2016年12月2日星期五 UTC+8下午9:57:53，Matthias Sohn写道：
Reply all
Reply to author
Forward
0 new messages