GIT clone using HTTPS protocol uses SSH queue

[Vineeth C]

I'm cloning git repo over https protocol and when I check the gerrit show-queue, it displays 'git-upload-pack' task being displayed on SSH-Interactive-Worker queue

Is it normal to see the HTTPS clone request to use the SSH interactive queue and its thread (or) could it be a misconfiguration on my gerrit setup? could someone share their view  

Clone URL -> git clone https://username@<gerrit_url>/gerrit/projxxxx 

Queue: SSH-Interactive-Worker
2b459c5f              16:16:46.720      git-upload-pack projxxxx (username)
------------------------------------------------------------------------------

Gerrit version : 3.5.6

Snip of gerrit config below

[sshd]
        listenAddress = *:29418
        backend = nio2
        threads = 180
        batchThreads = 50
        commandStartThreads = 8
        idleTimeout = 30m
        waitTimeout = 30m
        maxConnectionsPerUser = 24

[httpd]
   listenUrl = proxy-https://localhost:9080/gerrit
        registerMBeans = true
        requestLog = false
        maxThreads = 150

[Luca Milanesio]

On 21 Jul 2024, at 15:45, Vineeth C <vineet...@gmail.com> wrote:

I'm cloning git repo over https protocol and when I check the gerrit show-queue, it displays 'git-upload-pack' task being displayed on SSH-Interactive-Worker queue

Is it normal to see the HTTPS clone request to use the SSH interactive queue and its thread (or) could it be a misconfiguration on my gerrit setup? could someone share their view  

Yep, that’s expected. The SSH thread pool is just wrongly named, as it serves as executor for the Git operations.

Luca.

Clone URL -> git clone https://username@<gerrit_url>/gerrit/projxxxx 

Queue: SSH-Interactive-Worker
2b459c5f              16:16:46.720      git-upload-pack projxxxx (username)
------------------------------------------------------------------------------

Gerrit version : 3.5.6

Snip of gerrit config below

[sshd]
        listenAddress = *:29418
        backend = nio2
        threads = 180
        batchThreads = 50
        commandStartThreads = 8
        idleTimeout = 30m
        waitTimeout = 30m
        maxConnectionsPerUser = 24

[httpd]
   listenUrl = proxy-https://localhost:9080/gerrit
        registerMBeans = true
        requestLog = false
        maxThreads = 150

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f3297cdc-e80f-4390-91af-ed905d8e7e7bn%40googlegroups.com.

[Vineeth C]

Thanks Luca for the update.

a.) It means any GIT clone/push operations using both ssh & https are executed in respective SSH queues => SSH-Interactive-Worker, SSH-Batch-Worker and SshCommandStart

b.) So trying to understand the use case for separate httpd worker thread pool in gerrit config -> httpd.maxThreads as 150

c.) With gerrit 3.5.6, we started seeing too many tasks being piled up in "waiting" state on these SSH queues and trying to understand the usage and the need to reconfigure

Queue: SSH-Batch-Worker

2c4e0c18 waiting .... 14:10:02.434      git-upload-pack /repoxxx (username)

6c2ba43e waiting .... 14:10:02.481      git-upload-pack /repoxxx (username)

ecff94fd waiting .... 14:10:02.837      git-upload-pack /repoxxx (username)

0cf0e831 waiting .... 14:10:02.841      git-upload-pack /repoxxx (username)

------------------------------------------------------------------------------

  103 tasks, 50 worker threads

Queue: SshCommandStart

4cf10040 waiting .... 14:10:02.893      start (user x)

ccfc101e waiting .... 14:10:02.920      start (user x)

2c0fec34 waiting .... 14:10:02.921      start (user x)

0c122860 waiting .... 14:10:02.922      start (user x)

ec18f484 waiting .... 14:10:02.938      start (user x)

6c408446 waiting .... 14:10:02.975      start (user x)

ac2d1cac waiting .... 14:10:02.978      start (user x)

------------------------------------------------------------------------------

  20 tasks, 8 worker threads

[sshd] 

threads = 180

batchThreads = 50

[httpd]

maxThreads = 150

OS specs:

RHEL 7.9

48 core cpu 

192GB RAM

Appreciate your feedback! 

Regards,

Vineeth

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/C7998D45-EC97-4D39-A9CD-623790E459E3%40gmail.com.

[Luca Milanesio]

With 48 CPUs, it would be hard to perform 180+50 = 230 concurrent Git operations efficiently.

If you graph the system load, I’m pretty sure you’re trashing and therefore the thread pools and responsiveness is on the rubbish bin :-(

You should:

- review your HW specs

- graph your system load

- reassess your capacity (are you using HA? Can you do balancing)

- reconfigure the total number of threads (batch + interactive) to be *at most* 2x the cores.

HTH

Luca.

[vineeth]

Yes, you are right with the CPU load pattern. 

Its always > 90% and we see slowness with gerrit and SshCommandStart queue piling up

Thanks Luca for your inputs. shall review your recommendations!

Regards,

Vineeth