Gerrit server HTTP sessions (web-sessions) disconnections

292 views
Skip to first unread message

Bar Elhanati

unread,
May 16, 2023, 3:01:07 AM5/16/23
to Repo and Gerrit Discussion
Hi,
Me and my team are using Gerrit as code management platform.
We are having HTTP sessions disconnections. Those disconnections appear (mostly) when a user clicks on a Gerrit links  URL (from mail / chat) and it asks him to refresh credentials. 
The issue also appears when a user is on on a different page then landing page for 5~15 mins and then it asks him to refresh credentials.
I would like to solve this issue as soon as possible.
For some other users this issue does not exist and for some others it does. I  can't put my finger on some pattern.

Some Gerrit's data:
We are using ldap for connecting.
We tried to play with httpd.idleTimeout and nothing happened
We have raised cache.web_sessions to 8192 

config file:
[gerrit]
        basePath = <path>
        canonicalWebUrl = <url>
        serverId = <id>
[auth]
        type = LDAP
        gitBasicAuthPolicy = HTTP_LDAP

[ldap]
        server = <ldap>
        sslVerify = false
        username = <ldap>
        password = <ldap>
        accountBase = <ldap>
        accountPattern = <ldap>
        groupBase = <ldap>
        groupPattern = <ldap>
        accountMemberField = <ldap>
        accountFullName = <ldap>
        accountSshUserName = <ldap>

[sendemail]
        smtpServer = <mail>
[container]
        user = <user> 
        javaOptions = <some_jaava_options>

[httpd]
        listenUrl = https://*:8080/
        sslKeyStore = <key>
        sslKeyPassword = <pass>
        idleTimeout = 30m
        maxThreads = 512
        gracefulStopTimeout = 1m
[cache]
        directory = cache
[cache "web_sessions"]
        memoryLimit = 8192
        maxAge = 1w
[sshd]
        threads = 500
        batchThreads = 100
        maxConnectionsPerUser = 750
        listenAddress = *:29418
        waitTimeout = 30m
        #commandStartThreads = 5

        gracefulStopTimeout = 1m
        idleTimeout = 3600
[core]
        packedGitOpenFiles = 4096
        packedGitLimit = 2g
        packedGitWindowSize = 2m

[uploadpack]
        allowTipSha1InWant = true
        allowReachableSHA1InWant = true
        allowAnySHA1InWant = true


[index]
       type = LUCENE
[receive]
       enableSignedPush = false
       timeout = 6m

[gc]
       startTime = 02:00
       interval = 1 day
       aggressive = false
[plugins]
       allowRemoteAdmin = true
       allowDeletionOfReposWithTags = true


Thank you for your help,
Bar Elhanati

Bar Elhanati

unread,
May 16, 2023, 3:13:35 AM5/16/23
to Repo and Gerrit Discussion
Hi, I have forgot to mention that my Gerrit server is running on Docker container.

ב-יום שלישי, 16 במאי 2023 בשעה 10:01:07 UTC+3, Bar Elhanati כתב/ה:

Matthias Sohn

unread,
May 16, 2023, 3:43:33 AM5/16/23
to Bar Elhanati, Repo and Gerrit Discussion
On Tue, May 16, 2023 at 9:01 AM Bar Elhanati <elhana...@gmail.com> wrote:
Hi,
Me and my team are using Gerrit as code management platform.
We are having HTTP sessions disconnections. Those disconnections appear (mostly) when a user clicks on a Gerrit links  URL (from mail / chat) and it asks him to refresh credentials. 
The issue also appears when a user is on on a different page then landing page for 5~15 mins and then it asks him to refresh credentials.
I would like to solve this issue as soon as possible.
For some other users this issue does not exist and for some others it does. I  can't put my finger on some pattern.

After successful authentication the UI returns a session cookie to the browser. 

Some Gerrit's data:

Which Gerrit version do you run ?
Which heap size do you use ?
 
[httpd]
        listenUrl = https://*:8080/
        sslKeyStore = <key>
        sslKeyPassword = <pass>
        idleTimeout = 30m
        maxThreads = 512
        gracefulStopTimeout = 1m
[cache]
        directory = cache
[cache "web_sessions"]
        memoryLimit = 8192

try to increase this limit and set diskLimit to persist the cache
 
        maxAge = 1w
[sshd]
        threads = 500
        batchThreads = 100

How many cores does your server have ?
As a rule of thumb you need 0.5-1 core per concurrent git command (typically dominated by fetch commands).
 
        maxConnectionsPerUser = 750

This means a single misbehaving user can occupy all threads processing git commands.
 
        listenAddress = *:29418
        waitTimeout = 30m
        #commandStartThreads = 5

        gracefulStopTimeout = 1m
        idleTimeout = 3600
[core]
        packedGitOpenFiles = 4096
        packedGitLimit = 2g
        packedGitWindowSize = 2m

[uploadpack]
        allowTipSha1InWant = true
        allowReachableSHA1InWant = true
        allowAnySHA1InWant = true


[index]
       type = LUCENE
[receive]
       enableSignedPush = false
       timeout = 6m

[gc]
       startTime = 02:00
       interval = 1 day
       aggressive = false

On a busy server serving large repos you better schedule gc to run in a separate process
 
[plugins]
       allowRemoteAdmin = true
       allowDeletionOfReposWithTags = true


Thank you for your help,
Bar Elhanati

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/3772d049-2945-4385-b8d4-eda734aa8969n%40googlegroups.com.

Bar Elhanati

unread,
May 16, 2023, 4:14:59 AM5/16/23
to Repo and Gerrit Discussion
Hi,
Thank you for your quick response

My Gerrit server has 96 cores.
and yea, the server is busy and serving large repos. what do you mean by increasing the GC? can you please demonstrate?
??Another thing, SSH  maxConnectionsPerUser affect HTTP web sessions (web UI connections)
What is the value that you recommend for this parameter? 
ב-יום שלישי, 16 במאי 2023 בשעה 10:43:33 UTC+3, Matthias Sohn כתב/ה:

Matthias Sohn

unread,
May 16, 2023, 4:50:18 AM5/16/23
to Bar Elhanati, Repo and Gerrit Discussion
On Tue, May 16, 2023 at 10:15 AM Bar Elhanati <elhana...@gmail.com> wrote:
Hi,

Thank you for your quick response

please don't use top posting on this list which makes it harder to follow the conversation, use interleaved style instead
 
My Gerrit server has 96 cores.
and yea, the server is busy and serving large repos. what do you mean by increasing the GC? can you please demonstrate?

I meant you should run git gc not in the Gerrit process but in another process, e.g. by scheduling it in a cronjob

 
??Another thing, SSH  maxConnectionsPerUser affect HTTP web sessions (web UI connections)

AFAIK ssh.maxConnectionsPerUser limits the number of concurrent ssh sessions a given user can open to a Gerrit ssh daemon.
It's unrelated to HTTP web sessions which are served by the servlet container e.g. embedded Jetty or Tomcat.
 
What is the value that you recommend for this parameter? 

This depends on which fraction of all available ssh threads you want to allow being used by a single user.
Default is 64.
 

Lior Amsalem

unread,
May 16, 2023, 8:01:55 AM5/16/23
to Repo and Gerrit Discussion
On Tuesday, May 16, 2023 at 11:50:18 AM UTC+3 Matthias Sohn wrote:
On Tue, May 16, 2023 at 10:15 AM Bar Elhanati <elhana...@gmail.com> wrote:
Hi,

Thank you for your quick response

please don't use top posting on this list which makes it harder to follow the conversation, use interleaved style instead

Sure, thanks for your valuable inputs so far,  more below.
 
My Gerrit server has 96 cores.
and yea, the server is busy and serving large repos. what do you mean by increasing the GC? can you please demonstrate?

I meant you should run git gc not in the Gerrit process but in another process, e.g. by scheduling it in a cronjob

Thanks, we'll look into it. But even if our Gerrit is loaded, it's not serving huge sized repositories and we don't see any impact on CPU and such during the GC time.
 

 
??Another thing, SSH  maxConnectionsPerUser affect HTTP web sessions (web UI connections)

AFAIK ssh.maxConnectionsPerUser limits the number of concurrent ssh sessions a given user can open to a Gerrit ssh daemon.
It's unrelated to HTTP web sessions which are served by the servlet container e.g. embedded Jetty or Tomcat.
 
What is the value that you recommend for this parameter? 

This depends on which fraction of all available ssh threads you want to allow being used by a single user.
Default is 64.

Since we serve a large volume of requests from a single automation user, this configuration was set as a need and we monitor the queues as well.
 
 
ב-יום שלישי, 16 במאי 2023 בשעה 10:43:33 UTC+3, Matthias Sohn כתב/ה:
On Tue, May 16, 2023 at 9:01 AM Bar Elhanati <elhana...@gmail.com> wrote:
Hi,
Me and my team are using Gerrit as code management platform.
We are having HTTP sessions disconnections. Those disconnections appear (mostly) when a user clicks on a Gerrit links  URL (from mail / chat) and it asks him to refresh credentials. 
The issue also appears when a user is on on a different page then landing page for 5~15 mins and then it asks him to refresh credentials.
I would like to solve this issue as soon as possible.
For some other users this issue does not exist and for some others it does. I  can't put my finger on some pattern.

After successful authentication the UI returns a session cookie to the browser. 

Yes, we see the cookie and if someone selects the remember me box the cookie will have the right expiration time per our config (1w). 


Some Gerrit's data:

Which Gerrit version do you run ?

We're  running 3.3.6 version.
We've monitoring the cache and increased it to 8k based on our findings, now we're around ~6k of the cache size.
As for diskLimit, AFAIU, it's only to keep all sessions alive after restart, which we don't really care for.

Is there a way to fetch all current sessions in Gerrit, I'm sure it'll help with further debug? (like we do for example in Jenkins with JavaMelody monitoring plugin.)

Any other configuration that can impact the behavior we're facing?

Nasser Grainawi

unread,
May 16, 2023, 10:33:26 AM5/16/23
to Lior Amsalem, Repo and Gerrit Discussion
Why limit this cache size at all though? You're going to expire any entries from it that are older than 1 week anyway. Your symptom definitely sounds like what would happen if cache entries are being evicted sooner than you expect, so I would try removing the size limit and see if the problem goes away. If that solves the problem and you still think you need a size limit, please let us know.
 

Is there a way to fetch all current sessions in Gerrit, I'm sure it'll help with further debug? (like we do for example in Jenkins with JavaMelody monitoring plugin.)

I don't know of a way to do that, sorry.
 

vlad...@gmail.com

unread,
Jul 31, 2023, 8:29:00 AM7/31/23
to Repo and Gerrit Discussion
Hello to respectful forum. 
I am working on the same issue with Lior and Bar.

Recently we realized that  chrome is limited to 180 cookies per domain. (firefox is less, safari is more)
As  a result our gerrit users  often loose session cookie,  since its address is "gerrit.<our_company>.com" and it shares same "<our_company>.com" domain as  dozens other sites that we use, and it is easy to get to 180 cookies. 
We are able to reproduce it  buy just opening many sites from  <our_company>.com domain,  and then refresh for gerrit web GUI, it will ask to re-login.

seems like  setting prio in specific customer browser helps, but not supported in gerrit config

Here is the question:
What is the best way to resolve this issue?
Reply all
Reply to author
Forward
0 new messages