Intermittent SSH connection drops in gerrit server

[davidch...@gmail.com]

Hi Community,

We are facing intermittent SSH connections drops from gerrit server (v2.16.22). We could see that application is not responding via SSH for few secs. At the time of the issue, the reported backlog queue is 51. This means that subsequent connection requests to this socket are being dropped as shown in the 'ss' statistics, as there's no room for more connection requests in the queue.

# sudo ss -peoanmi | egrep 'tcp.*LISTEN.*29418' | head -1

tcp LISTEN 0 50 [::]:29418 [::]:* users:(("java",pid=27412,fd=296)) timer:(keepalive,038ms,0) uid:1010 ino:42005 sk:ffff8ae00a0d8000 v6only:0 

 

 How can we increase this limit to overcome TCP connection drops.?

 

 Regards,

 Challs

[Luca Milanesio]

On 4 Jun 2021, at 17:04, davidch...@gmail.com <davidch...@gmail.com> wrote:

Hi Community,

We are facing intermittent SSH connections drops from gerrit server (v2.16.22). We could see that application is not responding via SSH for few secs.

Most of the times, that is due to a JVM GC activity, typically ergonomics GC or full-GC.

Have you checked the JVM GC log?

HTH

Luca.

At the time of the issue, the reported backlog queue is 51. This means that subsequent connection requests to this socket are being dropped as shown in the 'ss' statistics, as there's no room for more connection requests in the queue.

# sudo ss -peoanmi | egrep 'tcp.*LISTEN.*29418' | head -1

tcp LISTEN 0 50 [::]:29418 [::]:* users:(("java",pid=27412,fd=296)) timer:(keepalive,038ms,0) uid:1010 ino:42005 sk:ffff8ae00a0d8000 v6only:0 

 

 How can we increase this limit to overcome TCP connection drops.?

 

 Regards,

 Challs

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/9d037f06-7580-4faf-b8bb-f452181ac0een%40googlegroups.com.

[David Charles T M]

Hi Luca,

Thanks for the reply.

 

Most of the times, that is due to a JVM GC activity, typically ergonomics GC or full-GC.

Have you checked the JVM GC log?

We have checked Java GC logs, and we see that application threads were stopped for a few seconds whenever we face SSH drops. Reason why we were not accusing JVM G1GC for issues is that the application is running fine, most of the times when GC logs say application threads are stopped for 15-20 secs. Not all the application thread stoppage results in SSH connection drops. 

Regards,
Challs