Stale SSH connections

zafar shamim

unread,

Jul 27, 2018, 11:33:45 AM7/27/18

to Repo and Gerrit Discussion

We recently moved Gerrit Application (2.12.3) from RHEL 5 to RHEL 7.5 box.

Along with we upgraded to JDK 1.8 and Tomcat 9.

After the switchover, we are frequently observing piling list of SSH connections which have been initiated at least 15 mins before or more (while gerrit queue is empty).

This issue happened in the old box before, which got fixed after changing backend to NIO2.

Tried with idleTimeout and waitTimeout settings without any success.

Now with same Gerrit config, the issue is back again

threads = 128

batchThreads = 8

commandStartThreads = 5

maxConnectionsPerUser = 48

idleTimeout = 5m

waitTimeout = 15m

Gert van Dijk

unread,

Jul 27, 2018, 12:00:01 PM7/27/18

to Repo and Gerrit Discussion

On Friday, 27 July 2018 17:33:45 UTC+2, zafar shamim wrote:

We recently moved Gerrit Application (2.12.3) from RHEL 5 to RHEL 7.5 box.

Along with we upgraded to JDK 1.8 and Tomcat 9.

After the switchover, we are frequently observing piling list of SSH connections which have been initiated at least 15 mins before or more (while gerrit queue is empty).

This issue happened in the old box before, which got fixed after changing backend to NIO2.

Tried with idleTimeout and waitTimeout settings without any success.

I believe you should plan an upgrade path for Gerrit itself too. 2.12 is terribly old and numerous fixes have been incorporated in the embedded SSHD between 2.12.x and current stable 2.15.x.

HTH

zafar shamim

unread,

Jul 27, 2018, 12:16:56 PM7/27/18

to Repo and Gerrit Discussion

Thanks Gert ,we do have a plan for the upgrade to 2.15.3 in a month time.

But we need to get this fixed as once connections are exhausted our automation using this account is failing .

And most of these stale connections are for tasks associated with gerrit query.

The same query if I run from my account, finishes in a millisecond but for admin account, it does not logout eve after returning the results.

Matthias Sohn

unread,

Jul 27, 2018, 4:25:55 PM7/27/18

to zwith...@gmail.com, Repo and Gerrit Discussion

On Fri, Jul 27, 2018 at 6:17 PM zafar shamim <zwith...@gmail.com> wrote:

On Friday, 27 July 2018 12:00:01 UTC-4, Gert van Dijk wrote:
On Friday, 27 July 2018 17:33:45 UTC+2, zafar shamim wrote:

We recently moved Gerrit Application (2.12.3) from RHEL 5 to RHEL 7.5 box.

Along with we upgraded to JDK 1.8 and Tomcat 9.

After the switchover, we are frequently observing piling list of SSH connections which have been initiated at least 15 mins before or more (while gerrit queue is empty).

This issue happened in the old box before, which got fixed after changing backend to NIO2.

Tried with idleTimeout and waitTimeout settings without any success.

I believe you should plan an upgrade path for Gerrit itself too. 2.12 is terribly old and numerous fixes have been incorporated in the embedded SSHD between 2.12.x and current stable 2.15.x.

Thanks Gert ,we do have a plan for the upgrade to 2.15.3 in a month time.
But we need to get this fixed as once connections are exhausted our automation using this account is failing .
And most of these stale connections are for tasks associated with gerrit query.
The same query if I run from my account, finishes in a millisecond but for admin account, it does not logout eve after returning the results.

please avoid top posting

when you run the same query from your account do you use the same client ?

You should consider to at least update to the latest 2.12.8 service release, there are several fixes in 2.12.4 regarding ssh issues and

Lucene stability improvements in 2.12.5. See 2.12 release notes [1].

[1] https://www.gerritcodereview.com/releases/2.12.md#bugfix-releases

-Matthias

Reply all

Reply to author

Forward