not able to terminate ssh connection

[Åsmund Østvold]

Hi

Over time my Gerrit server accumulate "zombie" ssh connections that is not possible to close. I had this problem with 2.11.3. Following the mailing list I believe this issue was fixed in 2.12-2. I upgraded today and observed that I still get these:

>ssh -p 29418 cod.no.corp.com  gerrit show-connections

Session     Start     Idle   User            Remote Host

--------------------------------------------------------------

e1314d50 07:05:30 00:00:56   franklin        trout.no.corp.com

c1344942 07:05:30 00:00:48   franklin        trout.no.corp.com

fe848c25 07:07:36 00:00:00   aostvold        dhcp-oslo-4fl-10-172

--

SSHD Backend: mina

>ssh -p 29418 cod.no.corp.com  gerrit show-connections

Session     Start     Idle   User            Remote Host

--------------------------------------------------------------

e1314d50 07:05:30 00:07:04   franklin        ?

1e15e0e8 07:12:51 00:01:10   franklin        ?

9e6c7083 07:14:51 00:00:10   franklin        trout.no.corp.com

7e6f3c78 07:14:51 00:00:10   franklin        trout.no.corp.com

9e5310ca 07:16:02 00:00:00   aostvold        dhcp-oslo-4fl-10-172

This seems to correlate the stactrace from error_log:

[2016-05-24 06:54:37,180] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.12.2 ready
[2016-05-24 07:08:57,675] [NioProcessor-9] WARN  com.google.gerrit.sshd.GerritServerSession : Exception caught
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
        at sun.nio.ch.IOUtil.read(IOUtil.java:198)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:302)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:45)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:694)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
[2016-05-24 07:14:51,703] [NioProcessor-6] WARN  com.google.gerrit.sshd.GerritServerSession : Exception caught
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
        at sun.nio.ch.IOUtil.read(IOUtil.java:198)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:302)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:45)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:694)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)


Trying to close the connection do not remove the item :

>ssh -p 29418 cod.no.corp.com  gerrit close-connection e1314d50
closing connection e1314d50...

>ssh -p 29418 cod.no.corp.com  gerrit show-connections
Session     Start     Idle   User            Remote Host
--------------------------------------------------------------
e1314d50 07:05:30 01:46:59   franklin        ?
1e15e0e8 07:12:51 01:41:05   franklin        ?
0f9b7131 08:54:25 00:00:04   franklin        trout.no.corp.com
8fe1a1a8 08:55:54 00:00:02   franklin        trout.no.corp.com
0fcdb138 08:55:57 00:00:00   aostvold        dhcp-oslo-4fl-10-172
--
SSHD Backend: mina


Could it be that I have misconfigured something?


Regards,
Asmund
  

[Doug Kelly]

Hello Asmund,

No, this isn't a misconfiguration, but it is a known bug in the default SSH backend for Gerrit (MINA SSHD): https://code.google.com/p/gerrit/issues/detail?id=3685

For us, the workaround was to set the sshd.backend option to NIO2 in gerrit.config (see https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#sshd).

Hope this helps!

--Doug

 

[Åsmund Østvold]

Thank you Doug,

We have been running with nio2 for some days now and I have not observed again. 

Asmund

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.