sshd server exception

[Sundar Shankar]

I am running 2.6 and started to notice this exception. Server is configure with 32 sshd threads and 32 DB PoolLimit.
 I started to get only after i increased threads to meet increasing Jenkins jobs. Any idea why I get this errors?


[2013-08-08 12:35:47,847] WARN  org.apache.sshd.server.session.ServerSession : Exception caught
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
        at sun.nio.ch.IOUtil.read(IOUtil.java:171)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:280)
        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:44)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:677)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1111)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

[Martin Fick]

On Thursday, August 08, 2013 02:01:37 pm Sundar Shankar
wrote:

> I am running 2.6 and started to notice this exception.
> Server is configure with 32 sshd threads and 32 DB
> PoolLimit.
> I started to get only after i increased threads to meet
> increasing Jenkins jobs. Any idea why I get this errors?
>
>
> [2013-08-08 12:35:47,847] WARN
> org.apache.sshd.server.session.ServerSession : Exception
> caught java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)

This just looks like a client disconnected. It could be
that the client timed out? Are you using ssh channel
master?

However, if your thread count is 32, you need a higher DB
pool limit. You need your db pool limit to sum up to all
the possible threads (plus 2 I think). There are other
threads you need to consider, including the 2 other ssh
threads and any http threads...

-Martin

[sundar]

Thanks Martin. Yes, I use ssh channel. We were using 16 threads before but that was not enough so increased to 32.
Let me re-configure the server to use a DB poollimit of 32+8.  Btw, what's the recommended HW requirement for 32 or 64  sshd connections for best performance ?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/cS9K37gcb7A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Martin Fick]

On Thursday, August 08, 2013 03:14:19 pm sundar wrote:
> Thanks Martin. Yes, I use ssh channel.

It seems to fail when talking to gerrit at times, not sure
if it is the mina sshd server or openssh client (some
versions are a bit better) that is buggy, but the combo
seems to be unreliable. We don't use it. :(

> Let me re-configure the server to use a DB poollimit of
> 32+8.
> Btw, what's the recommended HW requirement for 32
> or 64 sshd connections for best performance ?

Depends on your repos sizes. Internally we have found that
clones tend to be the most problematic use case. If you
have large repos, they can cause massive java gcs with long
pauses. I can tell you that 32 simultaneous clones of one
of our large repos will bring our server to its knees.
That's with a 40GB heap. Not sure what it would take to
actually prevent that, and thus have a setup that can handle
more threads. We generally deal with the problem by adding
more slaves and load balancing between them. But it does
mean that our slaves are generally not being pushed very
hard and yet we sometimes run out of threads when slow
connections chew them up for long periods of time. :(

Perhaps if you use jgit's bitmaps the server will no longer
be brought to its knees by clones so you could likely then
safely bump the thread count?

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation