Git clone - sessions killed

[AndrewF]

I manage a gerrit installation that was 2.1.5 and is now 2.1.6.1

This past week users started experiencing an issue when they attempt
to clone any of the repos. This issue happens with repos in sizes
ranging from 20-41 MB.

On the git client, they get an SSH connection timed out.

After reading thru mutliple posts, I tried the following:

1) Increase ulimit for the gerrit2 user
2) upgraded to Gerrit 2.1.6.1
3) made changes to gerrit.config and restarted
core.packedGitOpenFiles = 4096
core.packedGitLimit = 2g
core.packedGitWindowSize = 16k
container.heapLimit = 3g
sshd.threads=24
sshd.bathThreads=2
keepTcpAlive = false

I also tried pack.deltacompression=true

Background info:

ubuntu 10.04
apache 2.2
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.2)
(6b20-1.9.2-0ubuntu1~10.04.1)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

VMware host with 4 GB ram allocated and dual core 1.5 Ghz.
Load Avg: is 0.
Free Ram: 2 GB out of 4 GB allocated.

This server is not heavily used and is dedicated just to Gerrit.

Here are excerpts from the logs:

sshd_log
[2011-01-13 14:39:27,123 -0500] 07157fd2 xxxx a/2 'git-upload-pack
'\''/xxx-android/xxxx-lib'\''' 0ms 69773ms killed

==> error_log <==
[2011-01-13 14:39:27,476] WARN
org.apache.sshd.server.session.ServerSession : Exception caught
org.apache.mina.core.write.WriteTimeoutException
at
org.apache.mina.core.session.AbstractIoSession.notifyWriteTimeout(AbstractIoSession.java:
1304)
at
org.apache.mina.core.session.AbstractIoSession.notifyIdleSession(AbstractIoSession.java:
1286)
at
org.apache.mina.core.session.AbstractIoSession.notifyIdleness(AbstractIoSession.java:
1263)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.notifyIdleSessions(AbstractPollingIoProcessor.java:
747)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access
$700(AbstractPollingIoProcessor.java:66)
at org.apache.mina.core.polling.AbstractPollingIoProcessor
$Processor.run(AbstractPollingIoProcessor.java:1089)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:
64)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
[2011-01-13 14:39:27,477] ERROR com.google.gerrit.sshd.BaseCommand :
Internal server error (user xxxx account 2) during git-upload-pack '/
xxx-android/xxxx-lib'
org.apache.sshd.common.channel.WindowClosedException: Already closed
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:
153)
at
org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:
104)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:
75)
at
org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:
154)
at
org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:
134)
at
org.eclipse.jgit.storage.pack.PackOutputStream.write(PackOutputStream.java:
124)
at org.eclipse.jgit.storage.file.PackFile.copyAsIs2(PackFile.java:
447)
at org.eclipse.jgit.storage.file.PackFile.copyAsIs(PackFile.java:290)
at
org.eclipse.jgit.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:
147)
at
org.eclipse.jgit.storage.pack.PackWriter.writeObject(PackWriter.java:
837)
at
org.eclipse.jgit.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:
161)
at
org.eclipse.jgit.storage.file.WindowCursor.writeObjects(WindowCursor.java:
153)
at
org.eclipse.jgit.storage.pack.PackWriter.writeObjects(PackWriter.java:
815)
at org.eclipse.jgit.storage.pack.PackWriter.writePack(PackWriter.java:
503)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:
657)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:352)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:313)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:50)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:
104)
at com.google.gerrit.sshd.AbstractGitCommand.access
$000(AbstractGitCommand.java:34)
at com.google.gerrit.sshd.AbstractGitCommand
$1.run(AbstractGitCommand.java:69)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:
395)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:
324)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
[2011-01-13 14:39:27,479] WARN
org.apache.sshd.server.session.ServerSession : Exception caught
org.apache.mina.core.write.WriteToClosedSessionException
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.clearWriteRequestQueue(AbstractPollingIoProcessor.java:
618)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.removeNow(AbstractPollingIoProcessor.java:
569)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.removeSessions(AbstractPollingIoProcessor.java:
539)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access
$600(AbstractPollingIoProcessor.java:66)
at org.apache.mina.core.polling.AbstractPollingIoProcessor
$Processor.run(AbstractPollingIoProcessor.java:1086)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:
64)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

[Shawn Pearce]

On Fri, Jan 14, 2011 at 15:02, AndrewF <tigge...@gmail.com> wrote:
> This past week users started experiencing an issue when they attempt
> to clone any of the repos. This issue happens with repos in sizes
> ranging from 20-41 MB.
>
> On the git client, they get an SSH connection timed out.

...

> sshd_log
> [2011-01-13 14:39:27,123 -0500] 07157fd2 xxxx a/2 'git-upload-pack
> '\''/xxx-android/xxxx-lib'\''' 0ms 69773ms killed

...

> [2011-01-13 14:39:27,477] ERROR com.google.gerrit.sshd.BaseCommand :
> Internal server error (user xxxx account 2) during git-upload-pack '/
> xxx-android/xxxx-lib'
> org.apache.sshd.common.channel.WindowClosedException: Already closed

...
>        at org.eclipse.jgit.storage.file.PackFile.copyAsIs2(PackFile.java:

Hmm. This looks like the connection is just dropping, and then the
server times out because the client wasn't receiving anymore data. I
don't have the code in front of me, but if I remember it right the
"killed" in the end of the sshd_log line entry indicates the local SSH
client was killed, sending a signal to the server to shutdown the
task. Which it did by closing down the pipe, resulting in the write
error.

As for why the client aborted... it might be because of the SSH
KeepAlive setting. Is it set really low and the client isn't able to
get the KeepAlive ping back from the server? I don't know, you would
need to use ssh with a -v (or multiple -v options like -vvv) to see
the debug trace on the client and watch for when it sends/receives
KeepAlive packets with the server.

[AndrewF]

The user is using the git client to repo the code via ssh path.

git clone ssh://username@gerrit:2224/xxx-android/xxx-lib

I dont see a mechanism for passing timeout parameters to the git
client.

I thought the connection was dropping as well, but a tcpdump filtered
to the remote IP, shows constant packets back and forth.