Hudson Unable to Clone from Gerrit

[Nicholas Mucci]

Hello,

We've been noticing a lot of failures where Hudson is unable to clone
from Gerrit. I'm not sure if this is a problem on the Hudson or
Gerrit side. Here's the scenarios we are experiencing though.
Apologies for the length of this:

1) Hudson will start a job, initiate cloning a repository, and then
the clone will hang forever. On the Gerrit side, I can see that there
is a git-upload-pack thread running for the clone, but it seems Hudson
never receives anything, and the thread runs forever, or until the
Hudson build is cancelled. These start to eat into the
sshd.batchThreads quickly, preventing other builds from starting.

2) Clones fail with the following Hudson output:

$ "C:\Program Files\Git\cmd\git.cmd" clone -o origin
ssh://gerrit.foo.com:29418/foobar/base C:\hudson\workspace\Gerrit
ERROR: Error cloning remote repo 'origin' : Could not clone
ssh://gerrit.foo.com:29418/foobar/base
ERROR: Cause: Error performing C:\Program Files\Git\cmd\git.cmd clone -
o origin ssh://gerrit.foo.com:29418/foobar/base C:\hudson\workspace
\Gerrit
Command returned status code 128: fatal: internal server error
fatal: early EOF
Cloning into C:\hudson\workspace\Gerrit...
fatal: The remote end hung up unexpectedly
fatal: index-pack failed

On the Gerrit server, I see the following:

[2010-12-02 13:16:25,772] ERROR com.google.gerrit.sshd.BaseCommand :
Internal server error (user buildy account 1000283) during git-upload-
pack '/foobar/base'
java.io.FileNotFoundException: /data/gerrit/repositories/foobar/
base.git/objects/pack/
pack-38a7a105e2d9760a5d8f844f2de08cc49b1d01c6.idx (No such file or
directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at org.eclipse.jgit.lib.PackIndex.open(PackIndex.java:87)
at org.eclipse.jgit.lib.PackFile.idx(PackFile.java:137)
at org.eclipse.jgit.lib.PackFile.getReverseIdx(PackFile.java:
523)
at
org.eclipse.jgit.lib.PackFile.findObjectForOffset(PackFile.java:253)
at org.eclipse.jgit.lib.PackFile.copyRawData(PackFile.java:
293)
at
org.eclipse.jgit.lib.PackedObjectLoader.copyRawData(PackedObjectLoader.java:
162)
at org.eclipse.jgit.lib.PackWriter.writeObject(PackWriter.java:
726)
at org.eclipse.jgit.lib.PackWriter.writeObject(PackWriter.java:
709)
at
org.eclipse.jgit.lib.PackWriter.writeObjects(PackWriter.java:694)
at org.eclipse.jgit.lib.PackWriter.writePack(PackWriter.java:
595)
at
org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:587)
at
org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:334)
at
org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:296)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:
45)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:
104)
at com.google.gerrit.sshd.AbstractGitCommand.access
$000(AbstractGitCommand.java:34)
at com.google.gerrit.sshd.AbstractGitCommand
$1.run(AbstractGitCommand.java:69)
at com.google.gerrit.sshd.BaseCommand
$TaskThunk.run(BaseCommand.java:395)
at java.util.concurrent.Executors
$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at com.google.gerrit.server.git.WorkQueue
$Task.run(WorkQueue.java:324)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

There are two other Gerrit internal server errors that I'm seeing, but
I don't know what is happening on the client side (yet):

3) [2010-12-07 12:08:55,607] WARN
org.apache.sshd.server.session.ServerSession : Exception caught
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:206)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:
236)
at
org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:
202)
at
org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:
42)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:
620)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:
598)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:
587)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access
$400(AbstractPollingIoProcessor.java:61)
at org.apache.mina.core.polling.AbstractPollingIoProcessor
$Processor.run(AbstractPollingIoProcessor.java:969)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:
64)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
[2010-12-07 12:09:13,071] ERROR com.google.gerrit.sshd.BaseCommand :
Internal server error (user buildy account 1000283) during git-upload-
pack '/foobar/base'
java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:
184)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:317)
at org.eclipse.jgit.util.IO.readFully(IO.java:97)
at org.eclipse.jgit.util.IO.readFully(IO.java:76)
at
org.eclipse.jgit.lib.UnpackedObjectLoader.<init>(UnpackedObjectLoader.java:
83)
at
org.eclipse.jgit.lib.ObjectDirectory.openObject2(ObjectDirectory.java:
285)
at
org.eclipse.jgit.lib.ObjectDatabase.openObjectImpl2(ObjectDatabase.java:
237)
at
org.eclipse.jgit.lib.ObjectDatabase.openObject(ObjectDatabase.java:
202)
at org.eclipse.jgit.lib.Repository.openObject(Repository.java:
485)
at
org.eclipse.jgit.lib.PackWriter.writeWholeObjectDeflate(PackWriter.java:
761)
at org.eclipse.jgit.lib.PackWriter.writeObject(PackWriter.java:
733)
at
org.eclipse.jgit.lib.PackWriter.writeObjects(PackWriter.java:694)
at org.eclipse.jgit.lib.PackWriter.writePack(PackWriter.java:
595)
at
org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:587)
at
org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:334)
at
org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:296)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:
45)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:
104)
at com.google.gerrit.sshd.AbstractGitCommand.access
$000(AbstractGitCommand.java:34)
at com.google.gerrit.sshd.AbstractGitCommand
$1.run(AbstractGitCommand.java:69)
at com.google.gerrit.sshd.BaseCommand
$TaskThunk.run(BaseCommand.java:395)
at java.util.concurrent.Executors
$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at com.google.gerrit.server.git.WorkQueue
$Task.run(WorkQueue.java:324)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)

4) [2010-12-07 14:38:58,540] WARN
org.apache.sshd.server.session.ServerSession : Exception caught
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:206)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:
236)
at
org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:
202)
at
org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:
42)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:
620)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:
598)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:
587)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access
$400(AbstractPollingIoProcessor.java:61)
at org.apache.mina.core.polling.AbstractPollingIoProcessor
$Processor.run(AbstractPollingIoProcessor.java:969)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:
64)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
[2010-12-07 14:38:58,541] ERROR com.google.gerrit.sshd.BaseCommand :
Internal server error (user buildy account 1000283) during git-upload-
pack '/foobar/base'
org.apache.sshd.common.SshException
at
org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:
128)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:
75)
at
org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:
154)
at
org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:
134)
at
org.eclipse.jgit.lib.PackOutputStream.write(PackOutputStream.java:77)
at
java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:56)
at org.eclipse.jgit.lib.PackFile.copyToStream(PackFile.java:
335)
at org.eclipse.jgit.lib.PackFile.copyRawData(PackFile.java:
290)
at
org.eclipse.jgit.lib.PackedObjectLoader.copyRawData(PackedObjectLoader.java:
162)
at org.eclipse.jgit.lib.PackWriter.writeObject(PackWriter.java:
726)
at
org.eclipse.jgit.lib.PackWriter.writeObjects(PackWriter.java:694)
at org.eclipse.jgit.lib.PackWriter.writePack(PackWriter.java:
595)
at
org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:587)
at
org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:334)
at
org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:296)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:
45)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:
104)
at com.google.gerrit.sshd.AbstractGitCommand.access
$000(AbstractGitCommand.java:34)
at com.google.gerrit.sshd.AbstractGitCommand
$1.run(AbstractGitCommand.java:69)
at com.google.gerrit.sshd.BaseCommand
$TaskThunk.run(BaseCommand.java:395)
at java.util.concurrent.Executors
$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at com.google.gerrit.server.git.WorkQueue
$Task.run(WorkQueue.java:324)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.sshd.common.channel.Window.waitForSpace(Window.java:146)
at
org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:
104)
... 28 more


These last two errors are not only occurring with Hudson builds;
regular git operations that require talking to the Gerrit server will
cause these as well.

I'd appreciate any insight or suggestions that can be given regarding
these errors. I'll try and find the right Hudson list to cross-post
this on. Thanks.

-Nick

[Mark]

We've had your 2nd problem (early EOF) from time to time. There are
two things which seem to help:

1. Run git gc on the server repository. This seems to help though I
can't comment on *why* and it doesn't always work. (I seem to have
slightly more success with git gc --aggressive, sometimes, maybe... )

2. If running git gc doesn't work then wiping out the hudson workspace
and cloning again from scratch can help.

This is certainly not ideal.

Can I ask what build of Git you're using on the Hudson box?

-mark

[Shawn Pearce]

On Wed, Dec 8, 2010 at 09:47, Nicholas Mucci <nick....@gmail.com> wrote:
> There are two other Gerrit internal server errors that I'm seeing, but
> I don't know what is happening on the client side (yet):
>

> [2010-12-07 12:09:13,071] ERROR com.google.gerrit.sshd.BaseCommand :
> Internal server error (user buildy account 1000283) during git-upload-
> pack '/foobar/base'
> java.nio.channels.ClosedByInterruptException
>        at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:
> 184)
>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:317)
>        at org.eclipse.jgit.util.IO.readFully(IO.java:97)
>        at org.eclipse.jgit.util.IO.readFully(IO.java:76)
>        at
> org.eclipse.jgit.lib.UnpackedObjectLoader.<init>(UnpackedObjectLoader.java:
> 83)

What version of Gerrit is this? I thought we had fixed all of these
close-by-interrupt problems.

[Nicholas Mucci]

Mark,

We run "git gc" three times daily on our Gerrit servers. We suspect
the "early EOF; index-pack failed" problem is related to a jgit bug or
with an issue Shawn mentioned here:

http://groups.google.com/group/repo-discuss/browse_thread/thread/86998aa883ae9a96#

which leads us into a similar situation that Fredrik Luthander was in
here:

http://groups.google.com/group/repo-discuss/browse_thread/thread/2c9f81bc45e48076

The problem only manifests on a very large (~700M) repository. Once
we see this problem, immediately after a "git gc" the problem goes
away temporarily.

Our core.streamFileThreshold value is set to 4096m, which should be
totally overkill but ensure functionality.

For all clean builds, the Hudson workspace is wiped out first. The
Hudson box is running Git 1.7.3.1.msysgit.0.

-Nick

> ...
>
> read more »

[Nicholas Mucci]

Shawn,

We're running Gerrit 2.1.5.

-Nick

On Dec 13, 10:49 am, Shawn Pearce <s...@google.com> wrote:

[Mark]

For what it's worth - we're also on 2.1.5 and see this very
intermittently (but since 2.1.3). Our repository is just under 160M
(i.e. big but not massive, well under 700M). I wonder if it will
become more frequent as the repository grows? I should also add that
this only happens on one repository - we've never seen the problem for
our other repos.

Nick, thanks for those threads I had missed those somehow.

cheers,

-mark

[Shawn Pearce]

On Mon, Dec 13, 2010 at 08:55, Nicholas Mucci <nick....@gmail.com> wrote:
>
> We're running Gerrit 2.1.5.

I'm pretty certain that the ClosedByInterruptException you see

>> > [2010-12-07 12:09:13,071] ERROR com.google.gerrit.sshd.BaseCommand :
>> > Internal server error (user buildy account 1000283) during git-upload-
>> > pack '/foobar/base'
>> > java.nio.channels.ClosedByInterruptException
>> >        at
>> > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:
>> > 184)
>> >        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:317)
>> >        at org.eclipse.jgit.util.IO.readFully(IO.java:97)
>> >        at org.eclipse.jgit.util.IO.readFully(IO.java:76)
>> >        at
>> > org.eclipse.jgit.lib.UnpackedObjectLoader.<init>(UnpackedObjectLoader.java:
>> > 83)

is fixed in Gerrit 2.1.6. JGit rewrote this part of the code to avoid
this problem, and Gerrit picked up the new version as part of 2.1.6.
I would encourage you to try upgrading Gerrit and see if at least this
problem disappears.

[Shawn Pearce]

On Mon, Dec 13, 2010 at 10:34, Mark <mark.b...@gmail.com> wrote:
> For what it's worth - we're also on 2.1.5 and see this very
> intermittently (but since 2.1.3). Our repository is just under 160M
> (i.e. big but not massive, well under 700M). I wonder if it will
> become more frequent as the repository grows? I should also add that
> this only happens on one repository - we've never seen the problem for
> our other repos.

...

>> > > [2010-12-07 12:09:13,071] ERROR com.google.gerrit.sshd.BaseCommand :
>> > > Internal server error (user buildy account 1000283) during git-upload-
>> > > pack '/foobar/base'
>> > > java.nio.channels.ClosedByInterruptException

Like I said to Nicholas, I think I have this fixed in 2.1.6. Its not
dependent on the size of the repository, but as the size increases the
longer it takes for a client to clone the repository. The longer the
clone is running, the chance of receiving an interrupt during the
clone increases. So as the repository gets bigger, its just more
likely that you will see these errors... until you upgrade to a
version that doesn't have this particular bug.

[Nicholas Mucci]

Shawn,

We are running 2.1.6 in our test environment and are planning to
upgrade soon...this will likely hasten that schedule. :)

Any ideas on the "index-pack failed/FileNotFoundException" errors? My
friendly local JGit hacker down the hall thinks the "Fix corrupted
large deltas (4522b07d)" change in JGit may fix this. Thoughts?

-Nick

On Dec 13, 12:48 pm, Shawn Pearce <s...@google.com> wrote: