git push failing 'fatal: the remote end hung up unexpectedly"

nitish kumar

unread,

Sep 17, 2019, 7:01:08 AM9/17/19

to Repo and Gerrit Discussion

hi,

can someone please help on the below issue

we have repo "UI" with around 2GB size and we are frequently getting push error's.

#######error message###########

:UI $ git push origin HEAD:refs/for/13.0_ib

Enumerating objects: 23, done.

Counting objects: 100% (23/23), done.

Delta compression using up to 12 threads

Compressing objects: 100% (12/12), done.

Writing objects: 100% (12/12), 1.11 KiB | 161.00 KiB/s, done.

Total 12 (delta 9), reused 0 (delta 0)

remote: Resolving deltas: 100% (9/9)

Connection to gerrit.com closed by remote host.

fatal: the remote end hung up unexpectedly

#######gerrit details #########

gerrit_version: 2.11.3

gerrit_sshd_config:

[sshd]

backend = NIO2

listenAddress = *:29418

threads = 20

loginGraceTime = 120s

batchThreads = 12

commandStartThreads = 5

maxConnectionsPerUser = 10

@gerrit logs]$ free -m

total used free shared buffers cached

Mem: 29636 17154 12481 664 1209 9647

-/+ buffers/cache: 6297 23339

Swap: 0 0 0

UI $ git count-objects -v

count: 0

size: 0

in-pack: 924519

packs: 3

size-pack: 1441059

prune-packable: 0

garbage: 0

size-garbage: 0

==> sshd_log <==

[2019-09-17 06:30:21,989 -0400] 8b1d3969 username a/1000177 LOGOUT

[2019-09-17 06:30:21,989 -0400] 8b1d3969 username a/1000177 git-receive-pack./UI 0ms 76610ms killed

==> error_log <==

[2019-09-17 06:30:21,989] WARN com.google.gerrit.server.git.AsyncReceiveCommits : Error in ReceiveCommits while processing changes for project UI

java.util.concurrent.ExecutionException: java.lang.InterruptedException

at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:211)

at com.google.gerrit.server.git.AsyncReceiveCommits.onPreReceive(AsyncReceiveCommits.java:162)

at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:208)

at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:161)

at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:115)\

at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:445)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.InterruptedException

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:460)

at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348)

at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:209)

... 16 more

==> sshd_log <==

[2019-09-17 06:30:55,988 -0400] 8b8699c1 username a/1000017 LOGOUT

[2019-09-17 06:30:55,988 -0400] 8b8699c1 username a/1000017 git-receive-pack./UI.git 0ms 67227ms killed

==> error_log <==

[2019-09-17 06:30:59,054] WARN com.google.gerrit.server.git.AsyncReceiveCommits : Error in ReceiveCommits while processing changes for project UI

java.util.concurrent.ExecutionException: java.lang.InterruptedException

at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:211)

at com.google.gerrit.server.git.AsyncReceiveCommits.onPreReceive(AsyncReceiveCommits.java:162)

at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:208)

at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:161)

at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:115)

at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:445)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.InterruptedException

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:460)

at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348)

at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:209)

... 16 more

==> sshd_log <==

thanks,

Nitish

Matthias Sohn

unread,

Sep 17, 2019, 8:08:21 AM9/17/19

to nitish kumar, Repo and Gerrit Discussion

On Tue, Sep 17, 2019 at 1:01 PM nitish kumar <vulli...@gmail.com> wrote:

hi,
can someone please help on the below issue
we have repo "UI" with around 2GB size and we are frequently getting push error's.

#######error message###########
:UI $ git push origin HEAD:refs/for/13.0_ib
Enumerating objects: 23, done.
Counting objects: 100% (23/23), done.
Delta compression using up to 12 threads
Compressing objects: 100% (12/12), done.
Writing objects: 100% (12/12), 1.11 KiB | 161.00 KiB/s, done.
Total 12 (delta 9), reused 0 (delta 0)
remote: Resolving deltas: 100% (9/9)
Connection to gerrit.com closed by remote host.
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly

#######gerrit details #########

gerrit_version: 2.11.3

This release is more than 4 years old.

You are not on the latest 2.11.x release which would be 2.11.12 so you potentially miss many bug fixes.

You should consider to upgrade to a maintained release soon, currently 2.15, 2.16, 3.0 are maintained [1].

gerrit_sshd_config:
[sshd]
backend = NIO2
listenAddress = *:29418
threads = 20
loginGraceTime = 120s
batchThreads = 12
commandStartThreads = 5
maxConnectionsPerUser = 10

try to increase sshd.idleTimeout [2] and receive.timeout [3], we use 10min and 5min.

[1] https://www.gerritcodereview.com/support.html#supported-versions

[2] https://gerrit-documentation.storage.googleapis.com/Documentation/2.11.12/config-gerrit.html#sshd.idleTimeout

[3] https://gerrit-documentation.storage.googleapis.com/Documentation/2.11.12/config-gerrit.html#receive.timeout

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/83cc4034-4071-4fbe-bb66-4d98244d9382%40googlegroups.com.

nitish kumar

unread,

Sep 17, 2019, 11:04:51 AM9/17/19

to Repo and Gerrit Discussion

To unsubscribe, email repo-d...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/83cc4034-4071-4fbe-bb66-4d98244d9382%40googlegroups.com.

Hi matthias sohn,

thanks for the input, update gerrit.config as suggested and monitoring for now.

Soon we will be upgrading to 2.15.15

Thanks,

Nitish

Matthias Sohn

unread,

Sep 17, 2019, 12:01:32 PM9/17/19

to nitish kumar, Repo and Gerrit Discussion

why not 2.15.16 [1] ?

[1] https://www.gerritcodereview.com/2.15.html#21516

Thanks,
Nitish

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/65b3f0b5-852a-4c9a-860b-87be8a12e7e2%40googlegroups.com.

nitish kumar

unread,

Sep 18, 2019, 8:59:32 AM9/18/19

to Repo and Gerrit Discussion

yes :) forgot I have upgraded one instance up to 2.15.15 (not production instance)

2.15.16 released 3 weeks ago right , will try 2.15.16, if no issues will update to 2.15.16.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/65b3f0b5-852a-4c9a-860b-87be8a12e7e2%40googlegroups.com.

nitish kumar

unread,

Sep 19, 2019, 8:40:55 AM9/19/19

to Repo and Gerrit Discussion

hi,

I am still getting the same issue.

identified there is a occasional packets loss in the network , getting rst in wireshark.

what happens to push and pull in case of packets loss any idea???

Note: I am using ssh to clone or push

thanks,

Nitish

nitish kumar

unread,

Sep 19, 2019, 10:10:53 AM9/19/19

to Repo and Gerrit Discussion

found below issue in logs.

looks like GC failed on that project.

any idea how can fix this, can i run git gc on that project in gerrit server???

[2019-09-19 10:01:38,753] ERROR com.google.gerrit.server.git.GarbageCollection : [UI]

org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.

at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:192)

at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:83)

at com.google.gerrit.sshd.commands.GarbageCollectionCommand.runGC(GarbageCollectionCommand.java:87)

at com.google.gerrit.sshd.commands.GarbageCollectionCommand.run(GarbageCollectionCommand.java:62)

at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:445)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown a910a17571eaecf132d8ae0e385f5564e3790fa1

at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148)

at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)

at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:865)

at org.eclipse.jgit.internal.storage.file.RefDirectory.doPeel(RefDirectory.java:482)

at org.eclipse.jgit.internal.storage.file.RefDirectory.peel(RefDirectory.java:461)

at org.eclipse.jgit.internal.storage.file.RefDirectory.peeledPackedRef(RefDirectory.java:693)

at org.eclipse.jgit.internal.storage.file.RefDirectory.pack(RefDirectory.java:632)

at org.eclipse.jgit.internal.storage.file.GC.packRefs(GC.java:496)

at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:167)

at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:175)

Matthias Sohn

unread,

Sep 19, 2019, 10:50:18 AM9/19/19

to nitish kumar, Repo and Gerrit Discussion

On Thu, Sep 19, 2019 at 4:10 PM nitish kumar <vulli...@gmail.com> wrote:

found below issue in logs.
looks like GC failed on that project.

any idea how can fix this, can i run git gc on that project in gerrit server???

considering that you run a very old version of Gerrit and JGit this may have been caused by some IO exception which

old JGit versions didn't handle properly so that the in-memory pack list for the affected repository is incomplete.

Try to run git gc using native git, this should fix such transient errors. Another way to workaround such transient

issues is a restart of Gerrit.

Upgrade to a recent release to get rid of these problems which were fixed quite some time ago.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f5012ca9-83db-4f15-9aa1-deb0778e00b7%40googlegroups.com.

nitish kumar

unread,

Sep 25, 2019, 1:07:03 PM9/25/19

to Repo and Gerrit Discussion

after running git gc on that repo not seeing any failed push sessions.

also able to fix jgit gc failure by removing below object.

error: refs/changes/83/144083/1 does not point to a valid object!

thanks,

Nitish

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f5012ca9-83db-4f15-9aa1-deb0778e00b7%40googlegroups.com.

Matthias Sohn

unread,

Sep 25, 2019, 1:38:10 PM9/25/19

to nitish kumar, Repo and Gerrit Discussion

On Wed, Sep 25, 2019 at 7:07 PM nitish kumar <vulli...@gmail.com> wrote:

after running git gc on that repo not seeing any failed push sessions.

this matches what I suspected in my last email. The problem can back any time when the next

exception such old JGit versions don't handle properly occur again.

You can workaround by running gc but to get rid of them you need to upgrade.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/914823a8-3a04-4620-bb38-9818689cd6a2%40googlegroups.com.

nitish kumar

unread,

Sep 25, 2019, 1:40:52 PM9/25/19

to Matthias Sohn, Repo and Gerrit Discussion

yes updating 2.15.17 this weekend :)

Reply all

Reply to author

Forward