fatal: the remote end hung up unexpectedly

183 views
Skip to first unread message

Raj Ravi

unread,
Jul 17, 2019, 5:33:36 AM7/17/19
to Repo and Gerrit Discussion
Hi All,

I am trying to use one of the branch & tag into our gerrit instance. But i am getting fatal error while pushing the branch and tag together by using the below steps. 

The SSH idletimeout value has been given as 10M in the gerrit.config.

Please let me know if i am missing any thing here.

Steps:
git clone ssh://<user_name>@<hostname>:8282/test_repo
cd test_repo
git fetch upstream
git checkout v5.1.15
git checkout -b 5.1.x/master v5.1.15
git push origin 5.1.x/master v5.1.15

Error:
time git push origin  5.1.x/master
Total 0 (delta 0), reused 0 (delta 0)
Connection to hostname closed by remote host.
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly

real    10m0.492s
user    0m0.195s
sys     0m0.108s


Error Log:

[2019-07-17 13:23:39,003] [sshd-SshServer[74727e41]-nio2-thread-15] WARN  org.apache.sshd.server.session.ServerSessionImpl : exceptionCaught(ServerSessionImpl[ID@/IP:47200])[state=Opened] InterruptedByTimeoutException: null
[2019-07-17 13:23:39,003] [SSH git-receive-pack /REPO (ID)] WARN  com.google.gerrit.server.git.AsyncReceiveCommits : Error in ReceiveCommits while processing changes for project REPO
java.util.concurrent.ExecutionException: java.lang.InterruptedException
        at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:208)
        at com.google.gerrit.server.git.AsyncReceiveCommits.onPreReceive(AsyncReceiveCommits.java:153)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:266)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:208)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:93)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:97)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:30)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:453)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:528)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at java.util.concurrent.TimeUnit.timedWait(Unknown Source)
        at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:206)
        ... 16 more


[2019-07-17 13:23:39,009] [SSH git-receive-pack project (user)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user account id) during git-receive-pack '/repo'
org.apache.sshd.common.SshException: write(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[user@/IP:47200]] SSH_MSG_CHANNEL_DATA) len=65520 - channel already closed
        at org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:106)
        at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:171)
        at org.eclipse.jgit.transport.SideBandOutputStream.flushBuffer(SideBandOutputStream.java:127)
        at org.eclipse.jgit.transport.BaseReceivePack.close(BaseReceivePack.java:1790)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:211)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:93)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:97)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:30)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:453)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:528)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)


Luca Milanesio

unread,
Jul 17, 2019, 6:28:12 AM7/17/19
to Raj Ravi, Luca Milanesio, Repo and Gerrit Discussion

On 17 Jul 2019, at 10:33, Raj Ravi <keerthish...@gmail.com> wrote:

Hi All,

I am trying to use one of the branch & tag into our gerrit instance. But i am getting fatal error while pushing the branch and tag together by using the below steps. 

The SSH idletimeout value has been given as 10M in the gerrit.config.

Please let me know if i am missing any thing here.

Steps:
git clone ssh://<user_name>@<hostname>:8282/test_repo
cd test_repo
git fetch upstream
git checkout v5.1.15
git checkout -b 5.1.x/master v5.1.15
git push origin 5.1.x/master v5.1.15

Error:
time git push origin  5.1.x/master
Total 0 (delta 0), reused 0 (delta 0)
Connection to hostname closed by remote host.
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly

real    10m0.492s
user    0m0.195s
sys     0m0.108s

That make sense if you have set 10 minutes timeout: after 10 minutes, Gerrit closes the connection if there is no network activity.
Why would you expect something different?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/23e705a7-5d8c-4fb6-a431-0328bd56c5f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Raj Ravi

unread,
Jul 17, 2019, 10:04:16 AM7/17/19
to Repo and Gerrit Discussion

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

while i am trying to push a tag its getting successful and pushing with the branch it's giving error and the 10M is Maximum timeout has been given in our environment. Are you sure increasing 10M will solve this issue?   or kindly let me know why the git is not able to process the objects.

Saša Živkov

unread,
Jul 17, 2019, 2:34:54 PM7/17/19
to Raj Ravi, Repo and Gerrit Discussion
It is hard to say without knowing anything about your Git repo size, number of refs, etc.. 
Do you regularly gc repositories in Gerrit?
Assuming the answer is yes:
We had such cases and the cause was often that the local Git repository was never repacked (gc-ed).
Therefore, please try:
$ git gc
and then push again. Hopefully, this would resolve your issue.




time git push origin  5.1.x/master
Total 0 (delta 0), reused 0 (delta 0)

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/3b0a0f4e-d058-47b1-8b86-f5941e6b7f0f%40googlegroups.com.

Raj Ravi

unread,
Jul 19, 2019, 5:34:39 AM7/19/19
to Repo and Gerrit Discussion
Yes, I have performed the GC in both sides(working copy and remote). But still the errors are occuring.

[root@host inux-stable]# git count-objects -v
count: 0
size: 0
in-pack: 7873141
packs: 2
size-pack: 2249212
prune-packable: 0
garbage: 0
size-garbage: 0
To unsubscribe, email repo-d...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

Luca Milanesio

unread,
Jul 20, 2019, 9:10:34 AM7/20/19
to Raj Ravi, Luca Milanesio, Repo and Gerrit Discussion

On 19 Jul 2019, at 04:34, Raj Ravi <keerthish...@gmail.com> wrote:

Yes, I have performed the GC in both sides(working copy and remote). But still the errors are occuring.

[root@host inux-stable]# git count-objects -v
count: 0
size: 0
in-pack: 7873141
packs: 2
size-pack: 2249212
prune-packable: 0
garbage: 0
size-garbage: 0

It's hard to say why your operations are taking over 10m to complete: we should have more data about your system (CPU, memory, disk), Gerrit config and clients.
Which sounds more like a consultancy work rather than a Q&A for a mailing list.

I would suggest to raise the timeout, for now, and see what is really the execution time that the operation would take, maybe you're lucky and is just above 10m :-)

Luca.


More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/e331ae93-84f9-4d33-a3c7-1cb2b9c1d3d7%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages