Connection to ‘xxx’ closed by remote host

[young yang]

Hello,

    I meet a problem when upgrading gerrit from 2.15 to 2.16.9, when I clone one git repository from gerrit, it will randomly pass or fail.

    Below is the detail about this issue:

#git clone ssh://jenk...@review.android.honeywell.com:29418/tools/repo.git
Cloning into 'repo'...
remote: Counting objects: 2863, done
remote: Finding sources: 100% (2863/2863)
Connection to review.android.honeywell.com closed by remote host.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

  Do you know what the problem is, and how can I fix this 

BR,

[Luca Milanesio]

Looks like a timeout error, what your gerrit.config looks like?

Luca.

[young yang]

Hello Luca,

 I checked the error_log, and it shows:

[2019-08-21 07:55:57,204] [SSH git-upload-pack /tools/repo.git (jenkinsSZ)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user jenkinsSZ account 1000083) during git-upload-pack '/tools/repo.git'
org.apache.sshd.common.channel.exception.SshChannelClosedException: write(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[jenkinsSZ@/172.31.48.13:50498]] SSH_MSG_CHANNEL_DATA) len=65520 - channel already closed
        at org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:110)
        at org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2183)
        at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:174)
        at org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:153)
        at org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:132)
        at org.eclipse.jgit.internal.storage.file.ByteArrayWindow.write(ByteArrayWindow.java:91)
        at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:561)
        at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:413)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:221)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1644)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeBase(PackWriter.java:1686)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1637)

and the gerrit.config looks like:

[sshd]
        listenAddress = *:29418
        maxConnectionsPerUser = 0
[httpd]
        sslKeyStore = etc/gerrit-android.jks
        listenUrl = https://*:8080/
[cache]
        directory = cache
[cache "web_sessions"]
        maxAge = 1y
[index]
        type = LUCENE
[receive]
        enableSignedPush = false
        maxBatchCommits = 100000

BR

[Luca Milanesio]

On 21 Aug 2019, at 09:12, young yang <njitdk...@gmail.com> wrote:

Hello Luca,

 I checked the error_log, and it shows:

[2019-08-21 07:55:57,204] [SSH git-upload-pack /tools/repo.git (jenkinsSZ)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user jenkinsSZ account 1000083) during git-upload-pack '/tools/repo.git'
org.apache.sshd.common.channel.exception.SshChannelClosedException: write(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[jenkinsSZ@/172.31.48.13:50498]] SSH_MSG_CHANNEL_DATA) len=65520 - channel already closed
        at org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:110)
        at org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2183)
        at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:174)
        at org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:153)
        at org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:132)
        at org.eclipse.jgit.internal.storage.file.ByteArrayWindow.write(ByteArrayWindow.java:91)
        at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:561)
        at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:413)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:221)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1644)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeBase(PackWriter.java:1686)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1637)

and the gerrit.config looks like:

[sshd]
        listenAddress = *:29418
        maxConnectionsPerUser = 0

You are missing the waitTimeout settings, which is 30s by default.

See:

https://gerrit-documentation.storage.googleapis.com/Documentation/3.0.1/config-gerrit.html#sshd.waitTimeout

[young yang]

Hello Luca,

    It seems it has no effect, I set the waitTimeout to 300s, and it failed again, and I also calculate the command time:

jenkins-slave@CH3UU0008:/tmp$ time git clone ssh://jenk...@review.android.honeywell.com:29418/tools/repo.git

Cloning into 'repo'...
remote: Counting objects: 2863, done
remote: Finding sources: 100% (2863/2863)
Connection to review.android.honeywell.com closed by remote host.

fatal: The remote end hung up unexpectedly1 KiB | 65.00 KiB/s
fatal: early EOFs:  63% (1804/2863), 323.91 KiB | 65.00 KiB/s
fatal: index-pack failed

real    0m9.067s
user    0m0.020s
sys     0m0.020s

[sshd]
        listenAddress = *:29418
        maxConnectionsPerUser = 0

        waitTimeout = 300s

BR,

[Luca Milanesio]

On 21 Aug 2019, at 09:33, young yang <njitdk...@gmail.com> wrote:

Hello Luca,

    It seems it has no effect, I set the waitTimeout to 300s, and it failed again, and I also calculate the command time:

jenkins-slave@CH3UU0008:/tmp$ time git clone ssh://jenk...@review.android.honeywell.com:29418/tools/repo.git
Cloning into 'repo'...
remote: Counting objects: 2863, done
remote: Finding sources: 100% (2863/2863)
Connection to review.android.honeywell.com closed by remote host.
fatal: The remote end hung up unexpectedly1 KiB | 65.00 KiB/s
fatal: early EOFs:  63% (1804/2863), 323.91 KiB | 65.00 KiB/s
fatal: index-pack failed

I see that this time you have at least some data received (323kB) whilst before it was nothing.

You are hitting now a different thing I believe: error logs?

Luca.

[young yang]

Hello Luca:

    In order to reproduce this issue, I deleted the error_log, and run the command again:

# time git clone ssh://jenk...@review.android.honeywell.com:29418/tools/repo.git

Cloning into 'repo'...
remote: Counting objects: 2863, done
remote: Finding sources: 100% (2863/2863)
Connection to review.android.honeywell.com closed by remote host.

fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

real    0m10.288s
user    0m0.048s
sys     0m0.012s

But I checked the folder log, no error_log file generated:

[gerrit@de08u1881 logs]$ ls error_log
ls: cannot access error_log: No such file or directory
[gerrit@de08u1881 logs]$ pwd
/home/gerrit/gerrit_hsm8x74/logs

BR,

[Luca Milanesio]

On 21 Aug 2019, at 09:45, young yang <njitdk...@gmail.com> wrote:

Hello Luca:

    In order to reproduce this issue, I deleted the error_log, and run the command again:

# time git clone ssh://jenk...@review.android.honeywell.com:29418/tools/repo.git
Cloning into 'repo'...
remote: Counting objects: 2863, done
remote: Finding sources: 100% (2863/2863)
Connection to review.android.honeywell.com closed by remote host.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

real    0m10.288s
user    0m0.048s
sys     0m0.012s

But I checked the folder log, no error_log file generated:

[gerrit@de08u1881 logs]$ ls error_log
ls: cannot access error_log: No such file or directory

The error_log would be on review.android.honeywell.com, not on the client.

Luca.=

[young yang]

Hello Luca:

Gerrit is in server: 10.218.130.193   and review.android.honeywell.com is the domain for this ip.

  /home/gerrit/gerrit_hsm8x74/   is the gerrit site.

[gerrit@de08u1881 gerrit_hsm8x74]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:50:56:BD:1B:F4
          inet addr:10.218.130.193  Bcast:10.218.131.255  Mask:255.255.254.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1718742101 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1433190318 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:323405752993 (301.1 GiB)  TX bytes:3782580412412 (3.4 TiB)

BR,

[Luca Milanesio]

On 21 Aug 2019, at 10:11, young yang <njitdk...@gmail.com> wrote:

Hello Luca:

Gerrit is in server: 10.218.130.193   and review.android.honeywell.com is the domain for this ip.

  /home/gerrit/gerrit_hsm8x74/   is the gerrit site.

[gerrit@de08u1881 gerrit_hsm8x74]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:50:56:BD:1B:F4
          inet addr:10.218.130.193  Bcast:10.218.131.255  Mask:255.255.254.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1718742101 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1433190318 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:323405752993 (301.1 GiB)  TX bytes:3782580412412 (3.4 TiB)

BR,

You should have an error_log in the $GERRIT_SITE/logs

Luca.

[young yang]

Hello Luca:

    Yes, it should be error_log in the $GERRIT_SITE/logs/, but I don't see any logs here, it should be a bug, any other ways I can debug with this issue?

BR

[Luca Milanesio]

On 21 Aug 2019, at 10:24, young yang <njitdk...@gmail.com> wrote:

Hello Luca:

    Yes, it should be error_log in the $GERRIT_SITE/logs/, but I don't see any logs here, it should be a bug, any other ways I can debug with this issue?

If your Gerrit doesn't produce any logs, they you have *much bigger problems*.

Have you done any testing of the upgrade in a dev environment?

Luca.

[young yang]

Hello Luca:

    I didn't do any testing of the upgrade in a dev environment, so could you tell me how to test it? I just upgraded from 2.15 to 2.16.9  two days ago.

BR

[Björn Pedersen]

Am Mittwoch, 21. August 2019 10:45:58 UTC+2 schrieb young yang:

Hello Luca:

    In order to reproduce this issue, I deleted the error_log, and run the command again:

Did you restart the gerrit instance after deleting the log?

Björn

[young yang]

Hello Björn:

    I made a mistake, I deleted the files under log folder, and gerrit.pid is deleted, so when I restart the gerrit, another gerrit restarted, no logs output. 

    Now, I killed all gerrit, and restart again, log appears.

    I attached the log, please help check, and the issue is not fixed.

BR,

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/15f8254e-6993-47ff-b770-9a3049712ae6%40googlegroups.com.

[Matthias Sohn]

On Wed, Aug 21, 2019 at 4:27 PM young yang <njitdk...@gmail.com> wrote:

Hello Björn:

    I made a mistake, I deleted the files under log folder, and gerrit.pid is deleted, so when I restart the gerrit, another gerrit restarted, no logs output. 

    Now, I killed all gerrit, and restart again, log appears.

    I attached the log, please help check, and the issue is not fixed.

The error appearing many times in your log

WARN  org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit/gerrit_hsm8x74/etc/ssh_host_ecdsa_521_key: Invalid DER: object is not an OID: SEQUENCE

looks similar to

https://groups.google.com/forum/#!msg/repo-discuss/VF5-TtBCbuM/Rp0pAcoxBwAJ 

The delete-projects and reviewers plugins didn't start, looks like you missed to upgrade them to 2.16.

Why didn't you upgrade to 2.16.9 and not to the latest 2.16 release which is 2.16.10 ?

https://www.gerritcodereview.com/2.16.html

-Matthias

[young yang]

Hi Matthias:

    Thanks for your advice, and I have installed  delete-projects and reviewers plugins , and I downloaded from https://gerrit-ci.gerritforge.com,

    The version didn't show 2.16.9, it shows v2.16-140-g1d528f2e6b and 23fea1744b, where can I get the version 2.16.9 for these two plugins?

    But the first thing I want to solve is the clone error:

#git clone ssh://jenk...@review.android.honeywell.com:29418/tools/repo.git
Cloning into 'repo'...
remote: Counting objects: 2863, done
remote: Finding sources: 100% (2863/2863)
Connection to review.android.honeywell.com closed by remote host.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

  I don't know how to solve this problem, any configuration I missed ?

BR,

[Luca Milanesio]

On 22 Aug 2019, at 03:16, young yang <njitdk...@gmail.com> wrote:

Hi Matthias:

    Thanks for your advice, and I have installed  delete-projects and reviewers plugins , and I downloaded from https://gerrit-ci.gerritforge.com,

    The version didn't show 2.16.9, it shows v2.16-140-g1d528f2e6b and 23fea1744b, where can I get the version 2.16.9 for these two plugins?

    But the first thing I want to solve is the clone error:

#git clone ssh://jenk...@review.android.honeywell.com:29418/tools/repo.git
Cloning into 'repo'...
remote: Counting objects: 2863, done
remote: Finding sources: 100% (2863/2863)
Connection to review.android.honeywell.com closed by remote host.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

  I don't know how to solve this problem, any configuration I missed ?

These are the sort of things you may want to test before you go live with a new release.

Matthias has highlighted that your upgraded Gerrit has many issues, at times one is the cause of the other.

If you want to start to tackle the Git clone over SSH, I would recommend:

a) Remove the plugins that aren't working

b) Fix the SSH keys problem on the logs

c) Enable verbose SSH logging on the Git client

GIT_SSH_COMMAND="ssh -vvv" git clone ...

And then tackle all the problems, one by one.

It isn't a simple configuration switch, Gerrit and Git are a bit more complex I'm afraid.

Luca.

BR,

On Thu, Aug 22, 2019 at 3:19 AM Matthias Sohn <matthi...@gmail.com> wrote:

On Wed, Aug 21, 2019 at 4:27 PM young yang <njitdk...@gmail.com> wrote:

Hello Björn:

    I made a mistake, I deleted the files under log folder, and gerrit.pid is deleted, so when I restart the gerrit, another gerrit restarted, no logs output. 

    Now, I killed all gerrit, and restart again, log appears.

    I attached the log, please help check, and the issue is not fixed.

The error appearing many times in your log

WARN  org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit/gerrit_hsm8x74/etc/ssh_host_ecdsa_521_key: Invalid DER: object is not an OID: SEQUENCE

looks similar to

https://groups.google.com/forum/#!msg/repo-discuss/VF5-TtBCbuM/Rp0pAcoxBwAJ 

The delete-projects and reviewers plugins didn't start, looks like you missed to upgrade them to 2.16.

Why didn't you upgrade to 2.16.9 and not to the latest 2.16 release which is 2.16.10 ?

https://www.gerritcodereview.com/2.16.html

-Matthias

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAMq6a3y8bBfE%3DkP7Lw4J3QNvMQ9maaKKULvyxYuniJWbBOGekg%40mail.gmail.com.

[young yang]

Hi Luca:

    I did as you said, I upgraded gerrit from 2.16.9 to 2.16.10, and no key warning appears now. 

    I tried again, and get the error message from errror_log, could you help me what the problem is?

#git clone "ssh://h12...@review.android.honeywell.com:29418/tools/repo"
Cloning into 'repo'...
remote: Counting objects: 2867, done
remote: Finding sources: 100% (2867/2867)

Connection to review.android.honeywell.com closed by remote host.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

error logs :

    [2019-08-22 16:10:52,130] [sshd-SshServer[578198d9]-nio2-thread-12] WARN  org.apache.sshd.server.session.ServerSessionImpl : exceptionCaught(ServerSessionImpl[jenkinsSZ@/172.31.48.11:39420])[state=Opened] SshChannelNotFoundException: Received SSH_MSG_CHANNEL_WINDOW_ADJUST on unknown channel 0

[2019-08-22 16:11:04,374] [sshd-SshServer[578198d9]-nio2-thread-8] WARN  org.apache.sshd.server.session.ServerSessionImpl : exceptionCaught(ServerSessionImpl[h127301@/172.31.48.9:35691])[state=Opened] SshChannelNotFoundException: Received SSH_MSG_CHANNEL_WINDOW_ADJUST on unknown channel 0

BR,

[Luca Milanesio]

On 22 Aug 2019, at 17:14, young yang <njitdk...@gmail.com> wrote:

Hi Luca:

    I did as you said, I upgraded gerrit from 2.16.9 to 2.16.10,

Did I say that?

and no key warning appears now. 

    I tried again, and get the error message from errror_log, could you help me what the problem is?

#git clone "ssh://h12...@review.android.honeywell.com:29418/tools/repo"
Cloning into 'repo'...
remote: Counting objects: 2867, done
remote: Finding sources: 100% (2867/2867)
Connection to review.android.honeywell.com closed by remote host.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

I've suggested to run: