Error when run repo sync.

[SeongUk Baek]

Hello, I'm getting an error when doing repo sync. git over ssh from Gerrit server.

On the client, I see this:

Total 30706 (delta 14), reused 23250 (delta 14)

Disconnecting: Protocol error: expected packet type 31, got 94

fatal: The remote end hung up unexpectedly

fatal: The remote end hung up unexpectedly

fatal: early EOF

fatal: early EOF

fatal: The remote end hung up unexpectedly

fatal: early EOF

fatal: index-pack failed

fatal: index-pack failed

fatal: index-pack failed

I'm using Gerrit 2.8.5.

Any suggestions would be much appreciated.

Thank you.

[Makson Lee]

We have exactly the same problem, if we try "repo sync", everything is fine, if we try "repo sync -j12", the problem happens.

[Dave Castagna (Motorola Mobility)]

Yeah we did exhaustive repo sync tests with 2.8.x releases - particularly 2.8.4 and 2.8.5.

Both of these releases seemed to perform well but when you pushed the repo command to higher thread counts things degraded quickly.  It is unclear if it is a problem with the repo code or with Gerrit but it does seem like it is the repo code.

Have you tried adding the -q ("quite mode") option to the "repo sync" command?  Do you get better results?

[Makson Lee]

No, got the same result even adding the -q option.

[Ian Kumlien]

We've had people seeing this here as well, since it's a ssh problem the client is seeing SSH_MSG_CHANNEL_DATA instead of a key-exchange message..

(Any of:

#define SSH2_MSG_KEXDH_REPLY 31
#define SSH2_MSG_KEX_DH_GEX_GROUP 31
#define SSH2_MSG_KEX_ECDH_REPLY 31
#define SSH2_MSG_KEX_ROAMING_AUTH_REQUIRED 31 )

This is about how far i got, haven't had time to look at anything else and haven't had anything that could reliably trigger it either.

If you run tests and see this, I wonder if it could be related to the master control socket (the -M -N options to the first connection created by repo)...

--

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Makson Lee]

I added the following content into ~/.ssh/config, and the problem gone, so this is a ssh client issue?

Host *

ControlMaster auto

ControlPath ~/.ssh/sockets/%r@%h-%p

ControlPersist 600

[Gustaf Lundh]

Pushed a workaround fix here:

https://gerrit-review.googlesource.com/#/c/60574/

Ian K. was on the right track.

Every now and then, sshd mina will issue a "rekey:ing" on the control master. Due to some synchronization issues, (especially during high load), the expected reply packet for the key re-exchange will arrive after a data message.

When the packet arrives out of order, the ControlMaster will disconnect and all ongoing git fetches that are muxed over the channel.

I see in the commit log for SSHD Mina that they have made efforts to solve this issue, but apparently it is completely fixed yet.

Using the change above you can turn off the time trigger limit and also set a very high bytes trigger limit, essentially disable this part of SSHD Mina.

We have verified that it solves the "Disconnecting: Protocol error: expected packet type 31, got 94"-issue for us.

Best regards

Gustaf

[Lundh, Gustaf]

> but apparently it is completely fixed yet.

 

Should of course be “but apparently it is _not_ completely fixed yet.”

 

/Gustaf

--

[Chunlin Zhang]

I am using gerrit 2.11.1, this issue happen again and again with 1 particular git project. But happened when run repo sync -j8 or more, if repo sync -j1 or use git directly, it will not happen.

```

Disconnecting: Protocol error: expected packet type 31, got 94

fatal: The remote end hung up unexpectedly

fatal: The remote end hung up unexpectedly

fatal: The remote end hung up unexpectedly

fatal: The remote end hung up unexpectedly

fatal: The remote end hung up unexpectedly

fatal: The remote end hung up unexpectedly

fatal: 过早的文件结束符（EOF）

fatal: 过早的文件结束符（EOF）

fatal: 过早的文件结束符（EOF）

fatal: The remote end hung up unexpectedly

fatal: The remote end hung up unexpectedly

fatal: 过早的文件结束符（EOF）

fatal: The remote end hung up unexpectedly

fatal: The remote end hung up unexpectedly

fatal: early EOF

fatal: 过早的文件结束符（EOF）

fatal: Could not read from remote repository.

Please make sure you have the correct access rights

and the repository exists.

fatal: Could not read from remote repository.

Please make sure you have the correct access rights

and the repository exists.

fatal: 过早的文件结束符（EOF）

fatal: 过早的文件结束符（EOF）

fatal: 过早的文件结束符（EOF）

fatal: 过早的文件结束符（EOF）

fatal: recursion detected in die handler

```

在 2014年10月8日星期三 UTC+8下午3:49:42，Gustaf Lundh写道：

[Lundh, Gustaf]

Hi Chunlin,

 

Try setting:

 

[sshd]

                             rekeyTimeLimit = 24h

                             rekeyBytesLimit = 1099511627776

 

in gerrit.config for a workaround. If you are not Gerrit Admin, you can turn off the ControlMaster on your machine. For instance by adding the following lines to ~/.ssh/config on your machine (please know this will turn of the ControlMaster for all servers).

 

Host *

ControlMaster no

 

Best regards

Gustaf

[Chunlin Zhang]

Gustaf,

Yes, I am gerrit admin, I will try the sshd config.

And I try to set ControlMaster = no already, it doesn't work.

[Chunlin Zhang]

Gustaf,

The sshd config workaround seems work.

I don't see this error message again after change the config.

在 2015年8月20日星期四 UTC+8下午4:49:24，Chunlin Zhang写道：

Gustaf,

Yes, I am gerrit admin, I will try the sshd config.

And I try to set ControlMaster = no already, it doesn't work.