Unable to connect via SSH: Bad packet length

[Fabiano Rosas]

After sending changes to Gerrit for review, all subsequent connections via ssh
fail with:

Bad packet length 1133989372.
Disconnecting: Packet corrupt

The log shows several instances of:

WARN com.google.gerrit.sshd.GerritServerSession : Exception caught
org.apache.sshd.common.SshException: Invalid packet length: -285787426

This happens every day and after it happens I need to restart Gerrit
to get it working again.

I run Gerrit 2.11 on a VM and the only listener on stream events is Jenkins
(v1.629) using the Gerrit-trigger plugin (v2.17.2). The Jenkins' machine is a VM
on the same host.

I've found this thread from mina-users which describes a somewhat similar
problem, but I'm not sure if it is related: http://goo.gl/5awoju

I'll attach a log with debug level set.

[Alex Blewitt]

If they always happen via the authentication stage (when the client is sending keys) then you should check to see what address the connection is being made from. I’m not sure if this is available in the server log, or whether you have to increase the debug logging to make that happen.

If the SSH client is sending a corrupt packet to the server, it’s possible that the exception results in the Apache SSHD threads being in a state where they don’t receive any further input; if so, there’s probably something that Gerrit needs to do to monitor the state of the thread pool.

Alternatively you can use the ‘show-connections’ SSH command into the Gerrit server, and if they are all hung, use the ‘close-connection’ to see if you can bring the server back into play without needing a restart.

Alex

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<badpacket.log>

[Fabiano Rosas]

Hi, thanks for replying.

The log shows different addresses trying to connect each time this issue happens, I don't think it's one particular client that is causing this.

As for the show-connections command, I can't use it because it is a SSH command and I cannot connect to Gerrit via ssh in any way after the problem happens.

[Fabiano Rosas]

Upon further investigation, I was able to consistently reproduce this
by running:

while true; do ssh user@localhost -p 29418 &> /dev/null; done

It will break after the 109th connection.

I have also run gerrit show-connections in the loop and it didn't show
any stale connections.