Re: OpenSSH 7.8p1 drops SSH connection with "Broken Pipe" IMMEDIATELY after successful login

886 views
Skip to first unread message

Ohad Frenkel

unread,
Aug 28, 2018, 3:39:17 PM8/28/18
to openssh-...@mindrot.org
Hi.

Environment:

* OS: Antergos Linux (Fully updated daily) (Running as a guest on VM).
* OpenSSH: 7.8p1

Description:
Immediately upon creating a successful SSH tunnel login, the connection is dropped with the "packet_write_wait: Connection to X.X.X.X port 22: Broken pipe" error.

I've verified the issue is due to OpenSSH 7.8p1 in the following ways:

* Use a different, older client (i.e. the Windows host machine's PuTTy client, which I haven't updated god-knows-when).
* Downgrade the Antergos guest to OpenSSH 7.7p2

Both these case produced a working SSH connection.

Reproduce effort:

* Upgrade SSH client to OpenSSH 7.8p1
* Try to SSH into a server
* Some options:
* de...@test.rebex.net (use "password" for password)
* Git - any remote operation (i.e. pull, push, fetch, clone...)

The expected behavior is a consistent SSH connection.
The actual behavior is that immediately after the authentication is successful the connection is dropped with "packet_write_wait: Connection to X.X.X.X port 22: Broken pipe" error occurs, the connection is dropped (and no work can be done, e.g. clone/push to Git remote repo).


________________________________
From: Ohad Frenkel <ohadfj...@outlook.com>
Sent: 28 August 2018 21:30
To: openssh-...@mindrot.org
Subject: OpenSSH 7.8p1 drops SSH connection with "Broken Pipe" IMMEDIATELY after successful login

Hi.

Environment:

* OS: Antergos Linux (Fully updated daily) (Running as a guest on VM).
* OpenSSH: 7.8p1

Description:
Immediately upon creating a successful SSH tunnel login, the connection is dropped with the "packet_write_wait: Connection to X.X.X.X port 22: Broken pipe" error.

I've verified the issue is due to OpenSSH 7.8p1 in the following ways:

* Use a different, older client (i.e. the Windows host machine's PuTTy client, which I haven't updated god-knows-when).
* Downgrade the Antergos guest to OpenSSH 7.7p2

Both these case produced a working SSH connection.

Reproduce effort:

* Upgrade SSH client to OpenSSH 7.8p1
* Try to SSH into a server
* Some options:
* de...@test.rebex.net (use "password" for password)
* Git - any remote operation (i.e. pull, push, fetch, clone...)

The expected behavior is a consistent SSH connection.
The actual behavior is that immediately after the authentication is successful the connection is dropped with "packet_write_wait: Connection to X.X.X.X port 22: Broken pipe" error occurs, the connection is dropped (and no work can be done, e.g. clone/push to Git remote repo).
_______________________________________________
openssh-unix-dev mailing list
openssh-...@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Job Snijders

unread,
Aug 28, 2018, 4:03:27 PM8/28/18
to Ohad Frenkel, openssh-...@mindrot.org
Dear Ohad,

What virtualisation software are you using?

Kind regards,

Job

Darren Tucker

unread,
Aug 28, 2018, 4:20:17 PM8/28/18
to Ohad Frenkel, openssh-...@mindrot.org
On 28 August 2018 at 13:36, Ohad Frenkel <ohadfj...@outlook.com> wrote:
> * OS: Antergos Linux (Fully updated daily) (Running as a guest on VM).

If the VM host is VMWare Fusion, see
https://lists.mindrot.org/pipermail/openssh-unix-dev/2018-August/037145.html
wherein it appears to have a bug in its networking stack that chokes
on the new default (valid) DSCP markings. If so you can work around
it by overriding the defaults, but the problem is within VMWare (maybe
in general, or maybe just the Fusion product).

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

Ohad Frenkel

unread,
Aug 29, 2018, 1:11:11 AM8/29/18
to Darren Tucker, openssh-...@mindrot.org
The virtualization is VMware Workstation 12.x, NOT Fusion.
Again, I'd like to point that downgrading to 7.7p2 "solved" the issue and that I'm using the same virtualization software for ages (which makes me disbelieve the issue lies with it).

I've seen the comment about zero-length passwords... I'm not techie enough to understand the code there (or the technical explanation), but it is true that my previous SSH keys were password-less (if that is the meaning of "zero-length password").
I say "previous" because one of the first things I did, trying to solve this on my end, was to assume the problem was with the keys and thus generated a new set of keys, all of which have positive value of characters in the password (which, by the way, did not solve the case. As I said, only downgrading to 7.7p2 did the trick).
________________________________
From: Darren Tucker <dtu...@dtucker.net>
Sent: 28 August 2018 22:18
To: Ohad Frenkel
Cc: openssh-...@mindrot.org
Subject: Re: OpenSSH 7.8p1 drops SSH connection with "Broken Pipe" IMMEDIATELY after successful login

Zach Cheung

unread,
Aug 29, 2018, 1:19:21 AM8/29/18
to ohadfj...@outlook.com, openssh-...@mindrot.org, dtu...@dtucker.net
Using for years doesn't mean it won't be its problem, would you try to
add 'IPQoS
lowdelay throughput' to 7.8p1 sshd_config and check whether fix the problem?

On Wed, Aug 29, 2018 at 1:10 PM Ohad Frenkel <ohadfj...@outlook.com>
wrote:

Ohad Frenkel

unread,
Aug 29, 2018, 1:25:53 AM8/29/18
to Zach Cheung, openssh-...@mindrot.org, dtu...@dtucker.net
Added the line as above: 'IPQoS lowdelay throughput', restarted the SSH service, tried SSH-ing and Git-ing... same error as before: 'packet_write_wait: Connection to X.X.X.X port 22: Broken pipe'.
________________________________
From: Zach Cheung <kuroro...@gmail.com>
Sent: 29 August 2018 07:17
To: ohadfj...@outlook.com
Cc: dtu...@dtucker.net; openssh-...@mindrot.org
Subject: Re: OpenSSH 7.8p1 drops SSH connection with "Broken Pipe" IMMEDIATELY after successful login

Using for years doesn't mean it won't be its problem, would you try to add 'IPQoS lowdelay throughput' to 7.8p1 sshd_config and check whether fix the problem?

On Wed, Aug 29, 2018 at 1:10 PM Ohad Frenkel <ohadfj...@outlook.com<mailto:ohadfj...@outlook.com>> wrote:
The virtualization is VMware Workstation 12.x, NOT Fusion.
Again, I'd like to point that downgrading to 7.7p2 "solved" the issue and that I'm using the same virtualization software for ages (which makes me disbelieve the issue lies with it).

I've seen the comment about zero-length passwords... I'm not techie enough to understand the code there (or the technical explanation), but it is true that my previous SSH keys were password-less (if that is the meaning of "zero-length password").
I say "previous" because one of the first things I did, trying to solve this on my end, was to assume the problem was with the keys and thus generated a new set of keys, all of which have positive value of characters in the password (which, by the way, did not solve the case. As I said, only downgrading to 7.7p2 did the trick).
________________________________
From: Darren Tucker <dtu...@dtucker.net<mailto:dtu...@dtucker.net>>
Sent: 28 August 2018 22:18
To: Ohad Frenkel
Cc: openssh-...@mindrot.org<mailto:openssh-...@mindrot.org>
Subject: Re: OpenSSH 7.8p1 drops SSH connection with "Broken Pipe" IMMEDIATELY after successful login

On 28 August 2018 at 13:36, Ohad Frenkel <ohadfj...@outlook.com<mailto:ohadfj...@outlook.com>> wrote:
> * OS: Antergos Linux (Fully updated daily) (Running as a guest on VM).

If the VM host is VMWare Fusion, see
https://lists.mindrot.org/pipermail/openssh-unix-dev/2018-August/037145.html
wherein it appears to have a bug in its networking stack that chokes
on the new default (valid) DSCP markings. If so you can work around
it by overriding the defaults, but the problem is within VMWare (maybe
in general, or maybe just the Fusion product).

--
Darren Tucker (dtucker at dtucker.net<http://dtucker.net>)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-...@mindrot.org<mailto:openssh-...@mindrot.org>

Zach Cheung

unread,
Aug 29, 2018, 2:18:19 AM8/29/18
to ohadfj...@outlook.com, openssh-...@mindrot.org, dtu...@dtucker.net
Sorry, your problem is related to ssh client not sshd, try to add following
config to /etc/ssh/ssh_config or ~/.ssh/config:
Host *
IPQoS lowdelay throughput


On Wed, Aug 29, 2018 at 1:24 PM Ohad Frenkel <ohadfj...@outlook.com>
wrote:

> Added the line as above: 'IPQoS lowdelay throughput', restarted the SSH
> service, tried SSH-ing and Git-ing... same error as before:
> 'packet_write_wait: Connection to X.X.X.X port 22: Broken pipe'.
> ------------------------------
> *From:* Zach Cheung <kuroro...@gmail.com>
> *Sent:* 29 August 2018 07:17
> *To:* ohadfj...@outlook.com
> *Cc:* dtu...@dtucker.net; openssh-...@mindrot.org
> *Subject:* Re: OpenSSH 7.8p1 drops SSH connection with "Broken Pipe"
> IMMEDIATELY after successful login
>
> Using for years doesn't mean it won't be its problem, would you try to add
> 'IPQoS lowdelay throughput' to 7.8p1 sshd_config and check whether fix
> the problem?
>
> On Wed, Aug 29, 2018 at 1:10 PM Ohad Frenkel <ohadfj...@outlook.com>
> wrote:
>
> The virtualization is VMware Workstation 12.x, NOT Fusion.
> Again, I'd like to point that downgrading to 7.7p2 "solved" the issue and
> that I'm using the same virtualization software for ages (which makes me
> disbelieve the issue lies with it).
>
> I've seen the comment about zero-length passwords... I'm not techie enough
> to understand the code there (or the technical explanation), but it is true
> that my previous SSH keys were password-less (if that is the meaning of
> "zero-length password").
> I say "previous" because one of the first things I did, trying to solve
> this on my end, was to assume the problem was with the keys and thus
> generated a new set of keys, all of which have positive value of characters
> in the password (which, by the way, did not solve the case. As I said, only
> downgrading to 7.7p2 did the trick).
> ________________________________
> From: Darren Tucker <dtu...@dtucker.net>
> Sent: 28 August 2018 22:18
> To: Ohad Frenkel
> Cc: openssh-...@mindrot.org
> Subject: Re: OpenSSH 7.8p1 drops SSH connection with "Broken Pipe"
> IMMEDIATELY after successful login
>
> On 28 August 2018 at 13:36, Ohad Frenkel <ohadfj...@outlook.com> wrote:
> > * OS: Antergos Linux (Fully updated daily) (Running as a guest on
> VM).
>
> If the VM host is VMWare Fusion, see
>
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2018-August/037145.html
> wherein it appears to have a bug in its networking stack that chokes
> on the new default (valid) DSCP markings. If so you can work around
> it by overriding the defaults, but the problem is within VMWare (maybe
> in general, or maybe just the Fusion product).
>
> --
> Darren Tucker (dtucker at dtucker.net)
> GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
> Good judgement comes with experience. Unfortunately, the experience
> usually comes from bad judgement.
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-...@mindrot.org

Ohad Frenkel

unread,
Aug 29, 2018, 2:45:06 AM8/29/18
to Zach Cheung, openssh-...@mindrot.org, dtu...@dtucker.net
Yes, thank you!
Now with the "Host *" line it works.

Thanks again.

Sent from BlueMail<http://www.bluemail.me/r?b=13569>
On Aug 29, 2018, at 08:15, Zach Cheung <kuroro...@gmail.com<mailto:kuroro...@gmail.com>> wrote:
Sorry, your problem is related to ssh client not sshd, try to add following config to /etc/ssh/ssh_config or ~/.ssh/config:
Host *
IPQoS lowdelay throughput


On Wed, Aug 29, 2018 at 1:24 PM Ohad Frenkel < ohadfj...@outlook.com<mailto:ohadfj...@outlook.com>> wrote:
Added the line as above: 'IPQoS lowdelay throughput', restarted the SSH service, tried SSH-ing and Git-ing... same error as before: 'packet_write_wait: Connection to X.X.X.X port 22: Broken pipe'.
________________________________
From: Zach Cheung <kuroro...@gmail.com<mailto:kuroro...@gmail.com>>
Sent: 29 August 2018 07:17
To: ohadfj...@outlook.com<mailto:ohadfj...@outlook.com>
Cc: dtu...@dtucker.net<mailto:dtu...@dtucker.net>; openssh-...@mindrot.org<mailto:openssh-...@mindrot.org>
Subject: Re: OpenSSH 7.8p1 drops SSH connection with "Broken Pipe" IMMEDIATELY after successful login

Using for years doesn't mean it won't be its problem, would you try to add ' IPQoS lowdelay throughput' to 7.8p1 sshd_config and check whether fix the problem?

On Wed, Aug 29, 2018 at 1:10 PM Ohad Frenkel < ohadfj...@outlook.com<mailto:ohadfj...@outlook.com>> wrote:
The virtualization is VMware Workstation 12.x, NOT Fusion.
Again, I'd like to point that downgrading to 7.7p2 "solved" the issue and that I'm using the same virtualization software for ages (which makes me disbelieve the issue lies with it).

I've seen the comment about zero-length passwords... I'm not techie enough to understand the code there (or the technical explanation), but it is true that my previous SSH keys were password-less (if that is the meaning of "zero-length password").
I say "previous" because one of the first things I did, trying to solve this on my end, was to assume the problem was with the keys and thus generated a new set of keys, all of which have positive value of characters in the password (which, by the way, did not solve the case. As I said, only downgrading to 7.7p2 did the trick).
________________________________
From: Darren Tucker < dtu...@dtucker.net<mailto:dtu...@dtucker.net>>
Sent: 28 August 2018 22:18
To: Ohad Frenkel
Cc: openssh-...@mindrot.org<mailto:openssh-...@mindrot.org>
Subject: Re: OpenSSH 7.8p1 drops SSH connection with "Broken Pipe" IMMEDIATELY after successful login

On 28 August 2018 at 13:36, Ohad Frenkel < ohadfj...@outlook.com<mailto:ohadfj...@outlook.com>> wrote:
> * OS: Antergos Linux (Fully updated daily) (Running as a guest on VM).

If the VM host is VMWare Fusion, see
https://lists.mindrot.org/pipermail/openssh-unix-dev/2018-August/037145.html
wherein it appears to have a bug in its networking stack that chokes
on the new default (valid) DSCP markings. If so you can work around
it by overriding the defaults, but the problem is within VMWare (maybe
in general, or maybe just the Fusion product).

--
Darren Tucker (dtucker at dtucker.net<http://dtucker.net>)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-...@mindrot.org<mailto:openssh-...@mindrot.org>

Stuart Henderson

unread,
Aug 29, 2018, 4:07:16 AM8/29/18
to Ohad Frenkel, Zach Cheung, openssh-...@mindrot.org, dtu...@dtucker.net
On 2018/08/29 06:42, Ohad Frenkel wrote:
> Yes, thank you!
> Now with the "Host *" line it works.

Can you confirm the type of networking the VM guest is using please -
NAT / Bridged / Host only?

I expect this is just with NAT which is only on the "workstation" type
products (VMware Workstation and Fusion).

Zach Cheung

unread,
Aug 29, 2018, 4:16:05 AM8/29/18
to Stuart Henderson, Ohad Frenkel Keshet, openssh-...@mindrot.org, dtu...@dtucker.net
Actually I also have this ssh client problem today, have problem with NAT,
while Bridged is OK.

On Wed, Aug 29, 2018 at 4:04 PM Stuart Henderson <s...@spacehopper.org>
wrote:

Ohad Frenkel

unread,
Aug 29, 2018, 4:44:19 AM8/29/18
to Zach Cheung, dtu...@dtucker.net, openssh-...@mindrot.org, Stuart Henderson
My virtual network is, as you expected, NAT.

Sent from BlueMail<http://www.bluemail.me/r?b=13569>
On Aug 29, 2018, at 10:13, Zach Cheung <kuroro...@gmail.com<mailto:kuroro...@gmail.com>> wrote:
Actually I also have this ssh client problem today, have problem with NAT, while Bridged is OK.

Reply all
Reply to author
Forward
0 new messages