VPN MTU limit breaks ssh connection to openssh 6.2p2 server

[Ernst Kratschmer]

Dear openssh developer,

I want to use a Win7 client with putty to access a Linux host running an
openssh 6.2p2 through a VPN connection. These connection worked relatively
flawless with all versions of openssh up until openssh 6.1p1. Since the
openssh 6.2p2 upgrade the ssh connection fail consistently with a message:
Network error, connection reset by peer. After inspecting the tcp
transmission between the putty client and openssh server it appears that
the maximum MTU limit of 1362 of the VPN connection, causes the server to
break the 1460 byte cipher string into two packets. At that point the ssh
client, putty or openssh, resets the connection.

This was not a problem with openssh 6.1p1 since the cipher string was only
1106 bytes and therefore transmitted in one packet. Since I am stuck with
the VPN MTU limit of 1362, I am hoping that you could help in some form to
get the ssh connection working again by, e.g., limit the cipher string to
less than 1322 bytes?


Thanks in advance for any help you can provide to fix this problem,

Ernst Kratschmer

_______________________________________________
openssh-unix-dev mailing list
openssh-...@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

[Damien Miller]

On Thu, 7 Nov 2013, Ernst Kratschmer wrote:

> Dear openssh developer,
>
> I want to use a Win7 client with putty to access a Linux host running an
> openssh 6.2p2 through a VPN connection. These connection worked relatively
> flawless with all versions of openssh up until openssh 6.1p1. Since the
> openssh 6.2p2 upgrade the ssh connection fail consistently with a message:
> Network error, connection reset by peer. After inspecting the tcp
> transmission between the putty client and openssh server it appears that
> the maximum MTU limit of 1362 of the VPN connection, causes the server to
> break the 1460 byte cipher string into two packets. At that point the ssh
> client, putty or openssh, resets the connection.
>
> This was not a problem with openssh 6.1p1 since the cipher string was only
> 1106 bytes and therefore transmitted in one packet. Since I am stuck with
> the VPN MTU limit of 1362, I am hoping that you could help in some form to
> get the ssh connection working again by, e.g., limit the cipher string to
> less than 1322 bytes?

I don't think this is something we can fix in OpenSSH. You could work
around it by shrinking the list of ciphers/MACs/key exchange algorithms
that are offered using sshd_config's "Ciphers", "MACs" and "KexAlgorithms"
options.

-d

[Flavien Lebarbe]

> On Thu, 7 Nov 2013, Ernst Kratschmer wrote:
>
> > Dear openssh developer,
> >
> > I want to use a Win7 client with putty to access a Linux host running an
> > openssh 6.2p2 through a VPN connection. These connection worked relatively
> > flawless with all versions of openssh up until openssh 6.1p1. Since the
> > openssh 6.2p2 upgrade the ssh connection fail consistently with a message:
> > Network error, connection reset by peer. After inspecting the tcp
> > transmission between the putty client and openssh server it appears that
> > the maximum MTU limit of 1362 of the VPN connection, causes the server to
> > break the 1460 byte cipher string into two packets. At that point the ssh
> > client, putty or openssh, resets the connection.
> >
> > This was not a problem with openssh 6.1p1 since the cipher string was only
> > 1106 bytes and therefore transmitted in one packet. Since I am stuck with
> > the VPN MTU limit of 1362, I am hoping that you could help in some form to
> > get the ssh connection working again by, e.g., limit the cipher string to
> > less than 1322 bytes?

Damien answered :

> I don't think this is something we can fix in OpenSSH.

I don't get it. Could you elaborate on this ?

Please correct me if I'm wrong, but from my point of view, TCP
splitting packets to respect MTU somewhere in transit should not have
any impact at the functionnal level. We should not expect the values
returned by read()/recv() to match those we gave to write()/send().
When read()-ing on a socket, the size returned is at the kernel (or
libc) discretion. If it prefers to send us the data byte by byte, it
is allowed to, and the process is supposed to wait for more data if
it expects more.

I'm not telling it's easy to implement in OpenSSH as I never seriously
looked at the code but saying it cannot be done surprises me.

Thanks for your clues !

Flavien.

[Darren Tucker]

On Fri, Nov 08, 2013 at 11:52:11AM +0100, Flavien Lebarbe wrote:
> > On Thu, 7 Nov 2013, Ernst Kratschmer wrote:

[vpn + mtu problem]

> Damien answered :
> > I don't think this is something we can fix in OpenSSH.
>
> I don't get it. Could you elaborate on this ?
>
> Please correct me if I'm wrong, but from my point of view, TCP
> splitting packets to respect MTU somewhere in transit should not have
> any impact at the functionnal level. We should not expect the values
> returned by read()/recv() to match those we gave to write()/send().

it doesn't. it does expect that they make it to the other end, though,
which doesn't seem to happen in this case.

> When read()-ing on a socket, the size returned is at the kernel (or
> libc) discretion. If it prefers to send us the data byte by byte, it
> is allowed to, and the process is supposed to wait for more data if
> it expects more.
>
> I'm not telling it's easy to implement in OpenSSH as I never seriously
> looked at the code but saying it cannot be done surprises me.

TCP is supposed to be a bi-directional reliable byte stream and the ssh
protocol doesn't rely on it being anything more than that.

The network in this case seems broken. It sounds like the classic "MTU
causes IP fragmentation then something else drops the fragments" problem.
Often something doing this will also reduce the TCP MSS so that the
endpoints don't emit IP packets big enough to fragment, but it seems
not in this case.

It can be worked around in the config as described by Damien but any
attempt to work around it in openssh would be an ugly hack at best and
would likely hurt performace on networks that aren't broken.

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

[Ernst Kratschmer]

Hi everybody,

thank you very much for your quick reply.

Let me first address Damien's comment. Following some other people's
"trial and errors" around the internet, I have already started to look for
ways to limit the "cipher string", but did not have any success. So far, I
tried using a config file in ~/.ssh/ which specified just a few Ciphers
and MACs. My debug showed that it did not change the packet length, still
1460. Next I took a look at /etc/ssh/sshd_config and noticed that it did
not explicitly specify any Ciphers or MACs, but ssh_config did. I then
copied the few Ciphers and MACs from ssh_config to sshd_config and
restarted sshd. Again, my debug showed that it did not change the packet
length, still 1460. Most likely, I am doing something wrong here, but in
any case I would greatly appreciate it if you could provide me with
instructions how to shorten the "cipher string" length by modifying the
config files.

If I understand Darren correctly, he is concerned that the packet
fragmentation causes fragments to get dropped. From my debug, I can see
that the 1460 byte packet get split into two packets exactly as expected
from the MTU limit. And both packets get sent to the client. The client
then resets the communication. This happens regardless of the type of
client software used, Cygwin, putty, openssh any version, etc. Therefore,
I wonder if Flavien's comment is correct, or, if somehow all ssh clients
expect to reveice the "cipher string" in a single packet. I admit that
this is a wild speculation from someone who hasn't examined the source
code. I just find it remarkable, that all ssh clients I tried show the
same behavior in this case. I hope you know the answer.


-Ernst

[Alex Bligh]

On 8 Nov 2013, at 15:26, Ernst Kratschmer wrote:

> If I understand Darren correctly, he is concerned that the packet
> fragmentation causes fragments to get dropped. From my debug, I can see
> that the 1460 byte packet get split into two packets exactly as expected
> from the MTU limit.

Run a ping with DF set across your tunnel and sweep the packet size. This
should stop working at one particular packet size.

Now run a ping without DF set across your tunnel and repeat. This should
work for all packet sizes. I bet one particular size fails. This will be
breaking pMTU discovery.

--
Alex Bligh

[Peter Stuge]

Ernst Kratschmer wrote:
> From my debug, I can see that the 1460 byte packet get split into
> two packets exactly as expected from the MTU limit.

Nobody is challenging this.

> And both packets get sent to the client.

Nor this. But what really matters is that they are both well-formed
and that they *arrive* at the client. I'm not sure you've provided
information about whether that is the case.

> The client then resets the communication.

If either packet is malformed then that's the right thing to do.

> This happens regardless of the type of client software used,
> Cygwin, putty, openssh any version, etc.

As you know, TCP is not implemented by the client software, but by
the operating system network stack. And given an invalid packet most
operating systems will do the same (right) thing.


Please analyze what the client receives, if it is two packets then
investigate whether they are well-formed. If they are, then look
further up the layers.


//Peter

[Ernst Kratschmer]

Hi Alex,

I am sorry, but I don't know what "ping with DF set" is.
I can run "ping -l size". This fails if size is greater than 1252, over
VPN or LAN.
Not sure if this helps.


-Ernst

From:
Alex Bligh <al...@alex.org.uk>
To:
Ernst Kratschmer/Watson/IBM@IBMUS,
Cc:
Alex Bligh <al...@alex.org.uk>, dtu...@zip.com.au, Damien Miller
<d...@mindrot.org>, openssh-...@mindrot.org
Date:
11/08/2013 11:11 AM
Subject:
Re: VPN MTU limit breaks ssh connection to openssh 6.2p2 server

On 8 Nov 2013, at 15:26, Ernst Kratschmer wrote:

> If I understand Darren correctly, he is concerned that the packet

> fragmentation causes fragments to get dropped. From my debug, I can see

> that the 1460 byte packet get split into two packets exactly as expected

> from the MTU limit.

Run a ping with DF set across your tunnel and sweep the packet size. This
should stop working at one particular packet size.

Now run a ping without DF set across your tunnel and repeat. This should
work for all packet sizes. I bet one particular size fails. This will be
breaking pMTU discovery.

--
Alex Bligh

[Peter Stuge]

Ernst Kratschmer wrote:
> I am sorry, but I don't know what "ping with DF set" is.
> I can run "ping -l size". This fails if size is greater than 1252, over
> VPN or LAN.
> Not sure if this helps.

On Linux:

$ man ping | grep -A 1 -B 5 DF

-M pmtudisc_opt
Select Path MTU Discovery strategy. pmtudisc_option may be
either do (prohibit fragmentation, even local one), want (do
PMTU discovery, fragment locally when packet size is large), or
dont (do not set DF flag).

$


//Peter

[Eitan Adler]

On Fri, Nov 8, 2013 at 12:33 PM, Ernst Kratschmer <ern...@us.ibm.com> wrote:
> Hi Alex,

[ top posting makes it difficult to follow ]

> I am sorry, but I don't know what "ping with DF set" is.

DF is the "Don't Fragment" bit

> I can run "ping -l size". This fails if size is greater than 1252, over
> VPN or LAN.

-l is not useful here. You may want to use -g and -G.
By default the DF bit is not set. You may use the -D flag to set it.

> Not sure if this helps.

--
Eitan Adler

[Ernst Kratschmer]

Hi Eitan et al.,

at the moment I only have Fedora 18 / 19 servers and a Win7 client behind
the firewall connecting via VPN available for testing. Neither system
accepts a "ping -DF" command. From the Win7 client I can do a "ping -l
size" (-l size Send buffer size.) or "ping -f -l size" (-f Set Don't
Fragment flag in packet (IPv4-only).). In either case the maximum number
of bytes I can pack into the ping request is 1252. This 1252 byte limit is
the same for sending the ping request over VPN or over the LAN (I lost you
as far as the purpose of this test is concerned).

I did some additional testing and noticed the following. With openssh
6.1p1 the "cipher string" gets broken up into two packet which add up to
exactly 840 bytes (same as the single packet at large MTU) when I set the
server to a smaller MTU size of e.g. 1200 (<<1362 VPN limit). With openssh
6.2p2 the "cipher string" gets broken up into two packet which add up to
1464 bytes, which is different from 1460 bytes of the single packet at
large MTU, when I set a server MTU size of e.g. 1200. This 4 byte
difference only happens with openssh 6.2p2, when the "cipher string" gets
split into two packets.

Things get even more weird with openssh 6.2p2 and an MTU of e.g. 600. The
"cipher string" gets broken up into 1160 and 384 bytes (again 4 more than
the original 1460), and not into three packets as I had expected. After
the 384 byte packet has been send from the server, both client and server
engage in a long list of resets (R flag set).


-Ernst

From:
Eitan Adler <li...@eitanadler.com>
To:
Ernst Kratschmer/Watson/IBM@IBMUS,
Cc:

Alex Bligh <al...@alex.org.uk>, "openssh-...@mindrot.org"
<openssh-...@mindrot.org>, Damien Miller <d...@mindrot.org>, Darren
Tucker <dtu...@zip.com.au>
Date:
11/08/2013 12:53 PM

Subject:
Re: VPN MTU limit breaks ssh connection to openssh 6.2p2 server

[twija...@gmail.com]

Hi Ernst,
The only way to rectify this is to reduce the alogrithm used.
OpenSSH is not the only one having this issue. Other proprietary product also having the same issue you encounter.