MULTINET SET/KERNEL TCP_DO_MTU_DISCOVERY 0
CATTANEO Marco
I would like some information about the undocumented command in Multinet
version 4.0
$ MULTINET SET/KERNEL TCP_DO_MTU_DISCOVERY 0
A colleague has discovered that if he does a TCP/IP transfer from a
machine on FDDI to a machine on Ethernet, interconnected by a DEC NI3
bridge, the transfer proceeds in bursts of 4000 bytes, one every 30
seconds, if TCP_DO_MTU_DISCOVERY is set to the default value of 1 on
the FDDI machine, but proceeds at a reasonable speed when the above
Multinet command is entered on the FDDI machine.
The FDDI machine is running
CISCO MultiNet V4.0 Rev A, AlphaServer 1000 4/200, OpenVMS AXP V6.2
The Ethernet machine is running
CISCO MultiNet V4.0 Rev A, AlphaStation 200 4/166, OpenVMS AXP V6.2
What exactly does this command do, and why is it necessary?
--
Marco Cattaneo / ECP-ALE, CERN, CH-1211 Geneva 23, Switzerland
--------------------------------------------------------------
Telephone: +41-22-767-4046 mailto:Marco.C...@cern.ch
Fax: +41-22-783-0672 http://alephwww.cern.ch/~cattanem/
Aa...@cisco.com
> I would like some information about the undocumented command in Multinet
> version 4.0
> $ MULTINET SET/KERNEL TCP_DO_MTU_DISCOVERY 0
Well, it turns off path MTU discovery (PMTUD), so that packets are
not sent out with the DF flag set, so that they can be
fragmented.
> A colleague has discovered that if he does a TCP/IP transfer from a
> machine on FDDI to a machine on Ethernet, interconnected by a DEC NI3
> bridge, the transfer proceeds in bursts of 4000 bytes, one every 30
> seconds, if TCP_DO_MTU_DISCOVERY is set to the default value of 1 on
> the FDDI machine, but proceeds at a reasonable speed when the above
> Multinet command is entered on the FDDI machine.
> The FDDI machine is running
> CISCO MultiNet V4.0 Rev A, AlphaServer 1000 4/200, OpenVMS AXP V6.2
> The Ethernet machine is running
> CISCO MultiNet V4.0 Rev A, AlphaStation 200 4/166, OpenVMS AXP V6.2
> What exactly does this command do, and why is it necessary?
The article below explains all the issues involved here.
In the configuration above, I recommend that both PMTUD
and TCP timestamps be disabled.
Best,
Aaron
Aaron Leonard aa...@cisco.com
cisco MultiNet for OpenVMS Support
---
Configuring a FDDI Interface on a LAN Bridged to an Ethernet
(Skip to Appendix A - "the bottom line" - if you just want
to see the prescription.)
A variety of problems can arise when using a LAN that is
bridged between FDDI and Ethernet. These problems are
due to the fact that the maximum frame size supported on
a FDDI ring is 4500 bytes, while an Ethernet only supports
1518-byte frames. You must make sure that, when a FDDI-attached
station is sending data, you don't wind up with frames getting
discarded because they're too large for the Ethernet to accept.
The problems will typically manifest as small packets sent
from a FDDI station to an Ethernet station getting thru, but
large packets failing. Therefore, when using such a LAN,
you must be careful to configure all FDDI stations correctly.
The relevant parameters to configure are:
- the IP MTU on the FDDI interface
- whether IP Path MTU discovery (kernel variable
TCP_DO_MTU_DISCOVERY) is in effect
- whether TCP timestamps (kernel variable TCP_DONT_TSECHO)
are in effect. (This impacts the size of the TCP header,
which can cause problems with MSS negotiation.)
The optimal configuration depends on the nature of the
FDDI-Ethernet bridge being used. Most FDDI-Ethernet
bridges (including ones from cisco and Digital) will,
when presented with an IP packet that needs to be forwarded
from the FDDI to the Ethernet, but which is too large for
the Ethernet MTU, perform IP fragmentation, allowing
the IP packet to reach the Ethernet destination. Some
other bridges are incapable of performing IP fragmentation,
and will simply discard such too-large frames.
1. IP MTU setting.
If the FDDI-Ethernet bridge in use can't do IP fragmentation,
then ALL stations on this FDDI *must* set their IP MTU
to 1500. Otherwise, they will try to send packets that
are too large to reach the Ethernet-attached stations.
Note that, if FDDI stations are restricted to an MTU of
1500 bytes, this may significantly impair data transfer
performance between FDDI stations, or increase host CPU
load. This is why it is a mistake to deploy FDDI-Ethernet
bridges that are incapable of IP fragmentation.
On FDDI LANs with bridges that CAN perform IP fragmentation
(or, for that matter, on FDDI LANs that are NOT bridged into
Ethernets), all attached FDDI interfaces should use the IP
MTU of 4352. This is required by RFC-1390. Unfortunately,
versions of MultiNet prior to V4.0 Rev A incorrectly set
the default MTU for FDDI interfaces to 4470.
Thus:
If using a fragmentation-incapable bridge:
Initialize all FDDI interfaces with an MTU of 1500
Else
If using MultiNet V3.5
Apply the SET_INTER_MTU-035 ECO
or
Initialize the FDDI interface with an MTU of 4352
Else (running V4.0 Rev A or above)
You're OK - default MTU is 4352
2. Path MTU Discovery
The Path MTU Discovery feature is designed to allow IP dynamically
to discover the MTU for a given path to a given destination.
However, it doesn't work if the destination is on a locally
attached interface. Therefore, if using a fragmentation-capable
bridge, you should be sure to disable this feature. Otherwise,
Path MTU Discovery will emit packets too large for the Ethernet
with the DF (Don't Fragment) bit set, which will cause the bridge
to discard them rather than forwarding them.
3. TCP Timestamps and Effective Send MSS Calculation.
Due to a bug in the MultiNet kernel (present through V4.0 Rev A),
if TCP timestamps are enabled (as they are by default), this will
cause us to miscalculate the effective send MSS for a packet
being sent to an Ethernet-attached station if our interface onto
that LAN has an MTU above 1500. This results in our sending packets
that are too large to fit on the destination Ethernet, resulting
in transmission failure (if Path MTU Discovery is enabled) or in
degraded performance (if Path MTU Discovery is disabled.)
The solution (thru V4.0 Rev A) is to disable TCP timestamps
(See appendix C below for a detailed explanation of the bug.)
Appendix A. The Bottom Line.
For a system with a FDDI interface that is on a LAN bridged
to an Ethernet:
If using an bridge that can't fragment:
Set the MTU on the FDDI interface to 1500
Else (using a bridge that CAN fragment):
Set the MTU to 4352 (if running a version prior to V4.0 Rev A)
Turn off Path MTU Discovery
Turn off TCP timestamps
Appendix B. How To Perform the Desired Configuration Changes
B.1. Setting a non-default MTU on an interface
Create a custom SEn_CONFIGURE.COM (or VEn_CONFIGURE.COM).
For example, let's say that the interface is SE1 and
that the desired MTU is 1500:
$! sample multinet:se1_configure.com
$! see multinet_root:[multinet.examples]se0_configure.com for comments
$!
$ command = "MULTINET SET/INTERFACE SE1 /LINK_LEVEL=ETHERNET" + -
"/MTU=1500 /PROTOCOL=" + P1
$ If P2 .NES. "NONE" THEN Command = Command + "/Address=" + P2
$ If P3 .NES. "NONE" THEN Command = Command + "/VMS_Device=" + P3
$!
$ If P1 .Nes. "IP" Then Goto Do_Command
$!
$ If P4 .NES. "NONE" THEN Command = Command + "/IP_Subnet=" + P4
$ If P5 .NES. "NONE" THEN Command = Command + "/IP_Broadcast=" + P5
$ If P6 .NES. "NONE" THEN Command = Command + "/Point_To_Point=" + P6
$ If P7 .NES. "NONE" THEN Command = Command + "/Peer=" + P7
$ If P8 .NES. "NONE" THEN Command = Command + "/Hardware_Dev=" + P8
$!
$ Do_Command:
$ Write Sys$Output F$Environment("PROCEDURE")," ---> ",Command
$ 'command'
Then reboot.
B.2. Setting kernel variables.
To set a kernel variable on the running system:
$ MULTINET SET/KERNEL variablename newvalue
To have kernel variable settings be in effect upon the
next boot, place the MULTINET SET/KERNEL command(s) into
MULTINET:LOCAL_ROUTES.COM.
B.2.1. Turning off Path MTU Discovery:
$ MULTINET SET/KERNEL TCP_DO_MTU_DISCOVERY 0
B.2.2. Turning off TCP timestamps:
$ MULTINET SET/KERNEL TCP_DONT_TSECHO 1
B.3. Fetching and installing the SET_INTER_MTU-035 ECO for V3.5
$ FTP ECO.TGV.CISCO.COM /USER=ANONYMOUS /PASSWORD=SET_INTER_MTU-035 -
GET ECO-SET_INTER_MTU035.A ! or .ZIP if you have UNZIP
$ @SYS$UPDATE:VMSINSTAL ECO-SET_INTER_MTU035 dev:[dir]
Then reboot.
Appendix C. TCP Timestamps and Effective Send MSS Calculation Bug
(CR-001903)
The bug works like this:
With the TCP timestamp option enabled, we will send TCP segments
with a 20-byte IP header and a 32-byte TCP header (as opposed to
the 20-byte TCP header that would be sent if no TCP options were
neabled.)
Let's say that we have TCP timestamps enabled, that our FDDI interface
has an MTU of 4352, that our FDDI is bridged to an Ethernet, and that
the Ethernet-attached station we're talking to is using the
normal TCP headersize of 20 bytes, and the normal Ethernet MTU
of 1500.
RFC-1122 has the following formula that governs our effective
send MSS (sec. 4.2.2.6):
The maximum size of a segment that TCP really sends, the
"effective send MSS," MUST be the smaller of the send MSS
(which reflects the available reassembly buffer size at the
remote host) and the largest size permitted by the IP layer:
Eff.snd.MSS =
min(SendMSS+20, MMS_S) - TCPhdrsize - IPoptionsize
where:
* SendMSS is the MSS value received from the remote host,
or the default 536 if no MSS option is received.
* MMS_S is the maximum size for a transport-layer message
that TCP may send.
* TCPhdrsize is the size of the TCP header; this is
normally 20, but may be larger if TCP options are to be
sent.
* IPoptionsize is the size of any IP options that TCP
will pass to the IP layer with the current message.
In this case, the Ethernet node will advertise an MSS (our
SendMSS) of 1460 (i.e. its MTU of 1500 less its IP+TCP header size of 40.)
We will have an MSS_S of 4300 (our MTU of 4352 less an IP header
size of 20 less a TCP header size of 32 [which is the normal
20-byte TCP header + 12 bytes for the TSECHO option.].)
Thus we should calculate
Eff.snd.MSS
= min(1460+20, 4300) - 32 - 0
= 1448
Unfortunately, we fail to perform the correct calculation, because
we ignore the size of our TCP options, and come up with an
Eff.snd.MSS of 1460. This yields an IP packet of length
1460 + 32 (TCP header) + 20 (IP header) = 1512, which is too
big to reach the destination Ethernet unfragmented.
This bug is supposedly fixed in V4.0 Rev A if the
TCP_MSS_ROOM_FOR_OPTIONS kernel variable is turned on,
but this turns out not to work.
Dan Wing
>I would like some information about the undocumented command in Multinet
>version 4.0
>
>$ MULTINET SET/KERNEL TCP_DO_MTU_DISCOVERY 0
>
>machine on FDDI to a machine on Ethernet, interconnected by a DEC NI3
>bridge, the transfer proceeds in bursts of 4000 bytes, one every 30
>seconds, if TCP_DO_MTU_DISCOVERY is set to the default value of 1 on
>the FDDI machine, but proceeds at a reasonable speed when the above
>Multinet command is entered on the FDDI machine.
>
>The FDDI machine is running
>CISCO MultiNet V4.0 Rev A, AlphaServer 1000 4/200, OpenVMS AXP V6.2
>
>The Ethernet machine is running
>CISCO MultiNet V4.0 Rev A, AlphaStation 200 4/166, OpenVMS AXP V6.2
>
>What exactly does this command do,
The command disables Path MTU Discovery, which is described in RFC1191.
RFC1191 also explains the behavior you're seeing (bursts every 30
seconds).
>and why is it necessary?
It isn't "necessary" - the transfer still works even with Path MTU
Discovery turned on (which is the default for MultiNet since about V3.5
I believe).
The reason for the slowness, and the reason disabling Path MTU Discovery
speeds the transfer is that your DEC NI3 bridge isn't returning the packets
marked "Don't Fragment" with ICMP "Fragmentation needed but DF set", as
required for Path MTU Discovery. Contact DEC and see if there is updated
firmware for the bridge that conforms to RFC1191.
-Dan Wing
dw...@tgv.com