TCP Segment of a Reassembled PDU
Will
PDU" in a sniffer trace of its conversations with other machines on the same
ethernet segment. Some brief research suggested that there may be a
problem with the TCP MTU size on either end.
As a test, on one of the Windows 2000 workstations that has performance
problems with this server, I hardcoded through the registry the DWORD MTU in
the adapter TCP/IP Properties, and then set this to decimal 1400 (default is
1500 for ethernet). The workstation has a copper GigE adapter. The
server has an optical GigE adapter.
We rebooted the Windows 2000 workstation and instantly performance was great
with the server. Opening a message on that server took 4.5 seconds before
the MTU change now took 1/2 second after the MTU change. So clearly MTU
setting of 1400 did something good here. However:
1) The sniffer trace still shows the reassembled PDU messages.
2) The workstation immediately had severe problems using the file server and
was not able to use the Internet at all.
I backed out the MTU change and all resumed to its "normal" state.
Are there other settings that must be changed when you lower the MTU setting
on Windows 2000? What about Windows XP/Windows 2003?
What are some possible causes of these symptoms. Ultimately I would like
to know what changes would:
1) Make conversations with the server fast.
2) Remove fragmentation suggested by the reassembled PDU sniffer messages.
3) Allow normal communication through routers and onto the Internet.
--
Will
Andy Furniss
> Are there other settings that must be changed when you lower the MTU setting
> on Windows 2000? What about Windows XP/Windows 2003?
Don't know - but poeple often use a little free util called drtcp.exe to
save regediting.
One thing to check is whether the card/drivers are doing tcp
segmentation offload - maybe there is some messup to do with that.
Andy.
Barry Margolin
"Will" <weste...@noemail.nospam> wrote:
> One of my servers gives the continuous message "TCP segment of a reassembled
> PDU" in a sniffer trace of its conversations with other machines on the same
> ethernet segment. Some brief research suggested that there may be a
> problem with the TCP MTU size on either end.
Are you talking about Wireshark (formerly Ethereal)? That message
doesn't indicate a problem, and it's not talking about IP
fragmentation/reassembly.
What Wireshark does is automatically combine all the TCP segments of an
application-layer message into a single row in the display. For
instance, it will show an HTTP download as a single "Reassembled PDU".
All the segments that contribute to it are marked with the above
message, to indicate that they're duplicated in the merged row.
--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
Will
news:barmar-534748....@comcast.dca.giganews.com...
> In article <aPOdnQ0UwoCCjOzY...@giganews.com>,
> "Will" <weste...@noemail.nospam> wrote:
> > One of my servers gives the continuous message "TCP segment of a
reassembled
> > PDU" in a sniffer trace of its conversations with other machines on the
same
> > ethernet segment. Some brief research suggested that there may be a
> > problem with the TCP MTU size on either end.
>
> Are you talking about Wireshark (formerly Ethereal)? That message
> doesn't indicate a problem, and it's not talking about IP
> fragmentation/reassembly.
>
> What Wireshark does is automatically combine all the TCP segments of an
> application-layer message into a single row in the display. For
> instance, it will show an HTTP download as a single "Reassembled PDU".
> All the segments that contribute to it are marked with the above
> message, to indicate that they're duplicated in the merged row.
Well, that certainly explains the message. It doesn't however explain why
the workstation with bad performance had a query to one server go from 4.5
seconds to 0.5 seconds when the MTU was lowered to 1400.
The problem is what other settings need to be changed because now the
workstation doesn't work to servers that require it to send packets through
routers.
--
Will
Will
news:4571c56f.0@entanet...
I've seen that utility, but it sure looks like it was written for older
versions of Windows, and I guess I'm not sure if I should trust it for
Windows 2003.
I would also prefer to have the knowledge of what it is doing.
--
Will
Martijn Lievaart
> Well, that certainly explains the message. It doesn't however explain why
> the workstation with bad performance had a query to one server go from 4.5
> seconds to 0.5 seconds when the MTU was lowered to 1400.
You have to look further on the wire. Your problem sounds pretty weird.
This performance gain does not correspond to "normal" MTU problems. Either
a transmission succeeds or it doesn't at all. "Normal" MTU problems occur
when a datagram is send with a size larger than the path MTU (PMTU) and
the datagram has the Don't Fragment bit set (PMTUD discovery is on, this
is the default for most modern systems) and something blocks the ICMP
"Fragmentation needed but Don't Fragment bit set" error. The datagram is
discarded, resend, discarded, repeat until time out. This is called a PMTU
blackhole.
But I seem to recall that Windows sometimes lowers the MTU on retries to
get around such blackholes, maybe this explains the behaviour you are
seeing.
Yes, the more I think about it, that would explain your problem. That
assumes a MTU problem somewhere which would also explain your other
problems (not being able to access the Internet and fileservers after
lowering the MTU). Something is blocking the ICMP "Fragmentation needed
but Don't Fragment bit set" error. And although I advice to check the
firewall settings on both client and server, this is probably happening
somewhere along the path.
Luckily for you, this is often easy to diagnose, even if it is somewhere
on the path. First check the firewalls on the client and server. If that
doesn't solve it, use tracert.exe with the appropriate options. Start
with a large MTU, and lower it until things start to work (the trace gets
through). The hop that lies one behind the one that is not responding with
the large MTU is the one that blocks the error (in some cases it may also
be the last one that DID respond).
If you cannot trace because something allows only certain traffic through,
that would be the first point to investigate. That point probably blocks
the ICMP error as well. You can also try ping with different MTUs and the
"record route" option. This assumes ping is allowed when traceroute isn't
and doesn't always work as record route is often seen as a security risk
and filtered out.
A whole other aproach would be to look at the network path the packets
take and see where the MTU is lower and where ICMP "Fragmentation needed
but Don't Fragment bit set" errors are blocked. Or get in someone who
really knows networking and have that person have a look at the network
(there more often than not are other problems when you have PMTU
blackholes).
Or start sniffing on various parts of the path. Find out that way where
the packets are dropped. Works best with two persons and two (time
synchronized!) sniffers and do a kind of binary search along the path.
Last resort would be to boot the 2000 workstation of a Linux live-CD that
has hping2 (Auditor comes to mind, but beware that you need to enable
networking manually after booting, that trips up everyone starting out
with Auditor). Hping2 can do a traceroute using tcp packets on the same
port you use for your client server communication. That is virtually
guarenteed to work (firewalls might drop packets with a to low TTL, but I
haven't seen it in practice) as whatever blocks those packets is already
allowing that client-server protocol. (You could also install a free
version of VMWare and install Linux in that).
I sincerely hope you don't work at one of those "all icmp is evil" shops.
Many icmp packets pose a real security risk, while adding virtually no
real value. Those can be blocked. Some can be blocked with certain side
effects. But "Fragmentation needed but Don't Fragment bit set" errors
should NEVER, NEVER, NEVER, EVER be blocked. It lands you in this kind of
trouble. I know, I once worked at such a shop. Even after one and a half
years of continious troubles they would not change that policy. The truth
is, anyone who advocates blocking of "Fragmentation needed but Don't
Fragment bit set" errors does not know enough about TCP/IP to be allowed
to make such decissions. It's simply wrong. Search Google for "PMTU
blackholes" if you need additional ammunition to convince people they
should not do this.
> The problem is what other settings need to be changed because now the
> workstation doesn't work to servers that require it to send packets
> through routers.
Besides the local firewalls, you are probably not going to solve this with
local settings. A workaround, if Windows allows this which I think it
does, is to set an explicit route with an explicit MTU on the
workstation for that particular server. But I advice against it, the
problems will come back to bite you again and again and again.
But this whole "solution" hinges on the assumption that Windows does retry
with a lower MTU. I'm actually pretty sure about that and the more I think
about it, the more I am convinced I'm right. Still it may be something
completely different. Time to get out that sniffer again and validate or
disprove my theory. Let us know how it went.
HTH,
M4
--
Redundancy is a great way to introduce more single points of failure.
Martijn Lievaart
> Luckily for you, this is often easy to diagnose, even if it is somewhere
> on the path. First check the firewalls on the client and server. If that
> doesn't solve it, use tracert.exe with the appropriate options. Start
> with a large MTU, and lower it until things start to work (the trace gets
> through). The hop that lies one behind the one that is not responding with
> the large MTU is the one that blocks the error (in some cases it may also
> be the last one that DID respond).
Correction, that last is completely bull. The MTU problem is
immediately after the last hop that responded. The blocking of the ICMP
error occurs somehere between (inclusive) the client and that hop.
Also be sure to turn on the Don't Fragment bit on any trace you do,
otherwise you won't see the problem.
A last quick fix, turn of PMTUD at the client completely. Fixes all your
problems, but is not the greatest of solutions. However, IF this solves
all your problems you are sure it is a PMTU blackhole problem.
bri...@encompasserve.org
> On Sat, 02 Dec 2006 17:20:06 -0800, Will wrote:
>
>> Well, that certainly explains the message. It doesn't however explain why
>> the workstation with bad performance had a query to one server go from 4.5
>> seconds to 0.5 seconds when the MTU was lowered to 1400.
>
> You have to look further on the wire. Your problem sounds pretty weird.
> This performance gain does not correspond to "normal" MTU problems. Either
> a transmission succeeds or it doesn't at all. "Normal" MTU problems occur
> when a datagram is send with a size larger than the path MTU (PMTU) and
> the datagram has the Don't Fragment bit set (PMTUD discovery is on, this
> is the default for most modern systems) and something blocks the ICMP
> "Fragmentation needed but Don't Fragment bit set" error. The datagram is
> discarded, resend, discarded, repeat until time out. This is called a PMTU
> blackhole.
>
> But I seem to recall that Windows sometimes lowers the MTU on retries to
> get around such blackholes, maybe this explains the behaviour you are
> seeing.
Yes. "PMTU black hole router detection"
e.g. http://support.microsoft.com/kb/925280
That article describes the response as reducing the MTU to 536 and
clearing the DF bit. [It is plausible that the MTU is a typo
for MSS or that 536 is a typo for 576 -- 536 seems like a strange
number]
Packet capture should provide conclusive evidence whether this is
taking place.