[lwip-users] Jumbo frame questions

[Bill Auerbach]

Hello,

 

I made changes to support Jumbo Frames (9k).  Besides the driver changes, I used TCP_MSS of 9000.  All is good with about 100Mb/S speed increase.

 

I dropped back to a connection that doesn’t support jumbo frames and I have connection problems because of TCP_MSS not being e.g. 1512.

 

I cannot use something like: #define TCP_MSS (UseJumbo ? 9000 : 1512).

 

How can I adjust TCP_MSS when I’m on a “normal” network?  Or… what else do I need to look at for supporting Jumbo frames transparently?  Shouldn’t my lwIP side see the other end’s MSS on connection and downsize accordingly?

 

If this is a basic networking or TCP question and off topic, please let me know where to find what I need.

 

Thank you,

 

Bill Auerbach

 

 

[Goldschmidt Simon]

Bill

 

Which version of lwIP are you using? I think there has been a change in CVS recently that fits your problem (remote mss not used for sending). Could you try to use the v1.3.0 release candidate?

 

Simon

 

---

Von: lwip-users-bounces+sgoldschmidt=de.pepperl...@nongnu.org [mailto:lwip-users-bounces+sgoldschmidt=de.pepperl...@nongnu.org] Im Auftrag von Bill Auerbach
Gesendet: Donnerstag, 31. Januar 2008 16:54
An: 'Mailing list for lwIP users'
Betreff: [lwip-users] Jumbo frame questions

[Bill Auerbach]

Hi Simon,

 

I updated and retested – so it’s the latest as of yesterday.  I can’t say if it’s an lwIP issue or not.  The problem exists only when lwIP is configured for jumbo frames, the connection is to the PC configured for large frames, but the network cannot support large frames.  WinXP works in this configuration but the lwIP device connected does not.  I can change the Frame Size to 1512 and both the PC and lwIP device work.

 

Could it be that the netif MTU has to be adjusted – it’s set to the frame size – 28 (for headers).

 

Thanks,

 

Bill Auerbach

 

---

[Kieran Mansley]

On Fri, 2008-02-01 at 12:33 -0500, Bill Auerbach wrote:
> Hi Simon,
>
>
>
> I updated and retested – so it’s the latest as of yesterday. I can’t
> say if it’s an lwIP issue or not. The problem exists only when lwIP
> is configured for jumbo frames, the connection is to the PC configured
> for large frames, but the network cannot support large frames. WinXP
> works in this configuration but the lwIP device connected does not. I
> can change the Frame Size to 1512 and both the PC and lwIP device
> work.
>
>
>
> Could it be that the netif MTU has to be adjusted – it’s set to the
> frame size – 28 (for headers).
>

There may be a bug here, but I think it's just you're hoping the lwIP
has a feature that it doesn't have: path MTU discovery. Normally if you
try and send a packet with a larger MTU than some piece of the network
can't handle it will either fragment it (if it's allowed) or drop it and
send an ICMP message back to the sender. I expect that lwIP does
nothing with this ICMP message. This could be construed a bug
(especially if we also set the DF - Don't Fragment - bit in the header)
but could also be seen as a property of lwIP being "lightweight".

For the avoidance of confusion:
MSS = largest TCP payload
MTU = largest IP packet (i.e. MSS + IP and TCP headers).
Frame size = largest ethernet frame (i.e. MTU + ethernet headers etc)
i.e. You may need to adjust your MSS settings as it looks like you're
giving it the values you want for the MTU or frame size.

Kieran

_______________________________________________
lwip-users mailing list
lwip-...@nongnu.org
http://lists.nongnu.org/mailman/listinfo/lwip-users

[Kieran Mansley]

On Tue, 2008-02-05 at 15:23 +0000, Kieran Mansley wrote:
> This could be construed a bug
> (especially if we also set the DF - Don't Fragment - bit in the header)
> but could also be seen as a property of lwIP being "lightweight".

I've just checked - we don't send the DF flag, and so the network should
fragment the packet if necessary. That it's not suggests that you might
have the two nodes on the same LAN. If that's correct, then you've got
a configuration error: all nodes on the same LAN are supposed to have
the same frame length and MTU, and it sounds like you've put the lwIP
node with an MTU of 9000 on a network with an MTU of 1500. That just
doesn't work I'm afraid.

[Kieran Mansley]

On Tue, 2008-02-05 at 15:47 +0000, Kieran Mansley wrote:
> If that's correct, then you've got
> a configuration error: all nodes on the same LAN are supposed to have
> the same frame length and MTU, and it sounds like you've put the lwIP
> node with an MTU of 9000 on a network with an MTU of 1500. That just
> doesn't work I'm afraid.

I've just had another look! If you use the LWIP_CALCULATE_EFF_SEND_MSS
option and then set the MTU appropriately in your netif (the example
ethernetif.c just always sets it to 1500 - I presume your changes to get
jumbo frames working did something more sophisticated) you should be
able to safely set TCP_MSS to 8960 (i.e. an MTU of 9000) and lwIP will
downsize the MSS appropriately.

The bit that's probably missing in your port is the setting of netif-
>mtu. This is normally configured using a tool like ifconfig on Linux,
but how you do this on your target will depend heavily on the OS (if
any) it's running.

Hope that helps,

[Bill Auerbach]

> Kieran Mansley
> Sent: Tuesday, February 05, 2008 10:24 AM

> To: Mailing list for lwIP users
> Subject: RE: [lwip-users] Jumbo frame questions
>

> There may be a bug here, but I think it's just you're hoping the lwIP
> has a feature that it doesn't have: path MTU discovery. Normally if you
> try and send a packet with a larger MTU than some piece of the network
> can't handle it will either fragment it (if it's allowed) or drop it and
> send an ICMP message back to the sender. I expect that lwIP does
> nothing with this ICMP message. This could be construed a bug
> (especially if we also set the DF - Don't Fragment - bit in the header)
> but could also be seen as a property of lwIP being "lightweight".
>
> For the avoidance of confusion:
> MSS = largest TCP payload
> MTU = largest IP packet (i.e. MSS + IP and TCP headers).
> Frame size = largest ethernet frame (i.e. MTU + ethernet headers etc)
> i.e. You may need to adjust your MSS settings as it looks like you're
> giving it the values you want for the MTU or frame size.

This may be a big part of my problem. I was trying to configure off of one
#define to build either Jumbo or normal lwIP.

If I have

#define PKTSIZE 1514

And

netif-mtu = PKTSIZE - 14;

And

TCP_MSS (PKTSIZE - 54)

This gets me an MTU of 1500 and MSS of 1460. This works much better now.

This works now with the PC's Ethernet controller at 1514 or 9000. I know
you say it's not supposed to, but I do see frames over 1514 in WireShark and
I can still access our LAN and WAN.

Thank you for setting me straight on these 3 values.

What are the reason(s) you wouldn't set MTU and MSS based on the packet size
as I have done?

Bill

[Bill Auerbach]

Thanks,

Bill Auerbach

> On Tue, 2008-02-05 at 15:23 +0000, Kieran Mansley wrote:
> > This could be construed a bug
> > (especially if we also set the DF - Don't Fragment - bit in the header)
> > but could also be seen as a property of lwIP being "lightweight".
>
> I've just checked - we don't send the DF flag, and so the network should
> fragment the packet if necessary. That it's not suggests that you might
> have the two nodes on the same LAN. If that's correct, then you've got
> a configuration error: all nodes on the same LAN are supposed to have
> the same frame length and MTU, and it sounds like you've put the lwIP
> node with an MTU of 9000 on a network with an MTU of 1500. That just
> doesn't work I'm afraid.

Could it be that my router sees 2 jumbo frame supporting Ethernet adapters
(PC and PPC) and is allowing jumbo frames, but with any other normal
adapter, fragmenting? I can't explain why my PC has jumbo frame support and
can access my LAN and WAN.

Bill

[Bill Auerbach]

Thanks,

Bill Auerbach

> -----Original Message-----
> From: lwip-users-bounces+bauerbach=arrayon...@nongnu.org
> [mailto:lwip-users-bounces+bauerbach=arrayon...@nongnu.org] On Behalf

> Of Kieran Mansley
> Sent: Tuesday, February 05, 2008 11:01 AM
> To: Mailing list for lwIP users
> Subject: RE: [lwip-users] Jumbo frame questions
>

> On Tue, 2008-02-05 at 15:47 +0000, Kieran Mansley wrote:
> > If that's correct, then you've got
> > a configuration error: all nodes on the same LAN are supposed to have
> > the same frame length and MTU, and it sounds like you've put the lwIP
> > node with an MTU of 9000 on a network with an MTU of 1500. That just
> > doesn't work I'm afraid.
>
> I've just had another look! If you use the LWIP_CALCULATE_EFF_SEND_MSS
> option and then set the MTU appropriately in your netif (the example
> ethernetif.c just always sets it to 1500 - I presume your changes to get
> jumbo frames working did something more sophisticated) you should be
> able to safely set TCP_MSS to 8960 (i.e. an MTU of 9000) and lwIP will
> downsize the MSS appropriately.

I do (or have left) this option enabled. My issue was setting MTU to the
packet size. I'm surprised I've gotten as far as I have. :)

> The bit that's probably missing in your port is the setting of netif-
> >mtu. This is normally configured using a tool like ifconfig on Linux,
> but how you do this on your target will depend heavily on the OS (if
> any) it's running.

I wasn't adjusting MTU (or MSS) for the headers. I am able to leave the
lwIP stack and it's controller set for Jumbo frames and it does communicate
on a network not supporting it. This was what I was after - and maybe it
works *only* with my router? lwIP was crashing this way because of the MTU.

Thank you for your help and replies. I didn't mean for you to expend so
much effort but it helped me. I can't explain the jumbo and non-jumbo mix
working.

Bill