[Lustre-discuss] improving wire speeds over ethernet

[konrad.g...@ffs.pl]

Hi,

I've been using couple gigabit NIC's (realtek 8169) with Lustre,
was wondering if you can improve (probably can) the wire speeds by using
jumbo frames ?
Do i need to change anything else than mtu of the interface for this to work
?
Also the driver supports frames only up to around 7K

TiA
Konrad Gutkowski

_______________________________________________
Lustre-discuss mailing list
Lustre-...@clusterfs.com
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss

[Andreas Dilger]

On Feb 22, 2007 21:16 +0100, konrad.g...@ffs.pl wrote:
> I've been using couple gigabit NIC's (realtek 8169) with Lustre,
> was wondering if you can improve (probably can) the wire speeds by using
> jumbo frames ?

Yes, this reduces the overhead of the TCP packets noticably.

> Do i need to change anything else than mtu of the interface for this to

> work? Also the driver supports frames only up to around 7K

You quite often need to restart the interface in order to change the MTU.
Having an MTU slightly under 8kB means that a send or reply can fit into
a 2-page allocation after the TCP/IP/eth headers are added (assuming there
is no bug in the code as was the case with e1000 that always allocated
16kB). I don't know the exact MTU size though.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

[belshaw, john]

Konrad,

We have been using Jumbo frames with MTU=9000 for quite a while now so I
can say that Lustre is very efficient at using the network even with
MTU=1500, so you do not gain much in raw speed. You do gain by using
less cpu on the
OSS and clients which may help generally.

You should change all hosts at the same time and, we have found up till
now that Lustre appears to check the MTU size only at startup time so
you have to do a complete restart on all hosts.

John Belshaw
CGGVeritas

-----Original Message-----
From: lustre-disc...@clusterfs.com
[mailto:lustre-disc...@clusterfs.com] On Behalf Of
konrad.g...@ffs.pl
Sent: 22 February 2007 20:16
To: lustre-...@clusterfs.com
Subject: [Lustre-discuss] improving wire speeds over ethernet

Hi,

I've been using couple gigabit NIC's (realtek 8169) with Lustre,
was wondering if you can improve (probably can) the wire speeds by using

jumbo frames ?

Do i need to change anything else than mtu of the interface for this to
work
?
Also the driver supports frames only up to around 7K

TiA
Konrad Gutkowski