[Lustre-discuss] adding 2nd network to lustre

[Chris Hunter]

We currently use lustre over an infiniband network. We would like to
allow ethernet clients (ie. no infiniband) to use the same lustre FS.

The MDT and OSTs were all created using only the MGS infiniband
interface (ie. mkfs.lustre --mgsnode=10.2.1.30@ib0).

We have modified the OSTs and MDS to route over ib & tcp networks via
lustre lnet module flags:
>options lnet networks=o2ib0(ib0),tcp(eth0)

For the ethernet clients, we specify only the tcp network for the lnet
module:
>options lnet network=tcp(eth0)

For the infiniband clients, we use only the ib network in lnet module:
>options lnet networks=o2ib(ib0)

From an infiniband client, we mount lustre using the MDS infiniband IP
address. eg)
mount -t lustre 10.2.1.30@ib0:/lustre0 /lustre0

On an ethernet client, mounting lustre using the MDS tcp interface
fails. eg)
mount -t lustre 10.1.1.30@tcp0:/lustre0 /lustre0

From the syslog messages, it appears the MGS tells the ethernet client
to use the ib0 interface (see attached syslog messages) to find the MDT.

So I am missing some configuration for the tcp network. My best guess we
need to use the "tunefs.lustre" command to add the mgs tcp NID to the
MDT. However this command won't execute while lustre is mounted.

Any suggestions how to add the tcp network to our existing lustre ib
network ?

Thanks,
--Chris

[Andreas Dilger]

On Jan 07, 2009 18:16 -0500, Chris Hunter wrote:
> We currently use lustre over an infiniband network. We would like to
> allow ethernet clients (ie. no infiniband) to use the same lustre FS.
>
> The MDT and OSTs were all created using only the MGS infiniband
> interface (ie. mkfs.lustre --mgsnode=10.2.1.30@ib0).
>

> So I am missing some configuration for the tcp network. My best guess we
> need to use the "tunefs.lustre" command to add the mgs tcp NID to the
> MDT. However this command won't execute while lustre is mounted.
>
> Any suggestions how to add the tcp network to our existing lustre ib
> network ?

I believe this is documented in the Lustre manual at http://manual.lustre.org/

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

_______________________________________________
Lustre-discuss mailing list
Lustre-...@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Chris Hunter]

Andreas Dilger wrote:
> On Jan 07, 2009 18:16 -0500, Chris Hunter wrote:
>> We currently use lustre over an infiniband network. We would like to
>> allow ethernet clients (ie. no infiniband) to use the same lustre FS.
>>
>> The MDT and OSTs were all created using only the MGS infiniband
>> interface (ie. mkfs.lustre --mgsnode=10.2.1.30@ib0).
>>
>> So I am missing some configuration for the tcp network. My best guess we
>> need to use the "tunefs.lustre" command to add the mgs tcp NID to the
>> MDT. However this command won't execute while lustre is mounted.
>>
>> Any suggestions how to add the tcp network to our existing lustre ib
>> network ?
>
> I believe this is documented in the Lustre manual at http://manual.lustre.org/

Section 7 of the operations manual does discuss multihomed lustre
servers. The example given is destructive -- lustre needs to be
reformatted (mkfs.lustre) for multihoming. Can we setup multihoming with
our re-formating our lustre FS ?

Thanks,
--Chris

[Andreas Dilger]

On Jan 09, 2009 18:31 -0500, Chris Hunter wrote:
> Andreas Dilger wrote:
>> On Jan 07, 2009 18:16 -0500, Chris Hunter wrote:
>>> We currently use lustre over an infiniband network. We would like to
>>> allow ethernet clients (ie. no infiniband) to use the same lustre FS.
>>>
>>> The MDT and OSTs were all created using only the MGS infiniband
>>> interface (ie. mkfs.lustre --mgsnode=10.2.1.30@ib0).
>>>
>>> So I am missing some configuration for the tcp network. My best guess
>>> we need to use the "tunefs.lustre" command to add the mgs tcp NID to
>>> the MDT. However this command won't execute while lustre is mounted.
>>>
>>> Any suggestions how to add the tcp network to our existing lustre ib
>>> network ?
>>
>> I believe this is documented in the Lustre manual at http://manual.lustre.org/
>
> Section 7 of the operations manual does discuss multihomed lustre
> servers. The example given is destructive -- lustre needs to be
> reformatted (mkfs.lustre) for multihoming. Can we setup multihoming with
> our re-formating our lustre FS ?

Yes, definitely. It should only require a --write-conf... I think this
has been answered previously on this list, but if not Isaac should get
it into the manual.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

_______________________________________________

[Chris Hunter]

Is there a way to backup the configuration information on the target
disk before changing the MGS NID (ie. adding second multi-homed network) ?

These config logs are created when the filesystem is created (ie.
mkfs.lustre) and not regenerated each time lustre is mounted ?

To add the second network, we plan to execute the command:
tune2fs.lustre --verbose --erase-param
--mgsnode=10.2.1.30@o2ib,10.1.1.30@tcp --writeconf /dev/sdb

PS It would be helpful to add this example to the manual, assuming it
works and is non-destructive. The current example for multi-homed
networks (mkfs.lustre) is destructive.

Thank-you in advance,

Chris Hunter