[Lustre-discuss] Dual NICs issue -- How to enforce Lustre to use the second NIC

[Daneil Goodman]

Hello list,

By searching the archive, I found a similar message dated back in January 2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks like there is no final solution and I am facing the similar situation and need your help.

I am running centos 5 on both server (MGS, MDS and OSS are on same node) and clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To simplify the issue, suppose the network is consist of one lustre server node and two lustre client nodes. The server node has two NICs: eth0(100Mb) and eth1(1Gb), each client node only has one NIC:eth0. The network layout is as below.

Server node eth0: 72.203.10.1 (Public network)    <==> Switch1 <==> Public node eth0:  72.203.10.2 (Public network)
Server node eth1: 192.168.10.1 (Internal network) <==> Switch2 <==> Private node eth0: 192.168.10.2 (Internal network)

Both SELinux and Fireware are turned off. Public node does not know Private node, but Private node do knows Public node.

The modprobe.conf likes the following:

On server: options lnet networks="tcp0(eth0),tcp1(eth1)"
On clients: options lnet networks=tcp  <--- since there is only one NIC, I did not specify it as tcp(eth0).

The procedure to build MGS, MDT and OST on server node as bellow:

[root@server ~]# mkfs.lustre --mgs /dev/sdb1
[root@server ~]# mkdir /mgs
[root@server ~]# mount -t lustre /dev/sdb1 /mgs
[root@server ~]# mkfs.lustre --mdt --fsname=data --mgsnode="72.203.10.1@tcp0,192.168.10.1@tcp1" /dev/sdb2
[root@server ~]# mkdir /data
[root@server ~]# mount -t lustre /dev/sdb2 /data
[root@server ~]# mkfs.lustre --ost --fsname=data --mgsnode="72.203.10.1@tcp0,192.168.10.1@tcp1" /dev/sdc1
[root@server ~]# mkdir /mnt/data
[root@server ~]# mount -t lustre /dev/sdc1 /mnt/data

On server node:
[root@server ~]# lctl list_nids
72.203.10.1@tcp
192.168.10.1@tcp1

[root@server ~]# lctl ping 72.203.10.1@tcp
12345-0@lo
12345-72.203.10.1@tcp
12345-192.168.10.1@tcp1

[root@server ~]# lctl ping 192.168.10.1@tcp1
12345-0@lo
12345-72.203.10.1@tcp
12345-192.168.10.1@tcp1

Therefore, every thing on server node looks good, and I can mount /data on public node:
[root@public ~]# mount -t lustre 72.203.10.1@tcp:/data /data

Also I can mount /data on private node using server's eth0 (public NIC)
[root@private ~]# mount -t lustre 72.203.10.1@tcp:/data /data

But I can not mount /data on private node using server's eth1 (private NIC).
[root@private ~]# mount -t lustre 192.168.10.1@tcp1:/data /data
mount.lustre: /data inaccessible: No such file or directory

On private node:
[root@private ~]# lctl which_nid 192.168.10.1@tcp1
No reachable NID

[root@private ~]# lctl which_nid 192.168.10.1@tcp
192.168.10.1@tcp

[root@private ~]# lctl which_nid 72.203.10.1@tcp
72.203.10.1@tcp

[root@private ~]# lctl ping 72.203.10.1@tcp
12345-0@lo
12345-72.203.10.1@tcp
12345-192.168.10.1@tcp1

We see the NID of 192.168.10.1@tcp from the private node, but can not ping it:
[root@private ~]# lctl ping 192.168.10.1@tcp1
failed to ping 192.168.10.1@tcp1: Input/output error

[root@private ~]# lctl ping 192.168.10.1@tcp
failed to ping 192.168.10.1@tcp: Input/output error

The error message in /var/log/messages are
LustreError: 27435:0:(lib-move.c:1265:lnet_send()) No route to 12345-192.168.10.1@tcp1
LustreError: 27435:0:(lib-move.c:2450:LNetGet()) error sending GET to 12345-192.168.10.1@tcp1: -113

But I can ping 192.168.10.1 using regular ping.

According to the response of Marc:
"As far as I know, LNET will use the shortest path on the network, so if you have two equivalent tcp networks, tcp0 and tcp1, LNET will just  use the first one.  If it fails, it should use the second one. If both NICs are in the same tcp network, LNET should use both.....".

Therefore, I can understand why I can mount /data on private node using server's eth0. But looks like LNET is not smart enough to pick up the "closer" NIC to mount the /data.

My questions are:
1. Is something wrong of the eth1configuration?
2. How to enforce LNET to use the second NIC(eth1) to mount /data on private node? I prefer to use eth1 not eth0 because eth1 connects to switch2 directly and fast (eth1-1Gb) than eth0 (100Mb).

Thanks for reading the detailed procedure and appreciate for your help in advance,

Goodman

[Isaac Huang]

On Wed, Nov 11, 2009 at 04:07:39PM -0600, Daneil Goodman wrote:
> Hello list,
> By searching the archive, I found a similar message dated back in
> January 2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks like
> there is no final solution and I am facing the similar situation and
> need your help.
> I am running centos 5 on both server (MGS, MDS and OSS are on same
> node) and clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To simplify
> the issue, suppose the network is consist of one lustre server node and
> two lustre client nodes. The server node has two NICs: eth0(100Mb) and
> eth1(1Gb), each client node only has one NIC:eth0. The network layout
> is as below.
> Server node eth0: 72.203.10.1 (Public network) <==> Switch1 <==>
> Public node eth0: 72.203.10.2 (Public network)
> Server node eth1: 192.168.10.1 (Internal network) <==> Switch2 <==>
> Private node eth0: 192.168.10.2 (Internal network)
> Both SELinux and Fireware are turned off. Public node does not know
> Private node, but Private node do knows Public node.
> The modprobe.conf likes the following:
> On server: options lnet networks="tcp0(eth0),tcp1(eth1)"
> On clients: options lnet networks=tcp <--- since there is only one

I think you'd need to make clients in the 72.203.10.* network use tcp0
and clients in the 192.168.10.* tcp1. To create a uniform module
option that works across the whole cluster, 'ip2nets' is your friend:

options lnet 'ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1) 192.168.10.[1-10]; tcp1(eth0) 192.168.10.[100-200]"

(assuming that servers are 192.168.10.[1-10] and clients are
192.168.10.[100-200].)

Isaac
_______________________________________________
Lustre-discuss mailing list
Lustre-...@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Daneil Goodman]

On Wed, Nov 11, 2009 at 9:20 PM, Isaac Huang <He.H...@sun.com> wrote:

On Wed, Nov 11, 2009 at 04:07:39PM -0600, Daneil Goodman wrote:
>    Hello list,
>    By searching the archive, I found a similar message dated back in
>    January 2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks like
>    there is no final solution and I am facing the similar situation and
>    need your help.
>    I am running centos 5 on both server (MGS, MDS and OSS are on same
>    node) and clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To simplify
>    the issue, suppose the network is consist of one lustre server node and
>    two lustre client nodes. The server node has two NICs: eth0(100Mb) and
>    eth1(1Gb), each client node only has one NIC:eth0. The network layout
>    is as below.
>    Server node eth0: 72.203.10.1 (Public network)    <==> Switch1 <==>
>    Public node eth0:  72.203.10.2 (Public network)
>    Server node eth1: 192.168.10.1 (Internal network) <==> Switch2 <==>
>    Private node eth0: 192.168.10.2 (Internal network)
>    Both SELinux and Fireware are turned off. Public node does not know
>    Private node, but Private node do knows Public node.
>    The modprobe.conf likes the following:
>    On server: options lnet networks="tcp0(eth0),tcp1(eth1)"
>    On clients: options lnet networks=tcp  <--- since there is only one

I think you'd need to make clients in the 72.203.10.* network use tcp0
and clients in the 192.168.10.* tcp1. To create a uniform module

It did the trick!  Thanks!

option that works across the whole cluster, 'ip2nets' is your friend:

options lnet 'ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1) 192.168.10.[1-10]; tcp1(eth0) 192.168.10.[100-200]"

(assuming that servers are 192.168.10.[1-10] and clients are
192.168.10.[100-200].)

There are three small issues about ip2nets:

1. Looks like LNET does not like the single quotes 'ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1) 192.168.10.*"'. It says

lnet: Unknown parameter `'ip2nets'

After removed single quotes, I can load lnet module.

2. According to my observation, on public network, to mount /data using above ip2nets option is slower than networks option.

3. On private network node, I cannot start LNET using ip2nets option
[root@private ~]# lsmod |grep lnet
lnet                  273084  1 ksocklnd
libcfs                136180  2 ksocklnd,lnet
[root@private ~]# lctl network configure
LNET configure error 100: Network is down

/var/log/messages shows:
LustreError: 31927:0:(socklnd.c:2545:ksocknal_startup()) Interface eth1 is down
LustreError: 105-4: Error -100 starting up LNI tcp

But if you use networks option (options lnet networks=tcp1), it works well. Do you think what is the problem?

Thanks,
Goodman

Isaacc

[Daneil Goodman]

3. On private network node, I cannot start LNET using ip2nets option
[root@private ~]# lsmod |grep lnet
lnet                  273084  1 ksocklnd
libcfs                136180  2 ksocklnd,lnet
[root@private ~]# lctl network configure
LNET configure error 100: Network is down

/var/log/messages shows:
LustreError: 31927:0:(socklnd.c:2545:ksocknal_startup()) Interface eth1 is down
LustreError: 105-4: Error -100 starting up LNI tcp

But if you use networks option (options lnet networks=tcp1), it works well. Do you think what is the problem?

I fixed this problem by changing the order of the interfaces in ip2nets line.
 

Thanks,
Goodman

Isaacc