[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp
Mike Hanby
I have a working Lustre file system set up using Infiniband:
1 x MDS/MGS server
2 x OSS/OST servers, in active active failover
25 x client nodes
All of these systems use Infiniband with Lustre.
Now, I have 60 older compute nodes that I'd like to add to the system. These only have Gigabit Ethernet.
I've added the tcp network to lnet (see steps below), but when I attempt to mount my luster filesystem on the tcp client it fails, and the error in /var/log/messages seems to indicate that it's trying to use o2ib, and not tcp:
# mount -t lustre 172.20.20.30@tcp:/lustre /lustre
mount.lustre: mount 172.20.20.30@tcp:/lustre at /lustre failed: No such file or directory
Is the MGS specification correct?
Is the filesystem name correct?
If upgrading, is the copied client log valid? (see upgrade docs)
kernel: LustreError: 2860:0:(events.c:460:ptlrpc_uuid_to_peer()) No NID found for 172.20.21.30@o2ib
kernel: LustreError: 2860:0:(client.c:69:ptlrpc_uuid_to_connection()) cannot find peer 172.20.21.30@o2ib!
kernel: LustreError: 2860:0:(ldlm_lib.c:329:client_obd_setup()) can't add initial connection
kernel: LustreError: 2860:0:(obd_config.c:370:class_setup()) setup lustre-MDT0000-mdc-ffff81007eb82400 failed (-2)
kernel: LustreError: 2860:0:(obd_config.c:1197:class_config_llog_handler()) Err -2 on cfg command:
kernel: LustreError: 15c-8: MGC172.20.20.30@tcp: The configuration from log 'lustre-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Here's my modprobe.conf entry for the OSS/MDS servers (is order important here or tcp and o2ib?)
options lnet networks="tcp0(eth0),o2ib(ib0)"
options ko2iblnd concurrent_sends=7
options ptlrpc at_max=600
options ost oss_num_threads=512
modprobe.conf file for the IB clients
options lnet networks="o2ib(ib0)"
options ko2iblnd concurrent_sends=7
options ptlrpc at_max=600
options ost oss_num_threads=512
And modprobe.conf for the TCP clients
options lnet networks="tcp0(eth0)"
options ptlrpc at_max=600
options ost oss_num_threads=512
The 'lctl list_nids' command prints the expected results on the servers and clients, listing the networks provided in the modprobe.conf file
I added the failover and mgsnode settings to each lun (6 luns) using the following:
tunefs.lustre --failnode=172.20.20.31@tcp --failnode=172.20.20.32 \
--mgsnode=172.20.20.30@tcp /dev/mpath/lun1
With the final parameters being:
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: failover.node=172.20.21.31@o2ib failover.node=172.20.21.32@o2ib mgsnode=172.20.21.30@o2ib failover.node=172.20.20.31@tcp failover.node=172.20.20.32@tcp mgsnode=172.20.20.30@tcp
The /etc/fstab on the TCP clients has this entry:
172.20.20.30@tcp0:/lustre /lustre lustre _netdev 0 0
I've rebooted all of the servers after making all of the changes and still I can't mount from the TCP clients, but can from the IB clients.
Any suggestions?
=================================
Mike Hanby
mha...@uab.edu
Information Systems Specialist II
IT HPCS / Research Computing
_______________________________________________
Lustre-discuss mailing list
Lustre-...@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
David Dillow
> I added the failover and mgsnode settings to each lun (6 luns) using the following:
> tunefs.lustre --failnode=172.20.20.31@tcp --failnode=172.20.20.32 \
> --mgsnode=172.20.20.30@tcp /dev/mpath/lun1
Did use --writeconf on the servers? You need to do so with LNET up with
the appropriate nids, so that clients (and the MDS) can find the servers
on both networks.
I think this is covered in the manual, so you should check there as
well.
--
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office
Mike Hanby
I reran the tunefs.lustre on each of the luns (after stopping the heartbeat service on each of the OSS nodes) using --writeconf:
tunefs.lustre --writeconf --erase-params \
--param="failover.node=172.20.21.31@o2ib" \
--param="failover.node=172.20.21.32@o2ib" \
--param="mgsnode=172.20.21.30@o2ib" \
--param="failover.node=172.20.20.31@tcp" \
--param="failover.node=172.20.20.32@tcp" \
--param="mgsnode=172.20.20.30@tcp" /dev/mpath/lun1
checking for existing Lustre data: found CONFIGS/mountdata
Reading CONFIGS/mountdata
Read previous values:
Target: lustre-OST0000
Index: 0
Lustre FS: lustre
Mount type: ldiskfs
Flags: 0x2
(OST )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: failover.node=172.20.21.31@o2ib failover.node=172.20.21.32@o2ib mgsnode=172.20.21.30@o2ib
Permanent disk data:
Target: lustre-OST0000
Index: 0
Lustre FS: lustre
Mount type: ldiskfs
Flags: 0x542
(OST update writeconf )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: failover.node=172.20.21.31@o2ib failover.node=172.20.21.32@o2ib mgsnode=172.20.21.30@o2ib failover.node=172.20.20.31@tcp failover.node=172.20.20.32@tcp mgsnode=172.20.20.30@tcp
Writing CONFIGS/mountdata
Following fresh boots of the OSSes and MDS and the tcp client, I'm still getting the same error:
kernel: LustreError: 2036:0:(events.c:460:ptlrpc_uuid_to_peer()) No NID found for 172.20.21.30@o2ib
kernel: LustreError: 2036:0:(client.c:69:ptlrpc_uuid_to_connection()) cannot find peer 172.20.21.30@o2ib!
kernel: LustreError: 2036:0:(ldlm_lib.c:329:client_obd_setup()) can't add initial connection
kernel: LustreError: 2036:0:(obd_config.c:370:class_setup()) setup lustre-MDT0000-mdc-ffff81003f7c3400 failed (-2)
kernel: LustreError: 2036:0:(obd_config.c:1197:class_config_llog_handler()) Err -2 on cfg command:
kernel: LustreError: 15c-8: MGC172.20.20.30@tcp: The configuration from log 'lustre-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
kernel: LustreError: 1959:0:(llite_lib.c:1171:ll_fill_super()) Unable to process log: -2
kernel: LustreError: 1959:0:(obd_config.c:441:class_cleanup()) Device 2 not setup
kernel: LustreError: 1959:0:(ldlm_request.c:1030:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
kernel: LustreError: 1959:0:(ldlm_request.c:1533:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
kernel: LustreError: 1959:0:(obd_mount.c:1997:lustre_fill_super()) Unable to mount (-2)
Do I need to run tunefs.lustre on the MGS node as well?
Thanks, Mike
Andreas Dilger
> kernel: LustreError: 15c-8: MGC172.20.20.30@tcp: The configuration
> from log 'lustre-client' failed (-2). This may be the result of
> communication errors between this node and the MGS, a bad
> configuration, or other errors. See the syslog for more information.
>
Yes, since you specified multiple NIDs for the MGS, the MGS itself
needs to
know to accept connections on that interface.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Mike Hanby
tunefs.lustre --dryrun --writeconf --erase-params \
--param="failover.node=172.20.21.30@o2ib" \
--param="failover.node=172.20.20.30@tcp" \
--param="mdt.group_upcall=NONE" /dev/loop0
Reading CONFIGS/mountdata
Read previous values:
Target: lustre-MDT0000
Index: 0
Lustre FS: lustre
Mount type: ldiskfs
Flags: 0x405
(MDT MGS )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: failover.node=172.20.21.30@o2ib mdt.group_upcall=NONE
Permanent disk data:
Target: lustre-MDT0000
Index: 0
Lustre FS: lustre
Mount type: ldiskfs
Flags: 0x545
(MDT MGS update writeconf )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: failover.node=172.20.21.30@o2ib failover.node=172.20.20.30@tcp mdt.group_upcall=NONE
exiting before disk write.
Mike Hanby
Anyhow, both networks are now working. Thanks to both for the clues, writeconf and adding to the mgs.
Mike