Infiniband clients after reboot on ethernet

[Sternberger]

Hello!

I've just noticed that my clients are after reboot connected to the storage and meta nodes

via tcp. when i restart the client afterwards with the rc script they are on RDMA again.

I think the problem is described also here 

http://kb.brightcomputing.com/faq/pdf.php?cat=11&id=23&artlang=en

OS: SL6.3

FHGFS: 2011.04.r22-el6.x86_64

dmesg:

fhgfs_opentk: modprobe: FhGFS OpenToolkit loaded. (ibverbs enabled)

fhgfs: modprobe(5108): File system registered. Type: fhgfs. Version: 2011.04-r22

ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready

fhgfs: mount(5119): FhGFS mount ready.

Is there a better way (without custom initrd) to fix this issue?

best regards!

Sven

[Sven Breuner]

Hi Sven,

Sternberger wrote on 03/11/2013 10:56 AM:
> OS: SL6.3
> FHGFS: 2011.04.r22-el6.x86_64
>
> dmesg:
> fhgfs_opentk: modprobe: FhGFS OpenToolkit loaded. (ibverbs enabled)
> fhgfs: modprobe(5108): File system registered. Type: fhgfs. Version:
> 2011.04-r22
> ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
> fhgfs: mount(5119): FhGFS mount ready.
>
> Is there a better way (without custom initrd) to fix this issue?

indeed, from the dmesg output it seems like fhgfs is mounted before your
ib is ready.

You could either check which /etc/init.d/xy script starts your ib device
and make sure that this script is included as depedency in the "Should
start" line at the top of the /etc/init.d/fhgfs-client script. If it
isn't, you would probably need to "chkconfig --del" and then "chkconfig
--add" the fhgfs-client script to make the new depedency become effective.

Or you might want to take a look at /etc/dracut.conf for a more
convenient way to add the ib modules to your initrd (see "man dracut" or
"man dracut.conf").

Best regards,
Sven

[Michael Ruepp]

Hi,

I was running exactly into the same problem. After one Month testing without flaws, I reprovisioned the nodes (bright) and since then the fhgfs-clients connect via tcp over ethernet. RDMA is used only after restarting the service after a reboot manually.

Despite the fact that I changed the /etc/init.de/fhgfs-client by adding "openibd", openibd is the init.d script which initializes ib (scientific linux 6.4) and chkconfig --del and --add fhgfs-client after changing it, I had no success to change the behaviour.

### BEGIN INIT INFO

# Provides:          fhgfs-client

# Required-Start:    $network $local_fs $syslog $time

# Required-Stop:     $network $local_fs $syslog $time

# Should-Start:      fhgfs-helperd fhgfs-meta fhgfs-storage openibd openib $named slapd autofs ypbind nscd nslcd opensm rdma sshd

# Should-Stop:       fhgfs-helperd fhgfs-meta fhgfs-storage openibd openib $named slapd autofs ypbind nscd nslcd opensm rdma sshd

# Default-Start:     2 3 4 5

# Default-Stop:      0 1 6

# chkconfig:         35 99 5

# Short-Description: FhGFS Client

# Description:       Start FhGFS Client

### END INIT INFO

Funny though, the fhgfs-storage script has the openibd integrated per default. So I guess, the lack of "openibd" instead of "openib" in the fhgfs-client could be a bug!?

### BEGIN INIT INFO

# Provides:          fhgfs-storage

# Required-Start:    $network

# Should-Start:      openibd rdma

# Required-Stop:     $network

# Should-Stop:       openibd rdma

I run the newest release.

Thanks,

Mike

[Michael Ruepp]

Update:

Even changing fhgfs-client init script to:

### BEGIN INIT INFO

# Provides:          fhgfs-client

# Required-Start:    $network $local_fs $syslog $time openibd

# Required-Stop:     $network $local_fs $syslog $time openibd

# Should-Start:      fhgfs-helperd fhgfs-meta fhgfs-storage openib $named slapd autofs ypbind nscd nslcd opensm rdma sshd

# Should-Stop:       fhgfs-helperd fhgfs-meta fhgfs-storage openib $named slapd autofs ypbind nscd nslcd opensm rdma sshd

# Default-Start:     2 3 4 5

# Default-Stop:      0 1 6

# chkconfig:         35 99 5

# Short-Description: FhGFS Client

# Description:       Start FhGFS Client

### END INIT INFO

makes no difference. Seems like openibd reports UP when ib0/1 arent actually.

After reboot, fhgfs-net shows TCP (ethernet)

dmesg looks like this:

mlx4_core 0000:05:00.0: irq 155 for MSI/MSI-X

<mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Oct 29 2013)

mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Oct 29 2013)

NET: Registered protocol family 10

lo: Disabled Privacy Extensions

Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16

Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16

ADDRCONF(NETDEV_UP): ib0: link is not ready

ADDRCONF(NETDEV_UP): ib1: link is not ready

ip6_tables: (C) 2000-2006 Netfilter Core Team

nf_conntrack version 0.5.0 (16384 buckets, 65536 max)

ADDRCONF(NETDEV_UP): eth0: link is not ready

igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

p4-clockmod: Warning: EST-capable CPU detected. The acpi-cpufreq module offers voltage scaling in addition to frequency scaling. You should use that instead of p4-clockmod, if possible.

fhgfs_opentk: modprobe: FhGFS OpenToolkit loaded. (ibverbs enabled)

fhgfs: modprobe(5895): File system registered. Type: fhgfs. Version: 2012.10-r9

eth0: no IPv6 routers present

mlx4_core 0000:04:00.0: mlx4_ib: Port 1 logical link is up

ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready

fhgfs: mount(5910): FhGFS mount ready.

mlx4_core 0000:05:00.0: mlx4_ib: Port 1 logical link is up

ADDRCONF(NETDEV_CHANGE): ib1: link becomes ready

ib0: no IPv6 routers present

type=1305 audit(1384875783.802:68545): auid=4294967295 ses=4294967295 subj=kernel op="remove rule" key=(null) list=4 res=1

type=1305 audit(1384875783.802:68546): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=kernel res=1

readahead-collector: sorting

readahead-collector: finished

ib1: no IPv6 routers present

[Sven Breuner]

Hi Michael,

this BrightComputing knowledge base article might apply to your case:
http://kb.brightcomputing.com/faq/index.php?action=artikel&cat=11&id=23&artlang=en

Best regards,
Sven

> Am Montag, 11. M�rz 2013 10:56:06 UTC+1 schrieb Sternberger:
>
> Hello!
>
> I've just noticed that my clients are after reboot connected to the storage
> and meta nodes
> via tcp. when i restart the client afterwards with the rc script they are on
> RDMA again.
> I think the problem is described also here
> http://kb.brightcomputing.com/faq/pdf.php?cat=11&id=23&artlang=en
> <http://kb.brightcomputing.com/faq/pdf.php?cat=11&id=23&artlang=en>
>
> OS: SL6.3
> FHGFS: 2011.04.r22-el6.x86_64
>
> dmesg:
> fhgfs_opentk: modprobe: FhGFS OpenToolkit loaded. (ibverbs enabled)
> fhgfs: modprobe(5108): File system registered. Type: fhgfs. Version: 2011.04-r22
> ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
> fhgfs: mount(5119): FhGFS mount ready.
>
> Is there a better way (without custom initrd) to fix this issue?
>
> best regards!
>
>
> Sven
>

> --
> You received this message because you are subscribed to the Google Groups
> "fhgfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to fhgfs-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Michael Ruepp]

I know this article.

However, I am curious why the dependency system is not working and I don´t want to fill up my initrd with the mellanox modules, because of the image nature of bright (maintain more images than absolutely necessary).

Thanks anyway,

Mike
_________________
michael ruepp
mic...@ruepp.at
fon +43 676 911 40 90
skype michaelruepp

CONFIDENTIALITY NOTICE
This message (including any attachments transmitted with it) contains confidential information and is intended only for the individual named herein. If you are not the herein named addressee you should not disseminate, distribute, copy or otherwise make use of this message. Please notify the sender immediately by e-mail if you have received this message by mistake, and delete it from your systems.

[Bernd Schubert]

Hello Michael,

fhgfs-client.log should tell you why ib-rdma does not work.

Also, please not that the ib subnet manager may need quite some time to
properly initialize an ib port - up to several minutes. So starting
services is not sufficient to know if IB is already working properly.
Again, fhgfs-client.log should give you some indication.


Best regards,
Bernd

[Michael Ruepp]

Hi,

yes, I know. It takes some time. I will load the modules in the initrd.

But is there any polling mechanism on the fhgfs-side to let switch to rdma when possible?

Should I decrease connNonPrimaryExpiration = 10000?

Thanks,

Mike

_________________
michael ruepp
mic...@ruepp.at
fon +43 676 911 40 90
skype michaelruepp

CONFIDENTIALITY NOTICE
This message (including any attachments transmitted with it) contains confidential information and is intended only for the individual named herein. If you are not the herein named addressee you should not disseminate, distribute, copy or otherwise make use of this message. Please notify the sender immediately by e-mail if you have received this message by mistake, and delete it from your systems.

[Bernd Schubert]

Hello Michael,

it depends, if fhgfs-client is started when no ib interface is up, it
will not try to switch later on to IB. If the interface was up, but not
usable it will every connNonPrimaryExpiration requests try to switch to ib.

Regards,
Bernd

[Michael Ruepp]

For your Information:

Add
rdma_ucm
rdma_cm
ib_addr
ib_ipoib
mlx4_core
mlx4_ib
mlx4_en
mlx5_core
mlx5_ib
ib_uverbs
ib_umad
ib_ucm
ib_sa
ib_cm
ib_mad
ib_core

to the initrd.

Now all fhgfs-net of all nodes are RDMA.

Seems to be the solution. IB takes some time.

Regards Mike

_________________
michael ruepp
mic...@ruepp.at
fon +43 676 911 40 90
skype michaelruepp

CONFIDENTIALITY NOTICE
This message (including any attachments transmitted with it) contains confidential information and is intended only for the individual named herein. If you are not the herein named addressee you should not disseminate, distribute, copy or otherwise make use of this message. Please notify the sender immediately by e-mail if you have received this message by mistake, and delete it from your systems.

[Yann Sagon]

I'm using Centos 6.5 and I'm having the same issue. I tried what was working for Michael Ruepp with no luck.

@Michael: here is what I did without success:

edit the file /etc/dracut.conf =>
add_drivers+="rdma_ucm rdma_cm ib_addr ib_ipoib mlx4_core mlx4_ib mlx4_en mlx5_core mlx5_ib ib_uverbs ib_umad ib_ucm ib_sa ib_cm ib_mad ib_core"

generate the initramfs:
dracut --force

reboot.

same as before.

Any clue?

[Sven Breuner]

Hi Yann,

Yann Sagon wrote on 04/25/2014 05:36 PM:
> I'm using Centos 6.5 and I'm having the same issue. I tried what was
> working for Michael Ruepp with no luck.
>
> @Michael: here is what I did without success:
>
> edit the file /etc/dracut.conf =>
> add_drivers+="rdma_ucm rdma_cm ib_addr ib_ipoib mlx4_core mlx4_ib
> mlx4_en mlx5_core mlx5_ib ib_uverbs ib_umad ib_ucm ib_sa ib_cm ib_mad
> ib_core"
>
> generate the initramfs:
> dracut --force
>
> reboot.
>
> same as before.
>
> Any clue?

Does everything work with RDMA if you restart the fhgfs-client a few
minutes after system boot (e.g. fhgfs-net shows RDMA connections from
the client to the servers), so you only have the problem when the
fhgfs-client is started during system boot?

If this is the case, then:

* Do you see the "ib0: link becomes ready" message in dmesg before or
after the "FhGFS mount ready" message in dmesg during system boot?

* Does your /var/log/fhgfs-client.log list the ib0 interface in the
"Usable NICs" log line during system boot?

* Is the init.d script that brings up your IPoIB and the corresponding
Infiniband interface IP address included as a dependency in the
"Should-Start" line of the /etc/init.d/fhgfs-client script? (Note that
this dependency line was updated in the latest fhgfs 2014.01-r5 release.)

Best regards,
Sven Breuner
Fraunhofer

> 2013-11-20 14:19 GMT+01:00 Michael Ruepp <mic...@ruepp.at

> <mailto:mic...@ruepp.at>>:

>
> For your Information:
>
> Add
> rdma_ucm
> rdma_cm
> ib_addr
> ib_ipoib
> mlx4_core
> mlx4_ib
> mlx4_en
> mlx5_core
> mlx5_ib
> ib_uverbs
> ib_umad
> ib_ucm
> ib_sa
> ib_cm
> ib_mad
> ib_core
>
> to the initrd.
>
> Now all fhgfs-net of all nodes are RDMA.
>
> Seems to be the solution. IB takes some time.
>
> Regards Mike
>
>
> _________________
> michael ruepp

> mic...@ruepp.at <mailto:mic...@ruepp.at>
> fon +43 676 911 40 90 <tel:%2B43%20676%20911%2040%2090>

> skype michaelruepp
>
> CONFIDENTIALITY NOTICE
> This message (including any attachments transmitted with it)
> contains confidential information and is intended only for the
> individual named herein. If you are not the herein named addressee
> you should not disseminate, distribute, copy or otherwise make use
> of this message. Please notify the sender immediately by e-mail if
> you have received this message by mistake, and delete it from your
> systems.
>
>
>
>
> On 20.11.2013, at 12:59, Bernd Schubert
> <bernd.s...@itwm.fraunhofer.de

> <mailto:bernd.s...@itwm.fraunhofer.de>> wrote:
>
> > Hello Michael,
> >
> > it depends, if fhgfs-client is started when no ib interface is
> up, it will not try to switch later on to IB. If the interface was
> up, but not usable it will every connNonPrimaryExpiration requests
> try to switch to ib.
> >
> > Regards,
> > Bernd
> >
> > On 11/20/2013 12:07 PM, Michael Ruepp wrote:
> >> Hi,
> >>
> >> yes, I know. It takes some time. I will load the modules in the
> initrd.
> >>
> >> But is there any polling mechanism on the fhgfs-side to let
> switch to rdma when possible?
> >>
> >> Should I decrease connNonPrimaryExpiration = 10000?
> >>
> >> Thanks,
> >>
> >> Mike
> >>
> >> _________________
> >> michael ruepp

> >> mic...@ruepp.at <mailto:mic...@ruepp.at>
> >> fon +43 676 911 40 90 <tel:%2B43%20676%20911%2040%2090>

> >> skype michaelruepp
> >>
> >> CONFIDENTIALITY NOTICE
> >> This message (including any attachments transmitted with it)
> contains confidential information and is intended only for the
> individual named herein. If you are not the herein named addressee
> you should not disseminate, distribute, copy or otherwise make use
> of this message. Please notify the sender immediately by e-mail if
> you have received this message by mistake, and delete it from your
> systems.
> >>
> >>
> >>
> >>
> >> On 20.11.2013, at 11:55, Bernd Schubert
> <bernd.s...@itwm.fraunhofer.de

> <mailto:bernd.s...@itwm.fraunhofer.de>> wrote:
> >>
> >>> Hello Michael,
> >>>
> >>> fhgfs-client.log should tell you why ib-rdma does not work.
> >>>
> >>> Also, please not that the ib subnet manager may need quite some
> time to properly initialize an ib port - up to several minutes. So
> starting services is not sufficient to know if IB is already working
> properly. Again, fhgfs-client.log should give you some indication.
> >>>
> >>>
> >>> Best regards,
> >>> Bernd
> >>>
> >>>
> >>> On 11/20/2013 09:14 AM, Michael Ruepp wrote:
> >>>> I know this article.
> >>>>
> >>>> However, I am curious why the dependency system is not working
> and I don´t want to fill up my initrd with the mellanox modules,
> because of the image nature of bright (maintain more images than
> absolutely necessary).
> >>>>
> >>>> Thanks anyway,
> >>>>
> >>>> Mike
> >>>> _________________
> >>>> michael ruepp

> >>>> mic...@ruepp.at <mailto:mic...@ruepp.at>
> >>>> fon +43 676 911 40 90 <tel:%2B43%20676%20911%2040%2090>

> >>>> skype michaelruepp
> >>>>
> >>>> CONFIDENTIALITY NOTICE
> >>>> This message (including any attachments transmitted with it)
> contains confidential information and is intended only for the
> individual named herein. If you are not the herein named addressee
> you should not disseminate, distribute, copy or otherwise make use
> of this message. Please notify the sender immediately by e-mail if
> you have received this message by mistake, and delete it from your
> systems.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 20.11.2013, at 02:49, Sven Breuner <bre...@itwm.fhg.de

> <mailto:fhgfs-user%2Bunsu...@googlegroups.com>.

> >>>>>> For more options, visit
> https://groups.google.com/groups/opt_out.
> >>>>>
> >>>>> --
> >>>>> You received this message because you are subscribed to the
> Google Groups "fhgfs-user" group.
> >>>>> To unsubscribe from this group and stop receiving emails from
> it, send an email to fhgfs-user+...@googlegroups.com

> <mailto:fhgfs-user%2Bunsu...@googlegroups.com>.

> >>>>> For more options, visit https://groups.google.com/groups/opt_out.
> >>>>
> >>>
> >>> --
> >>> You received this message because you are subscribed to the
> Google Groups "fhgfs-user" group.
> >>> To unsubscribe from this group and stop receiving emails from
> it, send an email to fhgfs-user+...@googlegroups.com

> <mailto:fhgfs-user%2Bunsu...@googlegroups.com>.

> >>> For more options, visit https://groups.google.com/groups/opt_out.
> >>
> >
> > --
> > You received this message because you are subscribed to the
> Google Groups "fhgfs-user" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email to fhgfs-user+...@googlegroups.com

> <mailto:fhgfs-user%2Bunsu...@googlegroups.com>.

> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google
> Groups "fhgfs-user" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to fhgfs-user+...@googlegroups.com

> <mailto:fhgfs-user%2Bunsu...@googlegroups.com>.

> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "fhgfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com

> <mailto:fhgfs-user+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Sven Breuner]

Hi Yann,

Yann Sagon wrote on 04/29/2014 04:14 PM:

2014-04-29 11:36 GMT+02:00 Sven Breuner <sven.b...@itwm.fraunhofer.de>:

 

* Does your /var/log/fhgfs-client.log list the ib0 interface in the "Usable NICs" log line during system boot?

Yes:

(2) Apr28 16:49:32 *mount(7583) [App_logInfos] >> Usable NICs: ib0(RDMA) ib0(TCP) eth0(TCP)

But then I have this:

(3) Apr28 16:49:40 *mount(7583) [NodeConn (acquire stream)] >> Connect failed: fhgfs...@192.168.102.2:8005 (protocol: RDMA)
(3) Apr28 16:49:40 *mount(7583) [NodeConn (acquire stream)] >> Connected: fhgfs...@192.168.102.2:8005 (protocol: TCP; fallback route)

and this:

(3) Apr28 16:49:40 *fhgfs_Worker/1(7586) [NodeConn (acquire stream)] >> Connected: fhgfs-...@192.168.102.2:8003 (protocol: RDMA)
(3) Apr28 16:49:40 *fhgfs_Worker/2(7587) [NodeConn (acquire stream)] >> Connected: fhgfs-...@192.168.102.3:8003 (protocol: RDMA)

So it seems that it's failling to connect using RDMA only for the metadata. Before the upgrade it was the case for both metadata and storage.

I don't know if this is relevant or not, but I have this as well that seems strange:

(3) Apr28 16:49:35 *fhgfs_HBeatMgr(7585) [NodeConn (acquire stream)] >> Connect failed: fhgfs...@192.168.102.1:8008 (protocol: TCP)
(3) Apr28 16:49:35 *fhgfs_HBeatMgr(7585) [NodeConn (acquire stream)] >> Connected: fhgfs...@129.194.x.x:8008 (protocol: TCP; fallback route)

The second ip correspond to the public ip of the master node. When I restart the client, it's using the internal ip.

As a side note: In case you want to avoid the public IP of the master being used at all, you might want to check out this link and set the connInterfacesFile in fhgfs-mgmtd.conf:
http://www.fhgfs.com/wiki/FAQ#multiple_nics

Regarding the fallback to TCP, it really looks like a timing issue with the ib interface getting fully ready and able just a bit too late (i.e. somewhere in the middle of the client mount procedure), so that's a general ib problem.

The good news is that appearently the interface is "ready enough" (e.g. is generally marked as "up" and has an IP address associated) when the client is mounted, so the client sees "ib0(RDMA)" as usable NIC and considers the TCP route only as a fallback. That means the TCP fallback connection will automatically be dropped after the number of requests given by connNonPrimaryExpiration in fhgfs-client.conf (set to 10,000 by default) and then the client will try to reconnect via RDMA.

Another way to force the client to drop connections and thus force a reconnection attempt via RDMA (e.g. if you find a way to reliably detect that the interface is completely up later) would be to use:
$ echo 1 > /proc/fs/fhgfs/*/drop_conns
on the client.

[Sven Breuner]

Hi Yann,

Yann Sagon wrote on 04/29/2014 04:14 PM:
> (2) Apr28 16:49:32 *mount(7583) [App_logInfos] >> Usable NICs: ib0(RDMA)
> ib0(TCP) eth0(TCP)
>
> But then I have this:
>
> (3) Apr28 16:49:40 *mount(7583) [NodeConn (acquire stream)] >> Connect
> failed: fhgfs...@192.168.102.2:8005

> <http://fhgfs...@192.168.102.2:8005> (protocol: RDMA)

> (3) Apr28 16:49:40 *mount(7583) [NodeConn (acquire stream)] >>
> Connected: fhgfs...@192.168.102.2:8005

> <http://fhgfs...@192.168.102.2:8005> (protocol: TCP; fallback route)

the fhgfs 2014.01-r6 update (already available in the customers login
area at http://www.fhgfs.com/customerlogin and publicly available in a
few days) contains improvements for this case.
With the r6 update, the primary route will be re-evaluated every 15
minutes (configurable). Thus, the client should automatically re-connect
via RDMA after a few minutes.