Infiniband works, but no RDMA

[Urs Ganse]

Dear BeeGFS people,

I've installed BeeGFS according on a couple of cluster nodes with
Mellanox ConnectX-3 Infiniband-HBAs, and it works fine via TCP, however
no RDMA interfaces are detected.

The opentk library autodetects and enables infiniband:
- - - - - - - - - - - - - - - - -
root@node-1:/mnt# fhgfs-opentk-lib-update-ib
Running Infiniband auto-detection...

Setting symlink in /opt/fhgfs/lib: libfhgfs-opentk.so ->
libfhgfs-opentk-enabledIB.so
- - - - - - - - - - - - - - - - -

Also building the client module works successfully, and lsmod does show
a dependency of it for rdma_cm.

However, the log files only report TCP interfaces (even for the ib0 device):
- - - - - - - - - - - - - - - - -
(2) Jan27 17:54:49 Main [App] >> Usable NICs: ib0(TCP) br0(TCP) br0(TCP)
(4) Jan27 17:54:49 Main [App] >> Extended list of usable NICs:
+ ib0[ip addr: 10.11.13.3; hw addr: 80.00.00.48.fe.80; metric: 0;
bandwidth: 66; type: TCP]
+ br0[ip addr: 172.23.0.1; hw addr: 0c.c4.7a.0f.99.a0; metric: 0;
bandwidth: 4; type: TCP]
+ br0[ip addr: 10.11.12.2; hw addr: 0c.c4.7a.0f.99.a0; metric: 0;
bandwidth: 4; type: TCP]
- - - - - - - - - - - - - - - - -
and no other messages mentioning Infiniband or RDMA are occurring.

RDMA itself works fine on the hosts, as I am successfully using NFS/RDMA
on the same systems.

What other conditions could there be that prevent RDMA from working?

Cheers,

//Urs Ganse

[Urs Ganse]

On 28/01/15 09:08, Urs Ganse wrote:
> However, the log files only report TCP interfaces (even for the ib0
> device):
> - - - - - - - - - - - - - - - - -
> (2) Jan27 17:54:49 Main [App] >> Usable NICs: ib0(TCP) br0(TCP) br0(TCP)
> (4) Jan27 17:54:49 Main [App] >> Extended list of usable NICs:
> + ib0[ip addr: 10.11.13.3; hw addr: 80.00.00.48.fe.80; metric: 0;
> bandwidth: 66; type: TCP]
> + br0[ip addr: 172.23.0.1; hw addr: 0c.c4.7a.0f.99.a0; metric: 0;
> bandwidth: 4; type: TCP]
> + br0[ip addr: 10.11.12.2; hw addr: 0c.c4.7a.0f.99.a0; metric: 0;
> bandwidth: 4; type: TCP]
> - - - - - - - - - - - - - - - - -
> and no other messages mentioning Infiniband or RDMA are occurring.

Ah, looks like I just kept looking at the wrong log file.
While the mgmt-daemon only uses TCP, everything else seems to be running
via RDMA.

Thanks anyway, and keep up the good work!

Cheers,

//Urs