Building beegfs client with RDMA on RHEL 7.4

[meh...@soysal.eu]

Hi,
we upgraded our nodes to RHEL 7.4 and switched from mellanox OFED to Redhat OFED/RDMA repo.
I have some problems to build the client with RDMA support.

It also seems that some packages were obsolete and have been replaced, e.b.
Package libibverbs-devel-1.2.1-1.el7.x86_64 is obsoleted by rdma-core-devel-13-7.el7.x86_64 which is already installed


I tried adding the line
buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/include/infiniband

to the /etc/beegfs/beegfs-client-autobuild.conf file.
I get this error while trying to rebuild the client module:
 /etc/init.d/beegfs-client rebuild
- BeeGFS module autobuild
Building beegfs client module
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c: In function ‘__IBVSocket_createNewID’:
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:87:7: warning: passing argument 1 of ‘rdma_create_id’ from incompatible pointer type [enabled by default]
       new_cm_id = rdma_create_id(__IBVSocket_cmaHandler, _this, RDMA_PS_TCP);
       ^
In file included from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVBuffer.h:7:0,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.h:4,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:8:
include/rdma/rdma_cm.h:172:20: note: expected ‘struct net *’ but argument is of type ‘int (*)(struct rdma_cm_id *, struct rdma_cm_event *)’
 struct rdma_cm_id *rdma_create_id(struct net *net,
                    ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:87:7: warning: passing argument 2 of ‘rdma_create_id’ from incompatible pointer type [enabled by default]
       new_cm_id = rdma_create_id(__IBVSocket_cmaHandler, _this, RDMA_PS_TCP);
       ^
In file included from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVBuffer.h:7:0,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.h:4,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:8:
include/rdma/rdma_cm.h:172:20: note: expected ‘rdma_cm_event_handler’ but argument is of type ‘struct IBVSocket *’
 struct rdma_cm_id *rdma_create_id(struct net *net,
                    ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:87:7: warning: passing argument 3 of ‘rdma_create_id’ makes pointer from integer without a cast [enabled by default]
       new_cm_id = rdma_create_id(__IBVSocket_cmaHandler, _this, RDMA_PS_TCP);
       ^
In file included from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVBuffer.h:7:0,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.h:4,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:8:
include/rdma/rdma_cm.h:172:20: note: expected ‘void *’ but argument is of type ‘int’
 struct rdma_cm_id *rdma_create_id(struct net *net,
                    ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:87:7: error: too few arguments to function ‘rdma_create_id’
       new_cm_id = rdma_create_id(__IBVSocket_cmaHandler, _this, RDMA_PS_TCP);
       ^
In file included from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVBuffer.h:7:0,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.h:4,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:8:
include/rdma/rdma_cm.h:172:20: note: declared here
 struct rdma_cm_id *rdma_create_id(struct net *net,
                    ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c: In function ‘__IBVSocket_createCommContext’:
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:506:47: error: macro "ib_alloc_pd" requires 2 arguments, but only 1 given
    commContext->pd = ib_alloc_pd(cm_id->device);
                                               ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:506:22: error: ‘ib_alloc_pd’ undeclared (first use in this function)
    commContext->pd = ib_alloc_pd(cm_id->device);
                      ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:506:22: note: each undeclared identifier is reported only once for each function it appears in
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:525:4: error: implicit declaration of function ‘ib_get_dma_mr’ [-Werror=implicit-function-declaration]
    commContext->dmaMR = ib_get_dma_mr(commContext->pd,
    ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:525:23: warning: assignment makes pointer from integer without a cast [enabled by default]
    commContext->dmaMR = ib_get_dma_mr(commContext->pd,
                       ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c: In function ‘IBVSocket_checkConnection’:
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:842:25: error: ‘struct ib_send_wr’ has no member named ‘wr’
 # define rdma_of(wr) (wr.wr.rdma)
                         ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:854:4: note: in expansion of macro ‘rdma_of’
    rdma_of(wr).remote_addr = _this->remoteDest->vaddr;
    ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:842:25: error: ‘struct ib_send_wr’ has no member named ‘wr’
 # define rdma_of(wr) (wr.wr.rdma)
                         ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:855:4: note: in expansion of macro ‘rdma_of’
    rdma_of(wr).rkey = _this->remoteDest->rkey;
    ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c: In function ‘__IBVSocket_createCompletionQueue’:
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:1821:7: warning: passing argument 5 of ‘ib_create_cq’ makes pointer from integer without a cast [enabled by default]
       return ib_create_cq(device, comp_handler, event_handler, cq_context, cqe, 0);
       ^
In file included from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVBuffer.h:6:0,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.h:4,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:8:
include/rdma/ib_verbs.h:2904:15: note: expected ‘const struct ib_cq_init_attr *’ but argument is of type ‘int’
 struct ib_cq *ib_create_cq(struct ib_device *device,
               ^
/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:1821:7: error: too many arguments to function ‘ib_create_cq’
       return ib_create_cq(device, comp_handler, event_handler, cq_context, cqe, 0);
       ^
In file included from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVBuffer.h:6:0,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.h:4,
                 from /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.c:8:
include/rdma/ib_verbs.h:2904:15: note: declared here
 struct ib_cq *ib_create_cq(struct ib_device *device,
               ^
cc1: some warnings being treated as errors
make[3]: *** [/opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [_module_/opt/beegfs/src/client/beegfs_client_module_6/build/../source] Error 2
make[1]: *** [module] Error 2
make: *** [auto_rebuild] Error 2


--------------------------
rpm -qa | grep beegfs
beegfs-meta-6.16-el7.x86_64
beegfs-common-6.16-el7.noarch
beegfs-opentk-lib-6.16-el7.x86_64
beegfs-client-6.16-el7.noarch
beegfs-storage-6.16-el7.x86_64
beegfs-mgmtd-6.16-el7.x86_64
beegfs-utils-6.16-el7.x86_64
beegfs-helperd-6.16-el7.x86_64


And ideas what im doing wrong ?


best regards
M.Soysal

[Jérémie Sebban]

Hi,

I recently built the latest  BeeGFS client successfully on CentOS 7.4 using inbox - ie. shipped with OS-  RDMA drivers.
- You need to install "Infiniband Support" packages group
# yum groupinstall "infiniband support"
- Make sure rdma service is running
- With the "inbox" drivers, I use the following in my beegfs-client-autobuild.conf

buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/kernels/3.10.0-693.2.2.el7.x86_64/include

Hope that helps...

[meh...@soysal.eu]

Hi,
i tried that already but the client keeps using only tcp, while the storage shows RDMA connection.
~# uname -a
Linux node01 3.10.0-693.2.2.el7.x86_64 #1 SMP Sat Sep 9 03:55:24 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
~# yum grouplist
Installed Groups:
   infiniband
~# grep OFED /etc/beegfs/beegfs-client-autobuild.conf
buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/kernels/3.10.0-693.2.2.el7.x86_64/include

Setting up beeond with 2 nodes:
~#   beeond start -n nodefile -d /tmp/beeond -c /mnt/beeond
root@fhcn0001:~# beegfs-ctl --listnodes  --nodetype=storage  --details  --mount=/mnt/beeond
node01 [ID: 1]
   Ports: UDP: 9003; TCP: 9003
   Interfaces: ib0(RDMA) ib0(TCP) eno1(TCP)
node02 [ID: 2]
   Ports: UDP: 9003; TCP: 9003
   Interfaces: ib0(RDMA) ib0(TCP) eno1(TCP)

Number of nodes: 2
:~# beegfs-ctl --listnodes  --nodetype=client  --details  --mount=/mnt/beeond
9C0E-59D25550-node01 [ID: 1]
   Ports: UDP: 9004; TCP: 0
   Interfaces: ib0(TCP) eno1(TCP)
1698-59D25550-node02 [ID: 2]
   Ports: UDP: 9004; TCP: 0
   Interfaces: ib0(TCP) eno1(TCP)

Number of nodes: 2

[meh...@soysal.eu]

hi,
i found my mistake.
The old module was still loaded.
After i rebuild a new module i forget to unload the old without RDMA support.


best regards
M.soysal

[Sven Breuner]

Hi Jérémie and Mehmet,

Jérémie Sebban wrote on 02.10.2017 16:22:
> Hi,
>
> I recently built the latest  BeeGFS client successfully on CentOS 7.4 using
> inbox - ie. shipped with OS-  RDMA drivers.
> - You need to install "Infiniband Support" packages group
> # yum groupinstall "infiniband support"
> - Make sure rdma service is running
> - With the "inbox" drivers, I use the following in my beegfs-client-autobuild.conf
>
> buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1
> OFED_INCLUDE_PATH=/usr/src/kernels/3.10.0-693.2.2.el7.x86_64/include

when using the drives that come shipped with RHEL/CentOS, you can just omit the
OFED_INCLUDE_PATH. This way you don't have to touch anything when you update the
kernel later.

Best regards,
Sven