Infiniband on Ubuntu 16.04

[Chip Schweiss]

I'm following the wiki page on setting up Infiniband support, but I'm not able to get the client module to run.

I have the distribution OFED installed:  

# apt search ofed

Sorting... Done

Full Text Search... Done

hping3/xenial 3.a2.ds2-7 amd64

  Active Network Smashing Tool

knem-dkms/now 1.1.2.90mlnx2-OFED.4.0.1.6.3.1.g4faa297 all [installed,local]

  DKMS support for mlnx-ofed kernel modules

mlnx-ofed-kernel-dkms/now 4.0-OFED.4.0.2.0.0.1.g595cb61 all [installed,local]

  DKMS support for mlnx-ofed kernel modules

mlnx-ofed-kernel-utils/now 4.0-OFED.4.0.2.0.0.1.g595cb61 amd64 [installed,local]

  Userspace tools to restart and tune mlnx-ofed kernel modules

ofed-scripts/now 4.0-OFED.4.0.2.0.0 amd64 [installed,local]

  MLNX_OFED utilities

ramond/xenial 0.5-4 amd64

  IPv6 Router Advertisement MONitoring Daemon

In /etc/beegfs/beegfs-client-autobuild.conf:

buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1

The client rebuilds cleanly:

# /etc/init.d/beegfs-client rebuild

- BeeGFS module autobuild

Building beegfs client module

However the module will not load:

# /etc/init.d/beegfs-client restart

Shutting down BeeGFS Client:

- Unmounting directories from /etc/beegfs/beegfs-mounts.conf

- Unloading modules

Starting BeeGFS Client:

- Loading BeeGFS modules

modprobe: ERROR: could not insert 'beegfs': Invalid argument

- BeeGFS module autobuild

Building beegfs client module

modprobe: ERROR: could not insert 'beegfs': Invalid argument

WARNING: You probably need to specify OFED_INCLUDE_PATH in /etc/beegfs/beegfs-client-autobuild.conf

What am I missing here?

Thanks!

-Chip

[Schweiss, Chip]

Additional info:

I found the there are version problems in the module. However, the build appears to use the correct kernel sources.

From /var/log/syslog:

Oct 16 12:21:40 wr-master kernel: [ 9895.105496] beegfs: disagrees about version of symbol ib_create_cq

Oct 16 12:21:40 wr-master kernel: [ 9895.105503] beegfs: Unknown symbol ib_create_cq (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105551] beegfs: disagrees about version of symbol rdma_resolve_addr

Oct 16 12:21:40 wr-master kernel: [ 9895.105554] beegfs: Unknown symbol rdma_resolve_addr (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105572] beegfs: disagrees about version of symbol rdma_set_service_type

Oct 16 12:21:40 wr-master kernel: [ 9895.105575] beegfs: Unknown symbol rdma_set_service_type (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105648] beegfs: disagrees about version of symbol ib_dereg_mr

Oct 16 12:21:40 wr-master kernel: [ 9895.105650] beegfs: Unknown symbol ib_dereg_mr (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105662] beegfs: disagrees about version of symbol rdma_reject

Oct 16 12:21:40 wr-master kernel: [ 9895.105664] beegfs: Unknown symbol rdma_reject (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105685] beegfs: disagrees about version of symbol rdma_disconnect

Oct 16 12:21:40 wr-master kernel: [ 9895.105687] beegfs: Unknown symbol rdma_disconnect (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105757] beegfs: disagrees about version of symbol rdma_resolve_route

Oct 16 12:21:40 wr-master kernel: [ 9895.105759] beegfs: Unknown symbol rdma_resolve_route (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105765] beegfs: disagrees about version of symbol rdma_bind_addr

Oct 16 12:21:40 wr-master kernel: [ 9895.105767] beegfs: Unknown symbol rdma_bind_addr (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105777] beegfs: disagrees about version of symbol rdma_create_qp

Oct 16 12:21:40 wr-master kernel: [ 9895.105778] beegfs: Unknown symbol rdma_create_qp (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105794] beegfs: disagrees about version of symbol ib_destroy_cq

Oct 16 12:21:40 wr-master kernel: [ 9895.105796] beegfs: Unknown symbol ib_destroy_cq (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105807] beegfs: disagrees about version of symbol rdma_create_id

Oct 16 12:21:40 wr-master kernel: [ 9895.105809] beegfs: Unknown symbol rdma_create_id (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105869] beegfs: disagrees about version of symbol rdma_listen

Oct 16 12:21:40 wr-master kernel: [ 9895.105871] beegfs: Unknown symbol rdma_listen (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105876] beegfs: disagrees about version of symbol rdma_destroy_qp

Oct 16 12:21:40 wr-master kernel: [ 9895.105878] beegfs: Unknown symbol rdma_destroy_qp (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.105949] beegfs: Unknown symbol ib_get_dma_mr (err 0)

Oct 16 12:21:40 wr-master kernel: [ 9895.105985] beegfs: Unknown symbol ib_alloc_pd (err 0)

Oct 16 12:21:40 wr-master kernel: [ 9895.106014] beegfs: disagrees about version of symbol rdma_connect

Oct 16 12:21:40 wr-master kernel: [ 9895.106016] beegfs: Unknown symbol rdma_connect (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.106052] beegfs: disagrees about version of symbol rdma_destroy_id

Oct 16 12:21:40 wr-master kernel: [ 9895.106054] beegfs: Unknown symbol rdma_destroy_id (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.106077] beegfs: disagrees about version of symbol rdma_accept

Oct 16 12:21:40 wr-master kernel: [ 9895.106079] beegfs: Unknown symbol rdma_accept (err -22)

Oct 16 12:21:40 wr-master kernel: [ 9895.106124] beegfs: disagrees about version of symbol ib_dealloc_pd

Oct 16 12:21:40 wr-master kernel: [ 9895.106126] beegfs: Unknown symbol ib_dealloc_pd (err -22)

Uname:

# uname -r

4.4.0-97-generic

Turning off silent in /etc/init.d/beegfs-client rebuild:

make: Entering directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

make auto_rebuild_clean -j8 BEEGFS_OPENTK_IBVERBS=1

make[1]: Entering directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

rm -f *~ .beegfs??*

rm -f .*.cmd *.mod.c *.mod.o *.o *.ko *.ko.unsigned

rm -f Module*.symvers modules.order Module.markers

rm -f auto_rebuild_kernel.ver

rm -rf .tmp_versions/

find ../source/ -mount -name '*.o' -type f -delete

find ../source/ -mount -name '.*.o.cmd' -type f -delete

find ../source/ -mount -name '.*.o.d' -type f -delete

rm -f

make[1]: Leaving directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

make -j8 BEEGFS_OPENTK_IBVERBS=1

make[1]: Entering directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

Building beegfs client module

make -C /lib/modules/4.4.0-97-generic/build SUBDIRS=/opt/beegfs/src/client/beegfs_client_module_6/build/../source \

"EXTRA_CFLAGS=  -DOFED_HAS_SET_SERVICE_TYPE -DOFED_HAS_IB_CREATE_CQATTR -DOFED_HAS_NETNS -DOFED_SPLIT_WR   -DKERNEL_HAS_DROP_NLINK -DKERNEL_HAS_CLEAR_NLINK -DKERNEL_HAS_TEST_LOCK_VOID_TWOARGS -DKERNEL_HAS_ATTR_OPEN -DKERNEL_HAS_IHOLD -DKERNEL_HAS_TASK_IO_ACCOUNTING -DKERNEL_HAS_FSYNC_RANGE -DKERNEL_HAS_INC_NLINK -DKERNEL_HAS_S_D_OP  -DKERNEL_HAS_PDE_DATA -DKERNEL_HAS_GET_UNALIGNED_LE -DKERNEL_HAS_I_UID_READ -DKERNEL_HAS_ATOMIC_OPEN -DKERNEL_HAS_UMODE_T -DKERNEL_HAS_2_PARAM_INIT_WORK -DKERNEL_HAS_WRITE_CACHE_PAGES -DKERNEL_HAS_LIST_IS_LAST -DKERNEL_HAS_DIRECT_IO_ITER  -DKERNEL_HAS_FILE_INODE   -DKERNEL_HAS_POSIX_GET_ACL -DKERNEL_HAS_CONST_XATTR_HANDLER    -DKERNEL_HAS_IOV_ITER_TYPE -DKERNEL_HAS_IOV_ITER_FUNCTIONS  -DKERNEL_HAS_IOV_ITER_TRUNCATE  -DKERNEL_HAS_CURRENT_UMASK -DKERNEL_HAS_SHOW_OPTIONS_DENTRY -DKERNEL_HAS_XATTR_HANDLER_PTR_ARG -DKERNEL_HAS_DENTRY_XATTR_HANDLER  -DKERNEL_HAS_LOCKS_LOCK_INODE_WAIT  -DKERNEL_HAS_GENERIC_SEMAPHORE -DKERNEL_HAS_IS_VMALLOC_ADDR -DKERNEL_HAS_I_MMAP_LOCK -DKERNEL_HAS_I_MMAP_RWSEM  -DKERNEL_HAS_I_MMAP_RBTREE  -DKERNEL_HAS_LONG_IOV_DIO    -DKERNEL_HAS_INODE_LOCK -DKERNEL_HAS_PAGE_ENDIO   -DKERNEL_HAS_FILE_REMOVE_PRIVS  -DKERNEL_HAS_MEMDUP_USER -DKERNEL_HAS_CURRENT_FSUID -DKERNEL_HAS_FIRST_NET_DEVICE_NS  -DKERNEL_HAS_FAULTATTR_DNAME -DKERNEL_HAS_MNT_WANT_WRITE  -DKERNEL_HAS_IN4_PTON -DKERNEL_HAS_KSTRNDUP -DKERNEL_HAS_SOCK_CREATE_KERN_NS -DKERNEL_HAS_SOCK_SENDMSG_NOLEN -DKERNEL_HAS_MSGHDR_ITER -DKERNEL_HAS_IOV_ITER_INIT_DIR -DKERNEL_HAS_ITER_BVEC -DKERNEL_HAS_ITER_IS_IOVEC  -DKERNEL_HAS_IOV_ITER_IOVEC   -DKERNEL_HAS_SET_NLINK -DKERNEL_HAS_DENTRY_PATH_RAW -DKERNEL_HAS_MAPPING_SET_ERROR   -DKERNEL_HAS_ITER_FILE_SPLICE_WRITE  -DKERNEL_HAS_ITERATE_DIR -DKERNEL_HAS_ENCODE_FH_INODE -DKERNEL_HAS_D_DELETE_CONST_ARG  -DKERNEL_HAS_POSIX_ACL_XATTR_USERNS_ARG -DKERNEL_HAS_D_MAKE_ROOT -DKERNEL_HAS_GENERIC_WRITE_CHECKS_ITER -DKERNEL_HAS_GENERIC_PERMISSION_2  -DKERNEL_HAS_WRITE_ITER  -DKERNEL_HAS_INVALIDATEPAGE_RANGE -DKERNEL_HAS_PERMISSION_2     -DKERNEL_HAS_PROC_CREATE_DATA -DKERNEL_HAS_SB_BDI -DKERNEL_HAS_BDI_SETUP_AND_REGISTER -DKERNEL_HAS_FOLLOW_LINK_COOKIE -DKERNEL_HAS_FILEMAP_WRITE_AND_WAIT_RANGE   -DKERNEL_HAS_SET_ACL -DKERNEL_HAS_IOCB_DIRECT  -DKERNEL_HAS_COPY_FROM_ITER -DKERNEL_HAS_INIT_WORK_2 -DKERNEL_HAS_ALLOC_WORKQUEUE   -DKERNEL_HAS_CURRENT_FS_TIME  -DKERNEL_HAS_GENERIC_GETXATTR  -I/opt/beegfs/src/client/beegfs_client_module_6/build/../source -Wno-unused-parameter -DBEEGFS_MODULE_NAME_STR='\"beegfs\"' -Wuninitialized '-DBEEGFS_VERSION=\"6.16\"' '-DBEEGFS_VERSION_CODE=50728960' -DBEEGFS_OPENTK_IBVERBS" modules

make[2]: Entering directory '/usr/src/linux-headers-4.4.0-97-generic'

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/fault-inject/fault-inject.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/os/atomic64.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/os/OsCompat.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/os/OsDeps.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/net/message/NetMessageFactory.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/net/filesystem/FhgfsOpsCommKit.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/net/filesystem/FhgfsOpsCommKitVec.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/net/filesystem/FhgfsOpsRemoting.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/system/System.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/Socket.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/NicAddress.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/NetworkInterfaceCard.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/RDMASocket.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVSocket.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/IBVBuffer.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/ibv/No_IBVSocket.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/sock/StandardSocket.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/control/PeerInfoMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/control/AckMsgEx.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/helperd/GetHostByNameMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/helperd/LogMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/helperd/GetHostByNameRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/SimpleIntMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/NetMessage.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/locking/FLockEntryMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/locking/FLockAppendMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/locking/LockGrantedMsgEx.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/locking/FLockRangeMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/GetFileVersionRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/rw/ReadLocalFileV2Msg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/rw/WriteLocalFileMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/FSyncLocalFileMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/GetFileVersionMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/opening/OpenFileMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/opening/OpenFileRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/opening/CloseFileMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/session/BumpFileVersion.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/SimpleInt64Msg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/SimpleUInt16Msg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/SimpleStringMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/SimpleIntStringMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/RegisterNodeRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/RegisterNodeMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/HeartbeatRequestMsgEx.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/MapTargetsMsgEx.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/GetMirrorBuddyGroupsRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/GetTargetMappingsRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/HeartbeatMsgEx.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/RemoveNodeMsgEx.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/GetTargetStatesRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/GetStatesAndBuddyGroupsRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/RefreshTargetStatesMsgEx.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/GetNodesRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/nodes/SetMirrorBuddyGroupMsgEx.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/SimpleMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/moving/RenameMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/StatStoragePathRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/creating/HardlinkMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/creating/MkFileMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/creating/MkDirMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/creating/UnlinkFileMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/creating/MkDirRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/creating/RmDirMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/creating/MkFileRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/StatRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/ListXAttrRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/SetAttrMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/ListXAttrMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/RefreshEntryInfoMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/StatMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/GetXAttrRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/GetXAttrMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/RemoveXAttrMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/attribs/SetXAttrMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/lookup/LookupIntentRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/lookup/LookupIntentMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/TruncFileMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/listing/ListDirFromOffsetRespMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/net/message/storage/listing/ListDirFromOffsetMsg.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/threading/Thread.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/Serialization.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/MessagingTk.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/HashTk.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/ackstore/WaitAckMap.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/ackstore/AcknowledgmentStore.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/ackstore/AckStoreMap.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/Random.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/tree/IntMap.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/tree/StrCpyMap.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/tree/TargetStateInfoMap.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/tree/TargetNodeMap.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/tree/PointerRBTree.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/tree/MirrorBuddyGroupMap.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/tree/UInt16Map.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/NodesTk.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/SocketTk.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/MetadataTk.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/TimeAbs.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/StringTk.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/toolkit/ListTk.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/nodes/Node.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/nodes/TargetMapper.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/nodes/MirrorBuddyGroup.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/nodes/NumNodeID.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/nodes/TargetStateStore.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/nodes/NodeTree.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/nodes/MirrorBuddyGroupMapper.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/nodes/NodeConnPool.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/storage/StorageErrors.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/storage/StatData.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/storage/striping/BuddyMirrorPattern.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/storage/striping/SimplePattern.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/storage/striping/StripePattern.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/storage/striping/Raid10Pattern.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/storage/striping/Raid0Pattern.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/storage/PathInfo.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/program/Main.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/common/storage/EntryInfo.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/toolkit/FhgfsChunkPageVec.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/toolkit/StatFsCache.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/toolkit/NoAllocBufferStore.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/toolkit/InodeRefStore.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/toolkit/ExternalHelperd.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/toolkit/BitStore.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/components/DatagramListener.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/components/InternodeSyncer.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/components/Flusher.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/components/worker/RWPagesWork.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/components/AckManager.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/nodes/NodeStoreEx.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/ProcFs.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOpsPages.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOpsFile.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOpsInode.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOpsFileNative.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsInode.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/helper/IoctlHelper.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FsFileInfo.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/ProcFsHelper.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOpsSuper.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOps_versions.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsXAttrHandlers.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOpsDir.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOpsIoctl.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOpsExport.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/filesystem/FhgfsOpsHelper.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/app/App.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/app/log/Logger.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/app/config/MountConfig.o

  CC [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/app/config/Config.o

  LD [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/beegfs.o

  Building modules, stage 2.

  MODPOST 1 modules

  CC      /opt/beegfs/src/client/beegfs_client_module_6/build/../source/beegfs.mod.o

  LD [M]  /opt/beegfs/src/client/beegfs_client_module_6/build/../source/beegfs.ko

make[2]: Leaving directory '/usr/src/linux-headers-4.4.0-97-generic'

make[1]: Leaving directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

make install -j8 BEEGFS_OPENTK_IBVERBS=1

make[1]: Entering directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

install -D -m 644 beegfs.ko /lib/modules/4.4.0-97-generic/updates/fs/beegfs_autobuild/beegfs.ko

depmod -a

make[1]: Leaving directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

make: Leaving directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

make: Entering directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

rm -f *~ .beegfs??*

rm -f .*.cmd *.mod.c *.mod.o *.o *.ko *.ko.unsigned

rm -f Module*.symvers modules.order Module.markers

rm -f auto_rebuild_kernel.ver

rm -rf .tmp_versions/

find ../source/ -mount -name '*.o' -type f -delete

find ../source/ -mount -name '.*.o.cmd' -type f -delete

find ../source/ -mount -name '.*.o.d' -type f -delete

rm -f

make: Leaving directory '/opt/beegfs/src/client/beegfs_client_module_6/build'

Any hints on why the kernel module is not building properly would be greatly appreciated.

Thanks!

-Chip

--
You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sven Breuner]

Hi Chip,

the loaded RDMA drivers are not the ones that come included with your distro's
kernel.
You can see that from "/etc/init.d/beegfs-client restart" complaining with this
message:

"WARNING: You probably need to specify OFED_INCLUDE_PATH in
/etc/beegfs/beegfs-client-autobuild.conf"

This means the ib_core kernel driver came in from a speparate package. You can
confirm this by calling:
$ modinfo ib_core
...and you will most likely see here that the ib_core driver is in a
subdirectory named "updates", which means that a separate driver was installed.

You can either uninstall this separate driver and use the drivers that came with
the distro; or you need to specify the correct path to the header files for this
separate driver as OFED_INCLUDE_PATH in /etc/beegfs/beegfs-client-autobuild.conf.
Using "dpkg -S rdma_cm.h" might help you find the right path if the
corresponding .deb package was installed. You can also find examples for
OFED_INCLUDE_PATH values in beegfs-client-autobuild.conf.

(We would make this more convenient if it would be possible, but unfortunately,
it is not possible to automatically decide for a build mechanism whether it is
building against the correct headers or where the correct headers files are
located. This can only be detected by the kernel when you try to actually load a
module.)

Best regards,
Sven

[Schweiss, Chip]

Sven,

Thank you!  

Turns out my system had Mellanox drivers loaded.

After setting the buildArgs:

buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/include

It compiles and runs. 

Cheers!

-Chip