BeeGFS client not building for Rocky 5.14.0-503.26.1.el9_5.aarch64+64k kernel

[Florin Petrescu]

Greetings,

I have installed BeeGFS on a system client using:

dnf config-manager --add-repo https://www.beegfs.io/release/latest-stable/dists/beegfs-rhel9.repo

dnf -y install beegfs-mgmtd libbeegfs-ib beegfs-client beegfs-helperd beegfs-utils

The configuration files:

cat /etc/beegfs/beegfs-client.conf

# --- Section 1.1: [Basic Settings] ---
#
sysMgmtdHost                  = beegfs-mds01
#
# --- Section 1.2: [Advanced Settings] ---
#
connAuthFile                  = /etc/beegfs/connauthfile
connClientPortUDP             = 8004
connHelperdPortTCP            = 8006
connMgmtdPortTCP              = 8008
connMgmtdPortUDP              = 8008
connPortShift                 = 0
connCommRetrySecs             = 600
connFallbackExpirationSecs    = 900
connInterfacesFile            =
connRDMAInterfacesFile        =
connMaxInternodeNum           = 12
connMaxConcurrentAttempts     = 0
connNetFilterFile             =
connUseRDMA                   = true
connTCPFallbackEnabled        = true
connTCPRcvBufSize             = 0
connUDPRcvBufSize             = 0
connRDMABufNum                = 70
connRDMABufSize               = 8192
connRDMATypeOfService         = 0
connTcpOnlyFilterFile         =
logClientID                   = false
logHelperdIP                  =
logLevel                      = 3
logType                       = helperd
quotaEnabled                  = true
sysCreateHardlinksAsSymlinks  = false
sysMountSanityCheckMS         = 11000
sysSessionCheckOnClose        = false
sysSyncOnClose                = false
sysTargetOfflineTimeoutSecs   = 900
sysUpdateTargetStatesSecs     = 30
sysXAttrsEnabled              = false
tuneFileCacheType             = buffered
tunePreferredMetaFile         =
tunePreferredStorageFile      =
tuneRemoteFSync               = true
tuneUseGlobalAppendLocks      = false
tuneUseGlobalFileLocks        = false

cat /etc/beegfs/beegfs-helperd.conf

connAuthFile       = /etc/beegfs/connauthfile
connDisableAuthentication = false
connHelperdPortTCP = 8006
connPortShift      = 0
logNoDate          = false
logNumLines        = 50000
logNumRotatedFiles = 5
logStdFile         = /var/log/beegfs-client.log
runDaemonized      = true
tuneNumWorkers     = 2

cat /etc/beegfs/beegfs-mounts.conf
/mnt/beegfs /etc/beegfs/beegfs-client.conf

Building the client shows one error:

/etc/init.d/beegfs-client rebuild
- BeeGFS module autobuild
$OFED_INCLUDE_PATH = []
$OFED_INCLUDE_PATH = []
$OFED_INCLUDE_PATH = []
Building beegfs client module
feature detection gives: -DKERNEL_HAS_INODE_ATIME -DKERNEL_HAS_SCHED_SIG_H -DKERNEL_HAS_LINUX_STDARG_H -DKERNEL_HAS_STATX -DKERNEL_HAS_KREF_READ -DKERNEL_HAS_FILE_DENTRY                -DKERNEL_HAS_SUPER_SETUP_BDI_NAME -DKERNEL_HAS_KERNEL_READ -DKERNEL_HAS_SKWQ_HAS_SLEEPER -DKERNEL_HAS_CURRENT_TIME_SPEC64 -DKERNEL_WAKE_UP_SYNC_KEY_HAS_3_ARGUMENTS -DKE               RNEL_HAS_IOV_ITER_KVEC_NO_TYPE_FLAG_IN_DIRECTION -DKERNEL_HAS_PROC_OPS -DKERNEL_HAS_SOCKPTR_T -DKERNEL_HAS_SOCK_SETSOCKOPT_SOCKPTR_T_PARAM -DKERNEL_HAS_TIME64 -DKERNEL_H               AS_KTIME_GET_TS64 -DKERNEL_HAS_KTIME_GET_REAL_TS64 -DKERNEL_HAS_KTIME_GET_COARSE_REAL_TS64 -DKERNEL_HAS_GENERIC_FILE_SPLICE_READ -DKERNEL_HAS_SETATTR_PREPARE -DKERNEL_HA               S_GET_ACL -DKERNEL_HAS_SET_ACL -DKERNEL_HAS_SET_ACL_NS_INODE -DKERNEL_HAS_USER_NS_MOUNTS -DKERNEL_HAS_FOPS_ITERATE -DKERNEL_HAS_XATTR_HANDLERS_INODE_ARG -DKERNEL_HAS_INO               DE_GET_SET_CTIME
feature detection gives: -DKERNEL_HAS_INODE_ATIME -DKERNEL_HAS_SCHED_SIG_H -DKERNEL_HAS_LINUX_STDARG_H -DKERNEL_HAS_STATX -DKERNEL_HAS_KREF_READ -DKERNEL_HAS_FILE_DENTRY                -DKERNEL_HAS_SUPER_SETUP_BDI_NAME -DKERNEL_HAS_KERNEL_READ -DKERNEL_HAS_SKWQ_HAS_SLEEPER -DKERNEL_HAS_CURRENT_TIME_SPEC64 -DKERNEL_WAKE_UP_SYNC_KEY_HAS_3_ARGUMENTS -DKE               RNEL_HAS_IOV_ITER_KVEC_NO_TYPE_FLAG_IN_DIRECTION -DKERNEL_HAS_PROC_OPS -DKERNEL_HAS_SOCKPTR_T -DKERNEL_HAS_SOCK_SETSOCKOPT_SOCKPTR_T_PARAM -DKERNEL_HAS_TIME64 -DKERNEL_H               AS_KTIME_GET_TS64 -DKERNEL_HAS_KTIME_GET_REAL_TS64 -DKERNEL_HAS_KTIME_GET_COARSE_REAL_TS64 -DKERNEL_HAS_GENERIC_FILE_SPLICE_READ -DKERNEL_HAS_SETATTR_PREPARE -DKERNEL_HA               S_GET_ACL -DKERNEL_HAS_SET_ACL -DKERNEL_HAS_SET_ACL_NS_INODE -DKERNEL_HAS_USER_NS_MOUNTS -DKERNEL_HAS_FOPS_ITERATE -DKERNEL_HAS_XATTR_HANDLERS_INODE_ARG -DKERNEL_HAS_INO               DE_GET_SET_CTIME
Skipping BTF generation for /opt/beegfs/src/client/client_module_7/build/../source/beegfs.ko due to unavailability of vmlinux
$OFED_INCLUDE_PATH = []
$OFED_INCLUDE_PATH = []

Starting the client fails:

systemctl status beegfs-client.service
× beegfs-client.service - Start BeeGFS Client
     Loaded: loaded (/usr/lib/systemd/system/beegfs-client.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Fri 2025-02-28 12:47:04 CST; 1h 14min ago
    Process: 17088 ExecStart=/etc/init.d/beegfs-client start (code=exited, status=1/FAILURE)
   Main PID: 17088 (code=exited, status=1/FAILURE)
        CPU: 1min 21.918s

Feb 28 12:46:46 gpugh201.cluster beegfs-client[18025]: feature detection gives: -DKERNEL_HAS_INODE_ATIME -DKERNEL_HAS_SCHED_SIG_H -DKERNEL_HAS_LINUX_STDARG_H -DKERNEL_HAS_STATX -DKERN>
Feb 28 12:47:01 gpugh201.cluster beegfs-client[20496]: feature detection gives: -DKERNEL_HAS_INODE_ATIME -DKERNEL_HAS_SCHED_SIG_H -DKERNEL_HAS_LINUX_STDARG_H -DKERNEL_HAS_STATX -DKERN>
Feb 28 12:47:01 gpugh201.cluster beegfs-client[20836]: Skipping BTF generation for /opt/beegfs/src/client/client_module_7/build/../source/beegfs.ko due to unavailability of vmlinux
Feb 28 12:47:02 gpugh201.cluster beegfs-client[20844]: $OFED_INCLUDE_PATH = []
Feb 28 12:47:04 gpugh201.cluster beegfs-client[21088]: $OFED_INCLUDE_PATH = []
Feb 28 12:47:04 gpugh201.cluster beegfs-client[21340]: modprobe: ERROR: could not insert 'beegfs': Invalid argument
Feb 28 12:47:04 gpugh201.cluster systemd[1]: beegfs-client.service: Main process exited, code=exited, status=1/FAILURE
Feb 28 12:47:04 gpugh201.cluster systemd[1]: beegfs-client.service: Failed with result 'exit-code'.
Feb 28 12:47:04 gpugh201.cluster systemd[1]: Failed to start Start BeeGFS Client.
Feb 28 12:47:04 gpugh201.cluster systemd[1]: beegfs-client.service: Consumed 1min 21.918s CPU time.
systemctl start beegfs-client.service

systemctl start beegfs-client.service
Job for beegfs-client.service failed because the control process exited with error code.
See "systemctl status beegfs-client.service" and "journalctl -xeu beegfs-client.service" for details.

journalctl -xeu beegfs-client.service
░░
░░ A start job for unit beegfs-client.service has begun execution.
░░
░░ The job identifier is 7597.
Feb 28 14:01:41 gpugh201.cluster beegfs-client[25050]: Starting BeeGFS Client:
Feb 28 14:01:41 gpugh201.cluster beegfs-client[25050]: - Loading BeeGFS modules
Feb 28 14:01:41 gpugh201.cluster beegfs-client[25053]: modprobe: ERROR: could not insert 'beegfs': Invalid argument
Feb 28 14:01:41 gpugh201.cluster beegfs-client[25050]: - BeeGFS module autobuild
Feb 28 14:01:41 gpugh201.cluster beegfs-client[25061]: $OFED_INCLUDE_PATH = []
Feb 28 14:01:41 gpugh201.cluster beegfs-client[25308]: $OFED_INCLUDE_PATH = []
Feb 28 14:01:42 gpugh201.cluster beegfs-client[25560]: $OFED_INCLUDE_PATH = []
Feb 28 14:01:42 gpugh201.cluster beegfs-client[25801]: Building beegfs client module
Feb 28 14:01:49 gpugh201.cluster beegfs-client[25987]: feature detection gives: -DKERNEL_HAS_INODE_ATIME -DKERNEL_HAS_SCHED_SIG_H -DKERNEL_HAS_LINUX_STDARG_H -DKERNEL_HAS_STATX -DKERN>
Feb 28 14:02:04 gpugh201.cluster beegfs-client[28460]: feature detection gives: -DKERNEL_HAS_INODE_ATIME -DKERNEL_HAS_SCHED_SIG_H -DKERNEL_HAS_LINUX_STDARG_H -DKERNEL_HAS_STATX -DKERN>
Feb 28 14:02:04 gpugh201.cluster beegfs-client[28800]: Skipping BTF generation for /opt/beegfs/src/client/client_module_7/build/../source/beegfs.ko due to unavailability of vmlinux
Feb 28 14:02:04 gpugh201.cluster beegfs-client[28808]: $OFED_INCLUDE_PATH = []
Feb 28 14:02:06 gpugh201.cluster beegfs-client[29052]: $OFED_INCLUDE_PATH = []
Feb 28 14:02:07 gpugh201.cluster beegfs-client[29304]: modprobe: ERROR: could not insert 'beegfs': Invalid argument
Feb 28 14:02:07 gpugh201.cluster systemd[1]: beegfs-client.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://wiki.rockylinux.org/rocky/support
░░
░░ An ExecStart= process belonging to unit beegfs-client.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Feb 28 14:02:07 gpugh201.cluster systemd[1]: beegfs-client.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://wiki.rockylinux.org/rocky/support
░░
░░ The unit beegfs-client.service has entered the 'failed' state with result 'exit-code'.
Feb 28 14:02:07 gpugh201.cluster systemd[1]: Failed to start Start BeeGFS Client.
░░ Subject: A start job for unit beegfs-client.service has failed
░░ Defined-By: systemd
░░ Support: https://wiki.rockylinux.org/rocky/support
░░
░░ A start job for unit beegfs-client.service has finished with a failure.
░░
░░ The job identifier is 7597 and the job result is failed.
Feb 28 14:02:07 gpugh201.cluster systemd[1]: beegfs-client.service: Consumed 1min 22.082s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://wiki.rockylinux.org/rocky/support
░░
░░ The unit beegfs-client.service completed and consumed the indicated resources.

beegfs-net
df: no file systems processed
No active BeeGFS mounts found.

Ideas?

Thank you for your attention to this matter.

[Fedor Pollak]

Hi,

do you see any errors in the messages (system logs)?

Did you specified RDMA drivers? Have you tried to start it without RDMA support?

Best regards,
Fedor

[Tore H. Larsen]

Hi Florin,

Did you adjust /etc/alternative/ofa_kernel_headers  ?   

And specify MLNX OFED in the build script? 

In my experience, you should ensure that it is updated when you update kernels. At least on Ubuntu 22/24.

And then do /etc/init.d/beegfs-client rebuild

E.g. 

root@g002:~# ll /etc/alternatives/ofa_kernel_headers
lrwxrwxrwx 1 root root 45 Mar  6 15:05 /etc/alternatives/ofa_kernel_headers -> /usr/src/ofa_kernel/x86_64/5.15.0-1072-nvidia/

root@g002:~# grep -v ^# /etc/beegfs/beegfs-client-autobuild.conf

buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/include/
buildEnabled=true

--

Kind Regards / Mvh,

Tore H. Larsen

Chief Research Engineer HPC                                      Email:  to...@simula.no

Simula Research Laboratory - HPC department        Mobile: +47 918 33 670

Kristian Augusts gate 23, 0164 Oslo, Norway

--
You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/fhgfs-user/dd874dae-dee2-4420-8a8b-ddbe2ec3df31n%40googlegroups.com.