BeeGFS can't connect to RDMA

[Mertcan Guldu]

I’m running BeeGFS 8.2.2 and I’m having trouble getting RDMA to work between my BeeGFS clients and the BeeGFS servers. With RDMA enabled, the client is unable to establish a connection (TCP works, but RDMA does not).

● beegfs-storage.service - BeeGFS Storage Server
     Loaded: loaded (/lib/systemd/system/beegfs-storage.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2026-01-28 21:19:05 UTC; 8s ago
       Docs: http://www.beegfs.com/content/documentation/
   Main PID: 70497 (beegfs-storage/)
      Tasks: 20 (limit: 230996)
     Memory: 2.2M
        CPU: 8ms
     CGroup: /system.slice/beegfs-storage.service
             └─70497 /opt/beegfs/sbin/beegfs-storage cfgFile=/etc/beegfs/beegfs-storage.conf runDaemonized=false

Jan 28 21:19:05 bee01 beegfs-storage[70497]: Main [Sync results] >> Nodes added: 4 (Type: beegfs-storage)
Jan 28 21:19:05 bee01 beegfs-storage[70497]: Main [Sync results] >> Nodes added: 1 (Type: beegfs-meta)
Jan 28 21:19:05 bee01 beegfs-storage[70497]: Main [App] >> Registration and management info download complete.
Jan 28 21:19:05 bee01 beegfs-storage[70497]: Main [DGramLis] >> Listening for UDP datagrams: any Port 8003
Jan 28 21:19:05 bee01 beegfs-storage[70497]: Main [ConnAccept] >> Listening for TCP connections: Port 8003
Jan 28 21:19:05 bee01 beegfs-storage[70497]: Main [App] >> 1 sessions restored.
Jan 28 21:19:05 bee01 beegfs-storage[70497]: Main [App] >> Version: 8.2.2
Jan 28 21:19:05 bee01 beegfs-storage[70497]: Main [App] >> LocalNode: beegfs-storage node_storage_101 [ID: 101]
Jan 28 21:19:05 bee01 beegfs-storage[70497]: Main [App] >> Usable NICs:
                                             + enp33s0f1np1[ip addr: 192.168.3.60; type: TCP]



- root@bee01:~# ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 28.44.1036
node_guid: 605e:6503:002e:48cc
sys_image_guid: 605e:6503:002e:48cc
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000834
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 28.44.1036
node_guid: 605e:6503:002e:48cd
sys_image_guid: 605e:6503:002e:48cc
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000834
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

[Mertcan Guldu]

root@bee01:/opt/beegfs/debs-client# ofed_info -s
MLNX_OFED_LINUX-23.07-0.5.1.2:

[Johannes Sennahoj]

Same here: RDMA works well with 8.1, but not with 8.2. It always falls back to TCP. However, I did not investigate further and rolled back to 8.1. RDMA Nic is shown as available, but cannot establish a connection (dmesg).

[Joe McCormick]

Hello,

Please open an issue for this on GitHub and include the following output:

(1) Any relevant logs from clients and servers that are unable to establish connections using RDMA.

(2) Output of `beegfs health net` from a client that is unable to connect using RDMA.

(3) Output of `beegfs node list --with-nics` (optionally truncated to only include the servers and the client selected for (2)).

(4) Output from `beegfs node ping`.

All output would need to be collected while the system is on 8.2.2 and showing fallbacks to TCP.

Thank you,

~Joe

[Faraz Hussain]

I have run into similar issues and it was due to our Rocky Linux version not supporting the Mellanox drivers. Can dig up more history if you need.