Rocky Linux 8.4 and BeeGFS 7.2.3: OFED driver 5.4 not supported

571 views
Skip to first unread message

Manzano, Cristina

unread,
Aug 30, 2021, 4:57:01 AM8/30/21
to fhgfs...@googlegroups.com, Peter Niessen

Hi,

 

we are testing Rocky Linux 8.4 and found a problem when trying to install BeeGFS: it seems the Mellanox OFED driver version is not compatible with the version of BeeGFS we want to install.

 

We need BeeGFS 7.2.3 but Rocky 8.4 ships OFED 5.4 which is not supported and the auto build balks at some changed headers in OFED 5.4.

 

Is there a way to build the kernel module such that

 

[root@dp-cn50 build]# modinfo beegfs

filename:

/lib/modules/4.18.0-305.7.1.el8_4.x86_64/updates/fs/beegfs_autobuild/beegfs.ko

version:        7.2.3

alias:          fs-beegfs

author:         Fraunhofer ITWM, CC-HPC

description:    BeeGFS parallel file system client (http://www.beegfs.com)

license:        GPL v2

rhelversion:    8.4

srcversion:     533BB7E5866E52F63B9ACCB

depends:        ib_core,rdma_cm                  *** depends on ib_core,

rdma_cm ***

name:           beegfs

vermagic:       4.18.0-305.7.1.el8_4.x86_64 SMP mod_unload modversions

 

we get rid of the ib_core and rdma_cm dependencies?

 

Or even better: Is there a release date for OFED 5.4 support?

 

Thanks in advance for your help!

 

Cheers,

 

Cristina Manzano

 

--
Cristina Manzano
Juelich Supercomputing Centre JSC
Telefon: +49 2461 61-1958

Skype: c.manzano82
Internet: https://www.fz-juelich.de/ias/jsc/

 

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Forschungszentrum Juelich GmbH

52425 Juelich

Sitz der Gesellschaft: Juelich

Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498

Vorsitzender des Aufsichtsrats: MinDir Volker Rieke

Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),

Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,

Prof. Dr. Frauke Melchior

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

 

Philipp Falk

unread,
Aug 30, 2021, 9:47:04 AM8/30/21
to fhgfs...@googlegroups.com, Peter Niessen
Hello Christina,

we are currently also in the process of testing Rocky 8.4 and I am not yet
aware of any client build issues. Could you send us the full error output
you get when you try to build the module?

Best regards
- Philipp

* Manzano, Cristina <c.ma...@fz-juelich.de> [210830 10:57]:
> https://www.fz-juelich.de/ias/jsc/
>
>
>
> ----------------------------------------------------------------------------
> -----------------
>
> ----------------------------------------------------------------------------
> -----------------
>
> Forschungszentrum Juelich GmbH
>
> 52425 Juelich
>
> Sitz der Gesellschaft: Juelich
>
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>
> Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
>
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>
> Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,
>
> Prof. Dr. Frauke Melchior
>
> ----------------------------------------------------------------------------
> -----------------
>
> ----------------------------------------------------------------------------
> -----------------
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/0bb2ee5673bb4156859361a3c1658ff2%40fz-juelich.de.



--
Philipp Falk | System Engineer | m: philip...@thinkparq.com
ThinkParQ GmbH | Trippstadter Strasse 113 | 67663 Kaiserslautern | Germany
CEO: Frank Herold | COB: Dr. Franz-Josef Pfreundt | Registered: Amtsgericht Kaiserslautern HRB 31565 I VAT-ID-No.:DE 292001792

***Please note our change of address above***

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or proprietary information.

Manzano, Cristina

unread,
Aug 31, 2021, 8:25:41 AM8/31/21
to fhgfs...@googlegroups.com, Peter Niessen
Hi,

sure, I am using the following building arguments:

buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/include

and when trying to rebuild with "/etc/init.d/beegfs-client rebuild" I get the following output (also attached):


- BeeGFS module autobuild
Building beegfs client module
In file included from ./include/linux/skbuff.h:29,
from /usr/src/ofa_kernel/default/include/linux/skbuff.h:7,
from ./include/linux/if_ether.h:23,
from /usr/src/ofa_kernel/default/include/linux/if_ether.h:6,
from ./include/uapi/linux/ethtool.h:19,
from ./include/linux/ethtool.h:18,
from /usr/src/ofa_kernel/default/include/linux/ethtool.h:6,
from ./include/linux/netdevice.h:41,
from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
/usr/src/ofa_kernel/default/include/linux/net.h:19:20: Fehler: Redefinition von »sendpage_ok«
static inline bool sendpage_ok(struct page *page)
^~~~~~~~~~~
In file included from /usr/src/ofa_kernel/default/include/linux/net.h:6,
from ./include/linux/skbuff.h:29,
from /usr/src/ofa_kernel/default/include/linux/skbuff.h:7,
from ./include/linux/if_ether.h:23,
from /usr/src/ofa_kernel/default/include/linux/if_ether.h:6,
from ./include/uapi/linux/ethtool.h:19,
from ./include/linux/ethtool.h:18,
from /usr/src/ofa_kernel/default/include/linux/ethtool.h:6,
from ./include/linux/netdevice.h:41,
from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
./include/linux/net.h:318:20: Anmerkung: Vorherige Definition von »sendpage_ok« war hier
static inline bool sendpage_ok(struct page *page)
^~~~~~~~~~~
In file included from ./include/linux/netdevice.h:53,
from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
/usr/src/ofa_kernel/default/include/uapi/linux/pkt_cls.h:233:2: Fehler: Redeklaration von Aufzählung »TCA_FLOWER_KEY_CT_FLAGS_INVALID«
TCA_FLOWER_KEY_CT_FLAGS_INVALID = 1 << 4, /* Conntrack is invalid. */
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/src/ofa_kernel/default/include/uapi/linux/pkt_cls.h:7,
from ./include/linux/netdevice.h:53,
from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
./include/uapi/linux/pkt_cls.h:594:2: Anmerkung: Vorherige Definition von »TCA_FLOWER_KEY_CT_FLAGS_INVALID« war hier
TCA_FLOWER_KEY_CT_FLAGS_INVALID = 1 << 4, /* Conntrack is invalid. */
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./include/linux/netdevice.h:53,
from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
/usr/src/ofa_kernel/default/include/uapi/linux/pkt_cls.h:234:2: Fehler: Redeklaration von Aufzählung »TCA_FLOWER_KEY_CT_FLAGS_REPLY«
TCA_FLOWER_KEY_CT_FLAGS_REPLY = 1 << 5, /* Packet is in the reply direction. */
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/src/ofa_kernel/default/include/uapi/linux/pkt_cls.h:7,
from ./include/linux/netdevice.h:53,
from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
./include/uapi/linux/pkt_cls.h:595:2: Anmerkung: Vorherige Definition von »TCA_FLOWER_KEY_CT_FLAGS_REPLY« war hier
TCA_FLOWER_KEY_CT_FLAGS_REPLY = 1 << 5, /* Packet is in the reply direction. */
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
/usr/src/ofa_kernel/default/include/linux/netdevice.h:309:8: Fehler: Redefinition von »struct netdev_net_notifier«
struct netdev_net_notifier {
^~~~~~~~~~~~~~~~~~~
In file included from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
./include/linux/netdevice.h:1006:8: Anmerkung: ursprünglich hier definiert
struct netdev_net_notifier {
^~~~~~~~~~~~~~~~~~~
In file included from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
/usr/src/ofa_kernel/default/include/linux/netdevice.h:315:1: Fehler: In Konflikt stehende Typen für »register_netdevice_notifier_dev_net«
register_netdevice_notifier_dev_net(struct net_device *dev,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
./include/linux/netdevice.h:2744:5: Anmerkung: Vorherige Deklaration von »register_netdevice_notifier_dev_net« war hier
int register_netdevice_notifier_dev_net(struct net_device *dev,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
/usr/src/ofa_kernel/default/include/linux/netdevice.h:323:1: Fehler: In Konflikt stehende Typen für »unregister_netdevice_notifier_dev_net«
unregister_netdevice_notifier_dev_net(struct net_device *dev,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
./include/linux/netdevice.h:2747:5: Anmerkung: Vorherige Deklaration von »unregister_netdevice_notifier_dev_net« war hier
int unregister_netdevice_notifier_dev_net(struct net_device *dev,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
/usr/src/ofa_kernel/default/include/linux/netdevice.h:338:20: Fehler: Redefinition von »net_prefetch«
static inline void net_prefetch(void *p)
^~~~~~~~~~~~
In file included from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
./include/linux/netdevice.h:2316:20: Anmerkung: Vorherige Definition von »net_prefetch« war hier
static inline void net_prefetch(void *p)
^~~~~~~~~~~~
In file included from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
/usr/src/ofa_kernel/default/include/linux/netdevice.h:346:20: Fehler: Redefinition von »net_prefetchw«
static inline void net_prefetchw(void *p)
^~~~~~~~~~~~~
In file included from /usr/src/ofa_kernel/default/include/linux/netdevice.h:7,
from /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.c:8:
./include/linux/netdevice.h:2324:20: Anmerkung: Vorherige Definition von »net_prefetchw« war hier
static inline void net_prefetchw(void *p)
^~~~~~~~~~~~~
make[3]: *** [scripts/Makefile.build:315: /opt/beegfs/src/client/client_module_7/build/../source/os/OsDeps.o] Fehler 1
make[3]: *** Es wird auf noch nicht beendete Prozesse gewartet....
make[2]: *** [Makefile:1563: _module_/opt/beegfs/src/client/client_module_7/build/../source] Error 2
make[1]: *** [Makefile:140: module] Error 2
make: *** [AutoRebuild.mk:34: auto_rebuild] Error 2


Cheers,

Cristina Manzano

--
Cristina Manzano
Juelich Supercomputing Centre JSC
Telefon: +49 2461 61-1958
Skype: c.manzano82
Internet: https://www.fz-juelich.de/ias/jsc/

---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,
Prof. Dr. Frauke Melchior
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------

-----Ursprüngliche Nachricht-----
Von: fhgfs...@googlegroups.com [mailto:fhgfs...@googlegroups.com] Im Auftrag von Philipp Falk
Gesendet: Montag, 30. August 2021 15:47
An: fhgfs...@googlegroups.com
Cc: Peter Niessen <nie...@par-tec.com>
Betreff: Re: [beegfs-user] Rocky Linux 8.4 and BeeGFS 7.2.3: OFED driver 5.4 not supported
To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/YSzhVPq9GdMHDuE/%40xps13.
out

Vinícius Ferrão

unread,
Sep 2, 2021, 3:23:39 AM9/2/21
to beegfs-user, Peter Niessen
Hi Cristina,

I don't think BeeGFS 7.2.3 runs on EL 8.4 yet: https://doc.beegfs.io/latest/release_notes.html

That does not seems to be a Rocky specific issue but a version specific issue.

I'm not aware of anyone running BeeGFS on EL8.4 right now.

Guan Xin

unread,
Sep 4, 2021, 3:29:03 AM9/4/21
to beegfs-user
Been running BeeGFS 7.2.1 on CentOS 8.4 for more than two months and no issue.
Will upgrade to BeeGFS 7.2.3 when I have time.

Kernel: 4.18.0-305.3.1.el8.x86_64
MLNX_OFED_LINUX-5.3-1.0.0.1

Guan

Manzano, Cristina

unread,
Sep 6, 2021, 5:04:37 AM9/6/21
to fhgfs...@googlegroups.com

Hi,

 

thank you all for your feedback. The issue had to do with the installed OFED package: we were using one precompiled rpm and this seemed to be breaking things for us. So now we downloaded the driver from the official Mellanox site and compiled it from scratch with ./mlnxofedinstall -vvv --add-kernel-support --distro RHEL8.4. Things are working fine now.

 

Best regards,

 

Cristina Manzano

 

--
Cristina Manzano
Juelich Supercomputing Centre JSC
Telefon: +49 2461 61-1958

Skype: c.manzano82
Internet: https://www.fz-juelich.de/ias/jsc/

 

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Forschungszentrum Juelich GmbH

52425 Juelich

Sitz der Gesellschaft: Juelich

Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498

Vorsitzender des Aufsichtsrats: MinDir Volker Rieke

Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),

Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,

Prof. Dr. Frauke Melchior

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

 

Von: fhgfs...@googlegroups.com [mailto:fhgfs...@googlegroups.com] Im Auftrag von Guan Xin
Gesendet: Samstag, 4.
September 2021 09:29
An: beegfs-user <fhgfs...@googlegroups.com>
Betreff: Re: [beegfs-user] Rocky Linux 8.4 and BeeGFS 7.2.3: OFED driver 5.4 not supported

 

Been running BeeGFS 7.2.1 on CentOS 8.4 for more than two months and no issue.

Manzano, Cristina

unread,
Sep 6, 2021, 5:45:25 AM9/6/21
to fhgfs...@googlegroups.com

Hi,

 

small correction just in case it helps someone, these were actually the options used for the compilation:

 

./mlnxofedinstall -vvv --add-kernel-support --distro RHEL8.4 --without-fw-update

 

Best,

 

Cristina Manzano

 

--
Cristina Manzano
Juelich Supercomputing Centre JSC
Telefon: +49 2461 61-1958

Skype: c.manzano82
Internet: https://www.fz-juelich.de/ias/jsc/

 

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Forschungszentrum Juelich GmbH

52425 Juelich

Sitz der Gesellschaft: Juelich

Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498

Vorsitzender des Aufsichtsrats: MinDir Volker Rieke

Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),

Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,

Prof. Dr. Frauke Melchior

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

 

Nick Thorne

unread,
Sep 16, 2021, 4:24:13 PM9/16/21
to beegfs-user
Hello,

I'm facing similar issues and I'm curious about Guan's config because I'm running the same kernel:
4.18.0-305.3.1.el8_4.x86_64 but a newer Mellanox OFED because MLNX_OFED_LINUX-5.3-1.0.0.1 doesn't officially support CentOS8.4 - it ends at CentOS8.3.
I tried installing it anyway and it failed when building the kmod packages which is what I expected seeing as it doesn't know about that kernel.

mine errors out like this and I'm only just delving into why:
 /etc/init.d/beegfs-client rebuild

- BeeGFS module autobuild
Building beegfs client module
/opt/beegfs/src/client/client_module_7/build/../source/os/OsCompat.c: In function ‘d_walk’:
/opt/beegfs/src/client/client_module_7/build/../source/os/OsCompat.c:334:53: error: macro "spin_release" passed 3 arguments, but takes just 2
    spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
                                                     ^
/opt/beegfs/src/client/client_module_7/build/../source/os/OsCompat.c:334:4: error: ‘spin_release’ undeclared (first use in this function); did you mean ‘seq_release’?
    spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
    ^~~~~~~~~~~~
    seq_release
/opt/beegfs/src/client/client_module_7/build/../source/os/OsCompat.c:334:4: note: each undeclared identifier is reported only once for each function it appears in
make[3]: *** [scripts/Makefile.build:315: /opt/beegfs/src/client/client_module_7/build/../source/os/OsCompat.o] Error 1
make[3]: *** Waiting for unfinished jobs....

make[2]: *** [Makefile:1563: _module_/opt/beegfs/src/client/client_module_7/build/../source] Error 2
make[1]: *** [Makefile:137: module] Error 2

make: *** [AutoRebuild.mk:34: auto_rebuild] Error 2

build env=
MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.4-x86_64
Rocky Linux release 8.4 (Green Obsidian)
4.18.0-305.3.1.el8_4.x86_64
beegfs-client-7.2-el8.noarch

Just figured I'd post a "and me" for anyone out there searching for similar. I'll try and remember to post a solution if I get one.
Thanks
Nick

Nick Thorne

unread,
Sep 16, 2021, 4:24:16 PM9/16/21
to beegfs-user
Update on my issue - it was due to the BeeGFS version being flat 7_2 because I cloned the wrong repo. When I cloned this:
the kernel module built fine. Unfortunately my cluster is down temporarily so I can't actually test a mount but it should technically be fine.

On Monday, 6 September 2021 at 04:45:25 UTC-5 Manzano, Cristina wrote:

Guan Xin

unread,
Sep 18, 2021, 3:54:49 AM9/18/21
to beegfs-user
Hello Nick,

Unpack the MLNX_OFED tgz, then rebuild with
  ./mlnx_add_kernel_support.sh --mlnx_ofed . --distro rhel8.4 --make-tgz

Guan
Reply all
Reply to author
Forward
0 new messages