beegfs-client (7.2.5) failed to build kernel module on RHEL 8.5

[Ümit Seren]

When trying to build the kernel module for beegfs-client on RHEL 8.5 we get following error:

                                         

- BeeGFS module autobuild

Building beegfs client module

/opt/beegfs/src/client/client_module_7/build/../source/common/net/sock/ibv/IBVBuffer.c: In function ‘IBVBuffer_init’:

/opt/beegfs/src/client/client_module_7/build/../source/common/net/sock/ibv/IBVBuffer.c:26:28: error: implicit declaration of function ‘ib_dma_alloc_coherent’; did you mean ‘dma_alloc_coherent’? [-Werror=implicit-function-declaration]

       buffer->buffers[i] = ib_dma_alloc_coherent(ctx->pd->device, bufLen, &buffer->lists[i].addr,

                            ^~~~~~~~~~~~~~~~~~~~~

                            dma_alloc_coherent

/opt/beegfs/src/client/client_module_7/build/../source/common/net/sock/ibv/IBVBuffer.c:26:26: warning: assignment to ‘char *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]

       buffer->buffers[i] = ib_dma_alloc_coherent(ctx->pd->device, bufLen, &buffer->lists[i].addr,

                          ^

/opt/beegfs/src/client/client_module_7/build/../source/common/net/sock/ibv/IBVBuffer.c:41:10: error: implicit declaration of function ‘ib_dma_free_coherent’; did you mean ‘dma_free_coherent’? [-Werror=implicit-function-declaration]

          ib_dma_free_coherent(ctx->pd->device, buffer->bufferSize, buffer->buffers[i],

          ^~~~~~~~~~~~~~~~~~~~

          dma_free_coherent

cc1: some warnings being treated as errors

make[3]: *** [scripts/Makefile.build:316: /opt/beegfs/src/client/client_module_7/build/../source/common/net/sock/ibv/IBVBuffer.o] Error 1

make[3]: *** Waiting for unfinished jobs....

make[2]: *** [Makefile:1571: _module_/opt/beegfs/src/client/client_module_7/build/../source] Error 2

make[1]: *** [Makefile:144: module] Error 2

This is with a default /etc/beegfs/beegfs-client-autobuild.conf config. 

This wasn't the issue with 7.2.1 
According to the 7.2.5 documentation IB support is now enabled by default in 7.2.5. 

I checked but it seems also that the ib_dma_alloc_coherent functions where removed from the 4.18.0 kernel: 

[Ümit Seren]

I forgot to link the patch where the function was removed: https://www.spinics.net/lists/linux-rdma/msg97261.html

I checked the kernel source files and it seems that the last kernel version that contains the ib_dma_alloc_coherent function is 4.18.0-305.19.1

In kernel version 4.18.0-315.el8 the function is already removed. 

[Iain Milne]

So can we edit/patch the beegfs source to work around this (as has happened in the past)? My C/C++ is way too rusty to even contemplate it.

I still don't understand why this never happened with RHEL/CentOS 7.x and yet it seems almost every single time we go from 8.1 to 8.2 etc something has changed that stops beegfs from compiling. I thought these kernels were supposed to be stable for the entire lifetime of the major release?

[Dr. Thomas Orgis]

Am Thu, 18 Nov 2021 06:32:24 -0800 (PST)
schrieb Iain Milne <goo...@noognet.org>:

> I still don't understand why this never happened with RHEL/CentOS 7.x and
> yet it seems almost every single time we go from 8.1 to 8.2 etc something

It also happened with 7.x …

> I thought these kernels were
> supposed to be stable for the entire lifetime of the major release?

… because those kernels are _heavily_ patched and fit nowhere into the
kernel.org release numbers. RHEL 7.x had kernel 3.10, but in name only.
It contains lots of code from 4.x and maybe even 5.x nowadays.

That's why I switched to vanilla 4.14 at some point. To have a stable
LTS kernel and not what the distro provides. The distro supports
userspace API for a long time and tries to mess little with things like
base libs or the compiler (little, not nothing).

It should still be fine to run the 8.4 kernel on 8.5 system, though —
just those missing patches for the newest root exploits …


Alrighty then,

Thomas

--
Dr. Thomas Orgis
HPC @ Universität Hamburg

[Guan Xin]

These are simple aliases according to:

  https://patchwork.kernel.org/project/linux-rdma/patch/20201104095052...@lst.de/

and so must be easy to workaround, though I haven't tried because of horrible recent bug reports on BeeGFS 7.1.5.

[Ümit Seren]

I am curious which bugs in 7.2.5 you are referring to because we are considering of upgrading our production beegfs system to 7.2

5 and would want to avoid it if there are major issues.

Best

Ümit

--
You received this message because you are subscribed to a topic in the Google Groups "beegfs-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fhgfs-user/XVspSRtcFFQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fhgfs-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/ef2156cf-0b51-452b-9073-24190de98dban%40googlegroups.com.

[Guan Xin]

o that's 7.1.5 in the "Metadata issues after beegfs-fsck" thread. Blame my aged eyesight!

i didn't see 7.2.5 in the source repo other than some commit comments. Has it been released, or still immature?

We did encounter some non-fatal problems with 7.2.1, though. Will report in another thread in some minutes.

Best,

Guan

[Steffen Grunewald]

On Fri, 2021-11-19 at 00:52:16 -0800, Guan Xin wrote:
> o that's 7.1.5 in the "Metadata issues after beegfs-fsck" thread. Blame my
> aged eyesight!
> i didn't see 7.2.5 in the source repo other than some commit comments. Has
> it been released, or still immature?

# git log | head -n20
commit f85c4a62b4b4de0efd6ca9789a634b20604ee8f6
Author: Philipp Falk <philip...@thinkparq.com>
Date: Fri Nov 5 12:46:52 2021 +0100

update to release 7.2.5

...
# grep url .git/config
url = https://git.beegfs.io/pub/v7.git


Binaries (amd64) for 7.2.5 are also there... https://www.beegfs.io/c/download/

- S



--
Steffen Grunewald, Cluster Administrator
Max Planck Institute for Gravitational Physics (Albert Einstein Institute)
Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany
~~~
Fon: +49-331-567 7274
Mail: steffen.grunewald(at)aei.mpg.de
~~~

[André Gemünd]

It should be very easy to patch, as the functions were just wrappers. As you can see on the link

       buffer->buffers[i] = ib_dma_alloc_coherent(ctx->pd->device, bufLen, &buffer->lists[i].addr,

should just become

       buffer->buffers[i] = dma_alloc_coherent(ctx->pd->device->dma_device, bufLen, &buffer->lists[i].addr,

imho. I didn't test it though.

Greetings

André

--
You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/2e9c4916-47f8-464d-9755-c8efe27f47b0n%40googlegroups.com.

--

Dipl.-Inf. André Gemünd, Leiter IT / Head of IT
Fraunhofer-Institute for Algorithms and Scientific Computing
andre....@scai.fraunhofer.de
Tel: +49 2241 14-4199
/C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend

[Iain Milne]

Thanks for that. I've just tried (along with tweaking the other two lines in a similar way) and it appears ok. Always difficult to know for sure with storage, although in our case we don't even use IB so I can't imagine these code paths are even being called.

Modified file attached if anyone else is feeling brave...

[Jure Pečar]

On Thu, 18 Nov 2021 15:49:22 +0100 (CET)
André Gemünd <andre....@scai.fraunhofer.de> wrote:

> It should be very easy to patch, as the functions were just wrappers.

It's a bit more that that - there's also an incompatible pointer type to
figure out. Fresh docs say that device should be "struct device *", current
beegfs code has ctx->pd->device, which ends up as "struct ib_device *"
which is something from IB core, apparently.

Does anyone have a working patch for el8.5?


--

Jure Pečar

[Jure Pečar]

On Thu, 18 Nov 2021 15:49:22 +0100 (CET)
André Gemünd <andre....@scai.fraunhofer.de> wrote:

> It should be very easy to patch, as the functions were just wrappers.