rdma issue: ibv_reg_mr error -12 with pmem

[Bartosz Krakowiak]

Hi all,

I'm trying to use librpmem, librpma or directly use pmem with ibverbs, but ibv_reg_mr always fails.

My setup is two virtualbox VMs with fedora 33, rxe and kernel memmap option. I can use rdma with normal memory, but ibv_reg_mr always fails with -12 on pmem.

librpma example 03 with fsdax mode and 1M file on ext4 fs:
[vagrant@node2 03-read-to-persistent]$ ./server 192.168.33.11 2000 /pmem/file1
Aug 13 17:14:20.751250 [145661] *NOTE*  ep.c:  92: rpma_ep_listen: Waiting for incoming connection on 192.168.33.11:2000
Aug 13 17:14:20.751285 [145661] *ERROR* peer.c: 173: rpma_peer_mr_reg: ibv_reg_mr() failed: Invalid argument

With devdax mode:
[vagrant@node2 03-read-to-persistent]$ ./server 192.168.33.11 2000 /dev/dax0.0
Aug 13 17:18:14.091100 [145686] *NOTE* ep.c: 92: rpma_ep_listen: Waiting for incoming connection on 192.168.33.11:2000
Aug 13 17:18:14.091125 [145686] *ERROR* peer.c: 173: rpma_peer_mr_reg: ibv_reg_mr() failed: Invalid argument

dmesg output is same for both:
[384339.219335] rdma_rxe: err -12 from rxe_umem_get

I have the same error -12 from rxe_umem_get with hello example from librpmem.

I've tested it in both modes on servers with real pmem and Mellanox RNICs but it also fails at ibv_reg_mr.

Do you have any idea what could be wrong?

Thank you,

Bartosz

[Bartosz Krakowiak]

Problem resolved. It was caused by max locked memory limit, after ulimit -l unlimited I can run rpma examples with devdax.