RDMA over PMEM

[S N]

HI All,

I have a system with PMEM configured in fsdax mode. I wanted to do RDMA to the PMEM data. I use libibverbs for this. So I did mmap() to the file stored in PMEM and tried to register the virtual address with ibv_mr_reg(). I could not register virtual address returned by mmap(). 

It fails with Error Code 95 - Operation not supported

The same is tested with DRAM as well. But with DRAM everything works!!

I have mlx_5 Inifiniband hardware. From my research I found that in fsdax one should enable the ON DEMAND PAGING in the NIC. This was mentioned in the discussion https://groups.google.com/g/pmem/c/Qpq2PmETsKs/m/CW5CSfNLBAAJ

so I added the additional flag of IBV_ACCESS_ON_DEMAND in ibv_reg_mr() function.

(Referred https://github.com/pmem/rpma/blob/66f112303d3e2b06d659c856b47debe3e4670952/src/peer.c#L201)

Unfortunately, this did not help. The Error has changed from Error Code 95 to

Error Code 14 - Bad Address

Has anyone got the same experience with PMEM?

Any help or pointers would be greatly appreciated.

Thanks in advance!

~Kannoth

[Dorau, Lukasz]

Hi,

 

Please build the librpma: https://github.com/pmem/rpma
and try to run all examples with your PMem device using the following command:

[rpma]$ ./examples/run-all-examples.sh ./build/examples/ /mnt/pmem/file

 

and please let me know if it works for you?

 

Łukasz

 

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/e8ecb847-111b-487d-b3a8-323dcb6278a0n%40googlegroups.com.

[Jeff Moyer]

S N <subash....@gmail.com> writes:

> HI All,
>
> I have a system with PMEM configured in fsdax mode. I wanted to do RDMA to
> the PMEM data. I use libibverbs for this. So I did mmap() to the file
> stored in PMEM and tried to register the virtual address with ibv_mr_reg().
> I could not register virtual address returned by mmap().
> It fails with Error Code 95 - Operation not supported

This is likely because you can't do RDMA to fsdax pages. Try using
device dax instead.

See this upstream commit:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2bb6d2837083de722bfdc369cb0d76ce188dd9b4

Cheers,
Jeff

[S N]

Hi Lukaz,

First of all, thanks for the response.

I ran the tests in the system  (pmdk installed) and all the tests are succeeded!! Attached the logs.

Now, what is missing libibverbs that is there in librpma in  memory region registration? 

Best Regards,

Subash

[S N]

Hi Jeff,

Thank you for the response.

 I just did the tests with librpma as suggested by Lukasz and the tests are passing.!  Now I wonder, why doesn't it works with ibverbs? 

Best Regards,

Subash

[Dorau, Lukasz]

Hi,

 

Librpma uses libibverbs (and librdmacm). Please compare and check what are you doing in a different way than librpma does? Or just share your code (at least the minimal reproducible example) with us.

 

Lukasz

 

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/d8338864-b46c-44bc-b460-d6e9c1d87082n%40googlegroups.com.

[Dorau, Lukasz]

Hi,

 

You can compare what flags are passed to the mmap() and ibv_reg_mr() functions by your application and what by the librpma library (you can do it using a debugger by setting a breakpoint on these functions).

 

Do you pass to the ibv_reg_mr() function exactly the same pointer which you got from the mmap() function?

 

Lukasz

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/MWHPR11MB18398073A6371F879227203896329%40MWHPR11MB1839.namprd11.prod.outlook.com.