(R)DMA over fsdax/devdax

[XINDI ZUO]

Hi all! we want to implement DMA between GPU and PMEM (to be more specific, we want to enable GPU to directly access PMEM). We have tried to use CUDA API to pin the page of mmap-ed PMEM on fsdax so that we can utilize the CUDA Copy engine, but the results make me think this method doesn't help. There are some discussions suggesting using devdax instead of fsdax for RDMA, I am wondering what is the reason behind it.

If anyone ever has experience with (R)DMA over PMEM, could you share some useful articles/papers to read, or any insights about (R)DMA over PMEM? I would really appreciate any help or insight.

Thanks a lot in advance!

Cheers,
Xindi

[gro...@o2.pl]

Hi Xindi,

when it comes to general information about RDMA and PMEM, please take a look at:

 https://www.intel.com/content/www/us/en/developer/articles/technical/white-paper-remote-pmem-over-traditional-rdma.html

and 

https://www.intel.com/content/www/us/en/developer/articles/technical/persistent-memory-replication-over-traditional-rdma-part-1-understanding-remote-persistent.html

and

https://pmem.io/rpma/

For accessing fsdax, you need on-demand-paging (ODP) available in your system (OS) and HW (RNIC).

Please see the librpma library repository https://github.com/pmem/rpma/blob/d6db0342f4582035b0946060a268254538dd3e79/src/peer.c#L240 where you can see how the library handles ODP/fsdax.

Regards

Tomasz

[Igor]

Hi Xindi,

There has already been some research around leveraging persistent memory from a GPU. You may find some answers here: https://dl.acm.org/doi/10.1145/3503222.3507758 

Best regards,
Igor

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/da6f5247-0113-4006-8694-4da1f9894b2bn%40googlegroups.com.

[XINDI ZUO]

Hi Igor,

Thanks a lot for your comment! Actually we have studied this paper for a while. GPM achieves good performance by using the CUDA kernels, but we are afraid this method is not what we want. So we are looking for more work about PMEM DMA.

Anyway, thank you very much!

Cheers,
Xindi