Pinning PM for RDMA.

[Daniel Waddington]

Hi,
I'm trying to work out how PMDK ensures that pages are pinned for DMA in the RDMA libraries - can someone explain? Are they inherently pinned? I want to pin memory from PMEMpools for RDMA.  This is what I mean by pinning... https://lwn.net/Articles/600502/

Thanks
Daniel

[Daniel Waddington]

MADV_DONTFORK?

[Andy Rudoff]

Hi Daniel,

PMDK replies on libfrabric for RDMA, and that includes the memory registration that pins pages.  So you won't find any code related to pinning in PMDK.

Note that pinning filesystem DAX pages is not yet supported by Linux (the fix for this is expected within the next couple of kernel releases), so for RDMA with DAX you must use Device DAX instead of filesystem DAX for now.

-andy

[Daniel Waddington]

Thanks Andy.
Just one follow up question.  If I use device DAX and I am using object pools, how do I get the "whole" memory region from the PMEMobjpool (returned from pmem_objcreate) so that I can register it with ibverbs/libfabric? (the addr field seems to be hidden).  I guess alternatively I could register each "object" but want to avoid registering memory in the fast-path.

Daniel

[pawel.szymanski]

Hi Daniel,

If you use a poolset consisting of single part, PMEMobjpool returned from pmem_create points to the beginning of the entire pool including pool header.

If you use a poolset consisting of multiple parts, PMEMobjpool retuned from pmem_create points to beginning of the first part (including its header) and next follow subsequent parts without their headers.

A bigger challenge is how to learn the size of the area. Currently PMDK does not expose any API to learn the information. You can sum sizes of the DEV DAXes and subtract size of headers (except the first header).

Alternatively you can just use the fact that PMEMOBJ can directly talk to RPMEM library and run replication transparently for the application.

You just need to add information about remote replicat into the poolset file.

Regards

Pawel

[pawel.szymanski]

Note that your code needs to know the address of the entire poolset and its size only when the code will register the poolset in libfabric directly.

If you want to use our RPMEM library, there is no need to add any replication related code into your application. You just need to add remote replicas to poolset file and properly configure SSH on each machine to allow RPMEM to automatically start replication daemons on remote nodes.

RPMEM library will register the memory automatically without any action from the application.

Pawel  

[Daniel Waddington]

OK thanks, Pawel, I will look into this further and decide the best route for my needs.

Daniel

[Daniel Waddington]

Andy, have these kernel changes now been made?

Daniel

On Sunday, April 15, 2018 at 9:23:24 PM UTC-7, Andy Rudoff wrote:

[Andy Rudoff]

Hi Daniel,

I see we've been talking about "pinning" in two contexts on this list, so let me clarify the current Linux support.  Common DMA operations to memory also work as expected to persistent memory.  For example, if I open a traditional disk with O_DIRECT and then use the read() system call with pmem that was memory mapped from an FS DAX file as the buffer, the system will DMA directly to that pmem and the pages will be pinned during the operation so the file system won't try to relocate them during the DMA.

For RDMA, libraries like libfabric can issue "long-lived" memory registrations, where the pages are pinned and they stay that way between I/O operations and sometimes even across multiple processes.  Those RDMA long-lived registrations are currently disallowed on FS DAX.  The registration call returns an error, so there's no chance you'll think they are working and the file system moves a page during an I/O.  There are two workarounds for people wanting to do RDMA directly to pmem.  One is to use Device DAX instead of FS DAX, since it doesn't allow a file system so there's no issue granting the long-lived memory registrations.  Another is to use FS DAX with a NIC that supports On Demand Paging (ODP), which is a way to avoid the long-live memory registrations.  I'm only aware of one NIC (a Mellanox NIC) that supports ODP today, but the solution works well with FS DAX and I'm hoping to see it show up in other NICs in the future.

By the way, the work the PMDK team did around RDMA is document in four papers that describe what I said above and quite a bit more:

[1] https://software.intel.com/en-us/articles/persistent-memory-replication-over-traditional-rdma-part-1-understanding-remote-persistent

[2] https://software.intel.com/en-us/articles/persistent-memory-replication-over-traditional-rdma-part-2-remote-persistent-memory-101

[3] https://software.intel.com/articles/persistent-memory-replication-over-traditional-rdma-part-3-rdma-enabling-for-data

[4] https://software.intel.com/en-us/articles/persistent-memory-replication-over-traditional-rdma-part-4-persistent-memory-development

-andy