DSA i/o at on pmem

[Mimo memo]

hi, pmem

I am digging into the Intel DSA(Date stream acceleration technology) it has several name like ioat userspace device in SPDK/DPDK and CB-DMA in nic. It should be a DMA engine for CPU for my understanding.

From this guide, it can copy data between DRAM and Pmem. https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator

mover capability to/from volatile memory, persistent memory, memory-mapped I/O, and through a Non-Transparent Bridge (NTB) device to/from remote volatile and persistent memory on another node in a cluster. 

Is there any guide or reference design for such usage?  my case is on devdax mode, copy from Dram to pmem, the i/o size could be 10kB or so in each.  is there any extra step for pmem, i.e to disable DDIO or extra persist step to write data to pmem in such usage. 

Thanks

[Andy Rudoff]

Hi,

I don't know of any specific examples yet, but you found the right page for information.  Notice the link to the DSA spec at the bottom of the page and notice the flags field in that spec that says you are moving data to "durable memory".  If you use that, that is similar to turning off DDIO.  Alternatively, you can follow a move operation with a flush command.  Configuring the flush command to behave like CLWB may provide better performance if the data will be accessed soon after being copied, since it can still be in the cache that way.  If the data won't be accessed for a while, using the DDIO-style move may be better since it avoids cache pollution.

Other than that, using DSA for DRAM<->pmem will behave just like using DSA DRAM<->DRAM, so once you have your devdax device mapped, you just use the addresses like you would any DRAM address.

Hope that helps.

-andy

[Mimo memo]

Thank you Andy. You information helps to go ahead.  My case is to replica data into pmem from Dram so a near-term reading is not applicable. A disable- DDIO style is good.  I googled  CLWB it seems to be eADR. This is a pretty feature. Data may still sit in the cache but make sure to persist in any case of hardware failure like power off.  If enabled this, it seems to make pmem to use as Dram without extra persist sync but keep data persistence in failures.  I am not sure if enabled eADR ,  persist step is needed in normal writing anymore.  turning DDIO off seems too expensive even for single pcie root port. Because nic device may sit with pmem on the same pcie root port. That's painful to balance both device.

Found more information in pmem programming guide. :)

data in the CPU caches must be flushed by the application using the CLWB, CLFLUSHOPT,
CLFLUSH, non-temporal stores, or WBINVD machine instructions.
Enhanced Asynchronous DRAM Refresh (eADR) requires that a non-maskable
interrupt (NMI) routine be called to flush the CPU caches before the ADR event can begin.
Applications running on an eADR platform do not need to perform flush operations
because the hardware should flush the data automatically, but they are still required
to perform an SFENCE operation to maintain write order correctness. Stores should be
considered persistent only when they are globally visible, which the SFENCE guarantees.