Why is regular msync less optimal for pmem?

[KP]

Hello,

In pmem.c I saw the following comment in the code:

/*

* pmem_msync -- flush to persistence via msync

*

* Using msync() means this routine is less optimal for pmem (but it

* still works) but it also works for any memory mapped file, unlike

* pmem_persist() which is only safe where pmem_is_pmem() returns true.

*/

int

pmem_msync(const void *addr, size_t len)

{

For any regular memory mapped file, my understanding of mysnc() is that it does the following:

a) Read from DRAM, all the modified memory segments (pages) into the CPU cache

b) Flush the CPU cache, resulting in finally persistence to the I/O device.

In case of a Pmem memory map, I presume msync has been modified to have only the following:

a) Flush the CPU cache, resulting in persistence to Pmem device.

I understand that using msync requires aligning with page boundary and hence calls for syncing more that what is needed:

/* increase len by the amount we gain when we round addr down */

len += (uintptr_t)addr & (Pagesize - 1);

/* round addr down to page boundary */

uintptr_t uptr = (uintptr_t)addr & ~((uintptr_t)Pagesize - 1);

 

But even here only the modified cache lines (dirty) will be flushed by the CPU into the Pmem device right? So where exactly is the overhead of calling msync() directly for pmem?

Second question on the same topic: Let's assume I am using Pmem only as an extension of DRAM, ie., only for volatile purposes and no persistence. In this case, I would do the following steps in my program:

a) Memory map a file from a DAX mounted FS using Pmem device.

b) Write data into the memory map. This results in store instructions in the CPU caches.

c) Do not issue msync() at all.

Even If I do not issue mysnc() explicitly, eventually the CPU would flush the data into Pmem device based on it's cache eviction policies right? Is there any flaw in relying on this method instead of explicitly calling msync() whenever there is a write? (I am not interested in power failure scenarios here as I am using pmem as volatile memory).

Thank you!

[Andy Rudoff]

Hi,

There are three areas where the overhead of msync can make a difference:

1. Small flushes get rounded up to large flushes.  For example, imagine you are implementing a btree where you are updating pointers during inserts, rebalancing, etc.  Each time you want to ensure an update has been made persistent, you flush some small ranges, like a few cache lines.  Each of those flushes gets rounded up to a page since that's the smallest granularity msync can handle.  The impact is not just the extra flushes, but also the side effect of those flushes which invalidates those extra cache lines, forcing cache misses when they are next accessed.

2. No way to tell msync that non-temporal stores were used for parts of the range.  In PMDK, you'll see lots of places where we use non-temporal stores for performance, and those stores "go around" the CPU caches.  For the kind of fine-grained changes I mentioned above, PMDK knows that only normal stores need flushing, and any ranges where NT stores were used can be considered persistent.  Note that even without PMDK in the picture, libc's memcpy will sometimes use NT stores for large ranges.  You have no visibility into that, so you must call msync to make changes persistent, even when there's no flush required due to NT stores.

3. The overhead of kernel sys calls.  msync can take locks, which can end up serializing highly multithreaded code.  And the loop done by pmem_persist() will be a shorter overall code path.  This is even more true on future platforms where CPU caches can be considered persistent.  PMDK applications will detect these platforms and skip the flushes, but msync is still a trip through the kernel.

Of course, it depends on the use case.  For flushing large ranges where the kernel entry/exit is a very small portion of the code, and assuming no lock contention and no NT stores, pmem_persist and msync will perform about the same.  And also please remember that you can only use pmem_persist when pmem_is_pmem returns true, meaning the file system has agreed to make it safe by granting mmap with the MAP_SYNC flag.

On your second question, if you do not use msync at all and understand that a power failure or similar crash that can't flush everything will leave your data in an inconsistent state, then you could theoretically skip the msync calls, but I would recommend at least adding logic to your program to detect normal program shutdown and flush during it.  For example, when the program shuts down normally, call msync or pmem_persist on any pmem that is potentially dirty, and then set some sort of "clean flag" and flush it to indicate the consistent state to the program next time it starts up.

-andy

[KP]

Hi Andy!

Thanks much for the great explanation! Looking forward to learn more.

Thanks,

-Keshav