Question about SFENCE and eADR

[Jerrin Shaji George]

Hi,

I had a question about programming pmem applications for a platform with eADR support. I read on the Programming Persistent Memory book that applications running on an eADR platform do not need to do cache flushes, but still need to do an SFENCE to consider stores as being persistent (ref).

I also read that x86 platform makes data globally visible in program store order. So my question is, on an x86 platform with eADR support, is it necessary to issue an SFENCE? Rather can this SFENCE be replaced by, say, a compiler barrier to avoid compiler from reordering the pmem writes, and thus ensuring that writes to pmem follow program order?

Thanks,
Jerrin

[Andy Rudoff]

Hi Jerrin,

First, just a reminder that I'm not aware of any platforms available with eADR yet (it is possible for someone to build such a custom platform -- eADR just isn't supported by Intel yet).  That text about eADR in the book is forward looking, because we had to make sure the programming model and ACPI, etc. were prepared for it.  We recommend all persistent memory programs check to see if the CPU caches are considered persistent and dynamically use or skip the cache flush instructions.

Now on to your question.  The recommendation that you use an SFENCE with eADR is a fairly conservative way of looking at it, and I'm sure there are cases where you could get rid of the SFENCE.  It wasn't so much about ordering as knowing something has reached global visibility before continuing to run the program.  Typically there's some point in the program where it needs to assume some stores are persistent before continuing.  I call the operation that ensures the persistence a "store barrier".  Today, the store barrier for pmem is CLWB+SFENCE.  When we have eADR systems, of course you are right that the stores will become visible in program store order on x86, but if the program needs a store barrier, a point at which it can assume something is persistent, then it needs to know that the stores have reached global visibility.  That's what the SFENCE is for.  There are certainly lots of cases where you can look at an instruction sequence and decide the SFENCE isn't needed because you either don't need to know something is persistent yet or some other instruction you're doing will force the store to become globally visible without the fence.

Hope my explanation made some sense.

-andy

[Jerrin Shaji George]

Thanks Andy. That was helpful. I had a follow up question. In an eADR platform, are the per-core store buffers also included in the power fail protected domain?

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/bf05e6c5-5289-4ff6-b178-4453986dfc5b%40googlegroups.com.

[Andy Rudoff]

Hi Jerrin,

On a platform where the CPU caches are considered persistent, the stores still must be globally-visible (i.e. part of the CPU cache coherency domain) to be considered persistent.  Buffers that are not coherent, like the write-combining buffers used by NT stores, would not be specifically protected and software would have no expectation that those stores are persistent if it hasn't issued a fence yet.  Of course, those stores may or may not end up making it to persistence, since there's no guarantee they will be dropped either.

-andy

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+unsubscribe@googlegroups.com.