pmem_memcpy_persist and ordering

[brian.b...@hpe.com]

Consider the following scenario:

Thread A fills a buffer and sets a flag (whose initial value is 0):

pmem_memcpy_persist(pmembuf, ...);

flag = 1; 

While thread B waits for the flag and then takes a snapshot of pmembuf:

 while (1 != flag);

 memcpy(tmpbuf, pmembuf, ...);

When the buffer alignment and copy size allow pmem_memcpy_persist() to use only non-temporal instructions and if drain does not include a fence (i.e. CLFLUSH only), then, in our testing, thread B will sometimes see stale contents in pmembuf (that is, values held before thread A's memcpy).

An SFENCE in thread A after the copy but before the flag set seems to ensure that thread B sees the new values in pmembuf.

What is the expected/intended behavior of pmem_memcpy_persist() in this regard?  Should pmem_memcpy_persist() include an SFENCE or similar?

Thanks!

Brian

[Krzysztof Czurylo]

Hi Brian,

Indeed, it looks like if the platform does not support CLFLUSHOPT/PCOMMIT, then SFENCE is missing in pmem_mem*_persist() functions.

Here is a patch that should fix this issue:

https://github.com/pmem/nvml/commit/fba96d829e89bf26521d4cceac3e82c248cabadb

Please, verify and let me know if it helps.

Regards,

Krzysztof

[brian.boyls...@gtempaccount.com]

Hi Krzysztof,

Thanks for the patch!  I applied this to the 0.2 release of the NVML and it seems to work.

Thanks,

Brian