Questions about eADR, sfence and Intel TSX

[mingk...@gmail.com]

Hi everyone,

I have two questions after reading the article introducing eADR [1].

1) Since x86 is TSO, why are the SFENCE instructions required to guarantee the write ordering?

2) When Intel TSX (hardware transactional memory) is used on pmem,  uncommitted stores are not globally visible, but the data is in the L1 cache. Will these uncommitted stores be durable upon power-failure?

[1] https://software.intel.com/content/www/us/en/develop/articles/build-pmem-apps-with-ras.html

Thanks,

Mingkai

[Andy Rudoff]

Hi Mingkai,

1) Since x86 is TSO, why are the SFENCE instructions required to guarantee the write ordering?

The SFENCE isn't required for ordering of normal, write-back cached stores because they follow TSO just as you suggest.  But if you use any non-temporal stores (which have become quite popular in pmem programming), you will need the fence to ensure the stores are visible.  Even with normal stores, the executing thread doesn't know the stores are visible without using a fence, but as you point out it does know what order they will become visible and often that is all you need, so I agree there are times when you are not using NT stores that you can omit the fence.

2) When Intel TSX (hardware transactional memory) is used on pmem,  uncommitted stores are not globally visible, but the data is in the L1 cache. Will these uncommitted stores be durable upon power-failure?

If the TSX transaction has been completed (via XEND) before the power failure, then yes, they will be globally visible and eADR will flush them to persistence.  If the TSX transaction has not yet been committed, then no, those stores will not be globally visible and the eADR flow will not flush them.  Think of the eADR flow as doing a CLWB instruction for all globally-visible lines in the CPU cache.

Thanks,

 -andy

[mingk...@gmail.com]

Wow, that's to say, besides lock-free data structures, TSX-based concurrent data structures can also work on pmem without modification. That's amazing. Thank you!

- Mingkai

[Andy Rudoff]

Wow, that's to say, besides lock-free data structures, TSX-based concurrent data structures can also work on pmem without modification. That's amazing. Thank you!

Almost.  Remember that TSX can have XABORTs.  When you get an XABORT, you are expected to fall back to traditional mutex-based locking (or something else that isn't TSX).  If you have a TSX-based algorithm written for volatile memory, and you are in the middle of handling an XABORT, it doesn't really matter if you crash while holding a lock since the lock is in volatile memory.  However, if you move that algorithm to pmem, now you have to deal with the XABORT case in a way that's safe for pmem (i.e. the way libpemobj does, for example).

I think it would be more accurate to say that eADR makes TSX useful for pmem, but a volatile memory use of TSX won't just work without handling the pmem aspects of the XABORT case.

-andy

[mingk...@gmail.com]

Sure. The fallback path needs special attention on pmem.

Thanks for pointing that out! :)

- Mingkai