pmem_drain() behaviour in multi thread application

[lokesh arora]

Hello

I am using pmem_memcpy with NO_DRAIN flag in a multi threaded application.

Once all the data is written, I invoke pmem_drain() to persist the data.

Link mentioned that pmem_drain will "wait for any pmem stores to drain from HW buffers".

Since pmem_drain() does not accept any parameter, is there a common pool from which all the data is drained irrespective of which thread invoked pmem_memcpy at any time.? Also, how can I use it safely in a multi threaded application.

Thanks

Lokesh

[Andy Rudoff]

Hi Lokesh,

The idea of the "drain" operation is not range based, but instead designed to ensure any buffers that stand between your previous stores and the persistence domain are drained so you know the stores are persistent.  What pmem_drain() does on each architecture depends on how that architecture defines the persistence domain.  For Intel systems, once a store is accepted by the memory subsystem, it is considered persistent because the buffers in the memory subsystem are powerfail protected (this is a requirement for all Intel systems supporting pmem).  Therefor, on Intel systems, we only need to make sure previous stores have been accepted by the memory subsystem so pmem_drain() is nothing but an SFENCE instruction.  It is thread safe and there are no correctness issues with multiple threads calling pmem_drain(), but you are also free to copy data using multiple threads and then have only one final call to pmem_drain() when they are done.

-andy

[Jungsik Choi]

Hello Andy,

It's hard to understand the sentence below.

"but you are also free to copy data using multiple threads and then have only one final call to pmem_drain() when they are done."

As far as I know, SFENCE doesn't affect other cores but the one upon which it was issued. SFENCE flushes the local store buffer to the L1 cache. Is this correct? If so, I think all threads running on multi-core need to call pmem_drain() after writing data.

I am a little confused. Is it possible to guarantee that the entire thread's stores are complete even if pmem_drain() is called only once from one of the multi-threads?

Thanks,

Jungsik Choi

2021년 7월 9일 (금) 오후 11:25, Andy Rudoff <an...@rudoff.com>님이 작성:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/e0426d63-7cb3-4611-b138-3d981a7cfd8bn%40googlegroups.com.

[Andy Rudoff]

Yes, I worded that sentence badly.

Best thing to do is just have each thread call pmemobj_drain() as you suggest.  My point was you could have one thread kick off a bunch of other threads that move data, and when those threads are done, it can make the final pmemobj_drain() call.  But this only works because the primitives that allow you to coordinate threads end up serializing stores just like an SFENCE would.  I was just trying to say there are lots of ways it could work, but I probably just confused the issue for no benefit.  Having each thread do multiple "nodrain" copies, followed by a drain call when they are done, is the cleanest.

-andy