Linux membarrier syscall

[Joseph Seigh]

Anyone know the difference between MEMBARRIER_CMD_PRIVATE_EXPEDITED and MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE for membarrier.

It would be nice if it actually does what I think it does.   I really don't want to have to read from procfs and do a lot of parsing and such.

Joe Seigh

[Dmitry Vyukov]

On Fri, Jun 9, 2023 at 8:00 AM Joseph Seigh <jseig...@gmail.com> wrote:
>
> Anyone know the difference between MEMBARRIER_CMD_PRIVATE_EXPEDITED and MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE for membarrier.
>
> It would be nice if it actually does what I think it does. I really don't want to have to read from procfs and do a lot of parsing and such.

Nice to see you here again, Joe

The docs are quite cryptic:
"all its running threads siblings have executed a core serializing instruction":
https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/membarrier.h#L155

In reality it does this:
https://elixir.bootlin.com/linux/latest/source/kernel/sched/membarrier.c#L322
https://elixir.bootlin.com/linux/latest/source/kernel/sched/membarrier.c#L184

which boils down to:
https://elixir.bootlin.com/linux/latest/source/arch/x86/include/asm/sync_core.h#L42

* This function forces the icache and prefetched instruction stream to
* catch up with reality in two very specific cases:
*
* a) Text was modified using one virtual address and is about to be executed
* from the same physical page at a different virtual address.
*
* b) Text was modified on a different CPU, may subsequently be
* executed on this CPU, and you want to make sure the new version
* gets executed. This generally means you're calling this in an IPI.


So it looks like it's intended for self-modifying code...

[Joseph Seigh]

Ok, thanks.   I'll examine it in a bit more detail, though at first glance it looks like they're using ipi to speed things up by not having to wait for slower occurring kernel events.

I'm doing some work on  a hazard pointer based proxy collector w/ memory barriers that I suggested ages ago.  I posted some smrproxy timing comparisons.  I was going to rework the atomic reference counted proxy collector in c11/c17 atomics as well but the approximated timings are so bad compared to smrproxy that I think I will pass on that.

Joe Seigh

[Dmitry Vyukov]

On Tue, Jun 13, 2023 at 9:07 PM Joseph Seigh <jseig...@gmail.com> wrote:
>
> Ok, thanks. I'll examine it in a bit more detail, though at first glance it looks like they're using ipi to speed things up by not having to wait for slower occurring kernel events.

Yes, my understanding that it always sends IPIs rather than waiting
for something passively. But it knows what CPUs are actually running
threads from the current process at the moment.

> I'm doing some work on a hazard pointer based proxy collector w/ memory barriers that I suggested ages ago. I posted some smrproxy timing comparisons. I was going to rework the atomic reference counted proxy collector in c11/c17 atomics as well but the approximated timings are so bad compared to smrproxy that I think I will pass on that.

Yes, I guess it's expected for an atomic RMW on a shared location.
But it may be interesting to benchmark with 10K threads instead of 10
(10K is real for our server scenarios). I assume any HP approach needs
to iterate over all threads, so 10K may penalize it.

Btw, have you looked at rseq? It's pretty interesting facility:
https://kib.kiev.ua/kib/rseq.pdf

We use it in tcmalloc for per-CPU caches:
https://google.github.io/tcmalloc/rseq.html

Say, it can make HP use per-CPU register instead per-thread and
membarrier support aborting all concurrent rseq sequences. This can
allow for even more interesting algorithms, e.g. registering a hazard
pointer even w/o a loop in a rseq section (effectively atomic
memory-to-memory move wrt to the current CPU).

> Joe Seigh
>
> On Tuesday, June 13, 2023 at 3:03:34 AM UTC-4 Dmitry Vyukov wrote:
>>
>> On Fri, Jun 9, 2023 at 8:00 AM Joseph Seigh <jseig...@gmail.com> wrote:
>> >
>> > Anyone know the difference between MEMBARRIER_CMD_PRIVATE_EXPEDITED and MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE for membarrier.
>> >
>> > It would be nice if it actually does what I think it does. I really don't want to have to read from procfs and do a lot of parsing and such.
>>
>> Nice to see you here again, Joe
>>
>> The docs are quite cryptic:
>> "all its running threads siblings have executed a core serializing instruction":
>> https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/membarrier.h#L155
>>
>> In reality it does this:
>> https://elixir.bootlin.com/linux/latest/source/kernel/sched/membarrier.c#L322
>> https://elixir.bootlin.com/linux/latest/source/kernel/sched/membarrier.c#L184
>>
>> which boils down to:
>> https://elixir.bootlin.com/linux/latest/source/arch/x86/include/asm/sync_core.h#L42
>>
>> * This function forces the icache and prefetched instruction stream to
>> * catch up with reality in two very specific cases:
>> *
>> * a) Text was modified using one virtual address and is about to be executed
>> * from the same physical page at a different virtual address.
>> *
>> * b) Text was modified on a different CPU, may subsequently be
>> * executed on this CPU, and you want to make sure the new version
>> * gets executed. This generally means you're calling this in an IPI.
>>
>>
>> So it looks like it's intended for self-modifying code...
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "Scalable Synchronization Algorithms" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lock-free+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/lock-free/02316b45-8d28-49f3-af60-c49ae27518ban%40googlegroups.com.



--
Dmitry Vyukov

All about lockfree/waitfree algorithms, multicore, scalability,
parallel computing and related topics:
http://www.1024cores.net

[Joseph Seigh]

rseq?   The concept seems vaguely familiar. :)

Might not get past POC w/ smrproxy.  The idea was to write portable code using c11/c17 atomics  but I don't thing that's possible given how borked its memory model semantics are.  And I mean even more borked than their discussions of what they think the memory model problems are.  And I really don't feel like writing non portable inline assembly just to ensure the code is doing what it should be doing.

Joe Seigh 

[Dmitry Vyukov]

On Fri, Jun 16, 2023 at 1:39 PM Joseph Seigh <jseig...@gmail.com> wrote:
>
> rseq? The concept seems vaguely familiar. :)

Yes, I meant mostly that it can now be played with easily and used in
production.

> Might not get past POC w/ smrproxy. The idea was to write portable code using c11/c17 atomics but I don't thing that's possible given how borked its memory model semantics are. And I mean even more borked than their discussions of what they think the memory model problems are. And I really don't feel like writing non portable inline assembly just to ensure the code is doing what it should be doing.
>
> Joe Seigh
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "Scalable Synchronization Algorithms" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lock-free+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/lock-free/ae8349fc-a322-421f-8acd-9a8c4aa3a32an%40googlegroups.com.

[Joseph Seigh]

Well, apparently MIcrosoft doesn't actually fully support C11/C17 yet.  Not entirely sure why they are bothering at all since by the time they get around to it, the world and everyone else will have moved way beyond that.

I'm tempted to make a list of "Things That Will Happen Before Microsoft Supports C11/C17".   Flying cars is definitely on that list.

Joe Seigh