stdatomic.h, cgo, and sync/atomic

[Scott Cotton]

Hi Gophers,

I've got some questions about the interoperability guarantees of sync/atomic and cgo with stdatomic.h (C11 standard).

So, the general question is:  under what circumstances, if any, is using sync/atomic/CompareAndSwap

safe with concurrent C stdatomic.h calls "on the same memory location".  By "on the same memory location", I mean: the pointer representing the memory location is a "C" pointer (eg malloc'd), treated as a Go pointer in go by unsafe casting eg:

    C:  uint32_t v;

          volatile uint32_t *p = (volatile uint32_t*)&v;

     Go:  q := (*uint32)(unsafe.Pointer(C.p))

then some C thread calls intermittently 

      atomic_compare_exchange_strong(p, expected, desired)

and some goroutine calls intermittently

             atomic.CompareAndSwap(q, expected, desired)

Then is the atomicity of the compare and swap preserved with the same guarantees as if it were all done in one of the two languages (C and Go)?

I guess there's a "volatile" type qualifier with some questionable requirements in cgo environment:

"""

Any attempt to read or write to an object whose type is volatile-qualified through a non-volatile lvalue results in undefined behavior:

""" 

(from https://en.cppreference.com/w/c/language/volatile)

But I don't see how to preserve "volatile" in a cgo type referenced in Go code.  Not sure if that makes a difference provided the memory location is the same, i.e. p is the same value as q above, since both sides (C and Go) would presumably use the respective atomic apis correctly. 

Thoughts?

Scott

        

[Ian Lance Taylor]

First I'll note that `volatile` in C is not useful for communicating
between separate threads. I wrote it about at
https://www.airs.com/blog/archives/154 and others have written similar
things. When thinking about communicating between multiple threads,
don't use the `volatile` qualifier. It doesn't help.

That said, the exact semantics of the sync/atomic operations are not
written down (that is https://golang.org/issue/5045). But in general
you can treat Go's atomic.CompareAndSwapT(addr, old, new) as roughly
equivalent to C's __atomic_compare_exchange_n(addr, &old, new, false,
__ATOMIC_ACQ_REL, __ATOMIC_RELAXED). They should interoperate fine.

Ian

[Scott Cotton]

Thanks Ian,

Nice article on volatile, even though I wasn't at all thinking of promoting "volatile" as a way to communicate between threads ;)  or goroutines with cgo involved for that matter.  May make sense for some application contexts though, such as mmap'd hardware memory or when real-time constraints present risk of failure in light of Go<->C calling overhead and/or full locks/channels.  Definitely not part of the concurrency or threads 101 syllabus.

So to summarise, there's no official Golang statement supporting inter-op of C11 atomic_ and Go sync/atomic, but it should work fine, and Go's sync/atomic is roughly equivalent to atomic_compare_exchange_strong from C11 stdatomic.h which is most safely translated to GCC

__atomic_compare_exchange_n with the extra arguments you specify.    For whatever reason, 

the question of preservation of the volatile qualifier in the Go view of the volatile C value via cgo doesn't spark concern. 

Sounds ok to me so long as I'm not relying on it for an airplane controller or automobile collision avoidance or the like, and so long as I'm darn sure the Go view of the C volatile is "the correct one".

Thanks again

Scott

   

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

[Ian Lance Taylor]

On Wed, Aug 1, 2018 at 8:21 AM, Scott Cotton <w...@iri-labs.com> wrote:
> Thanks Ian,
>
> Nice article on volatile, even though I wasn't at all thinking of promoting
> "volatile" as a way to communicate between threads ;) or goroutines with
> cgo involved for that matter. May make sense for some application contexts
> though, such as mmap'd hardware memory or when real-time constraints present
> risk of failure in light of Go<->C calling overhead and/or full
> locks/channels. Definitely not part of the concurrency or threads 101
> syllabus.
>
> So to summarise, there's no official Golang statement supporting inter-op of
> C11 atomic_ and Go sync/atomic, but it should work fine, and Go's
> sync/atomic is roughly equivalent to atomic_compare_exchange_strong from C11
> stdatomic.h which is most safely translated to GCC
> __atomic_compare_exchange_n with the extra arguments you specify. For
> whatever reason,
> the question of preservation of the volatile qualifier in the Go view of the
> volatile C value via cgo doesn't spark concern.
>
> Sounds ok to me so long as I'm not relying on it for an airplane controller
> or automobile collision avoidance or the like, and so long as I'm darn sure
> the Go view of the C volatile is "the correct one".

More or less, but in C11 stdatomic terms atomic.CompareAndSwap is like
atomic_compare_exchange_weak_explicit(p, &old, new,
memory_order_acq_rel, memory_order_relaxed). It doesn't have the
guarantees of atomic_compare_exchange_strong and it doesn't guarantee
sequential consistency even when it succeeds. Care is required for
portable code as on x86 there is no difference between
compare_exchange_weak and compare_exchange_strong, and
compare/exchange always gives you sequential consistency, but these
statements are not true on some non-x86 platforms.

Ian

[Scott Cotton]

Thanks for clarifying 

Thanks for your clarification.

From the gcc docs I understand, on success, I do get sequential consitency guarantee using 

"""

atomic_compare_exchange_weak_explicit(p, &old, new,
memory_order_acq_rel, memory_order_relaxed)

"""

but only if at most "another" thread is involved,  no sequential guarantees if more than one other thread is doing the atomic CAS on the same address. 

(the docs read

"""

__ATOMIC_ACQ_REL

Full barrier in both directions and synchronizes with acquire loads and release stores in another thread. 

__ATOMIC_SEQ_CST

Full barrier in both directions and synchronizes with acquire loads and release stores in all threads.

"""

So now here's a question:  Suppose that there is exactly one goroutine calling atomic.CompareAndSwap

and exactly one C thread calling atomic_cas(...) on the same address. 

Can I assume sequential   

[Scott Cotton]

Apologies,  a delay in my mail client response to my input something made me press send prematurely.

Let me repeat the partial message and finish it correctly here.

More or less, but in C11 stdatomic terms atomic.CompareAndSwap is like
atomic_compare_exchange_weak_explicit(p, &old, new,
memory_order_acq_rel, memory_order_relaxed).  It doesn't have the
guarantees of atomic_compare_exchange_strong and it doesn't guarantee
sequential consistency even when it succeeds.  Care is required for
portable code as on x86 there is no difference between
compare_exchange_weak and compare_exchange_strong, and
compare/exchange always gives you sequential consistency, but these
statements are not true on some non-x86 platforms.

Thanks for your clarification.

From the gcc docs I understand, on success, I do get sequential consitency guarantee using 

"""

atomic_compare_exchange_weak_explicit(p, &old, new,
memory_order_acq_rel, memory_order_relaxed)

"""

but only if at most "another" thread is involved,  no sequential guarantees if more than one other thread is doing the atomic CAS on the same address. 

(the docs read

"""

__ATOMIC_ACQ_REL

Full barrier in both directions and synchronizes with acquire loads and release stores in another thread. 

__ATOMIC_SEQ_CST

Full barrier in both directions and synchronizes with acquire loads and release stores in all threads.

"""

So now here's a question:  Suppose that there is exactly one goroutine calling atomic.CompareAndSwap

and exactly one C thread calling atomic_cas(...) on the same address. 

Can I assume sequential  consistency (acquire and release) between the goroutine and the thread?  Or is the fact that the go scheduler may put the goroutine on different threads over time going to pose a risk, thus (cringe) inviting a fix with runtime.LockOSThread()?

Hmm this is getting complicated...

Scott

[Ian Lance Taylor]

On Wed, Aug 1, 2018 at 10:29 AM, Scott Cotton <w...@iri-labs.com> wrote:
> Apologies, a delay in my mail client response to my input something made me
> press send prematurely.
>
> Let me repeat the partial message and finish it correctly here.
>
>> More or less, but in C11 stdatomic terms atomic.CompareAndSwap is like
>> atomic_compare_exchange_weak_explicit(p, &old, new,
>> memory_order_acq_rel, memory_order_relaxed). It doesn't have the
>> guarantees of atomic_compare_exchange_strong and it doesn't guarantee
>> sequential consistency even when it succeeds. Care is required for
>> portable code as on x86 there is no difference between
>> compare_exchange_weak and compare_exchange_strong, and
>> compare/exchange always gives you sequential consistency, but these
>> statements are not true on some non-x86 platforms.
>
>
> Thanks for your clarification.
>
> From the gcc docs I understand, on success, I do get sequential consitency
> guarantee using
> """
> atomic_compare_exchange_weak_explicit(p, &old, new,
> memory_order_acq_rel, memory_order_relaxed)
> """
> but only if at most "another" thread is involved, no sequential guarantees
> if more than one other thread is doing the atomic CAS on the same address.
>
> (the docs read
> """

> __ATOMIC_ACQ_RELFull barrier in both directions and synchronizes with

> acquire loads and release stores in another thread.

> __ATOMIC_SEQ_CSTFull barrier in both directions and synchronizes with

> acquire loads and release stores in all threads."""
>
> So now here's a question: Suppose that there is exactly one goroutine
> calling atomic.CompareAndSwap
> and exactly one C thread calling atomic_cas(...) on the same address.
>
> Can I assume sequential consistency (acquire and release) between the
> goroutine and the thread? Or is the fact that the go scheduler may put the
> goroutine on different threads over time going to pose a risk, thus (cringe)
> inviting a fix with runtime.LockOSThread()?

I don't think the fact that goroutines can change threads is going to
matter here. When a goroutine changes threads the new thread by
definition sees all the memory writes done by the old thread.

acquire-release is not the same as sequential consistency but I think
in this case it doesn't matter.

Ian

>> __ATOMIC_ACQ_RELFull barrier in both directions and synchronizes with

>> acquire loads and release stores in another thread.

>> __ATOMIC_SEQ_CSTFull barrier in both directions and synchronizes with

[jake...@gmail.com]

On Wednesday, August 1, 2018 at 10:17:38 AM UTC-4, Ian Lance Taylor wrote:

[...]

That said, the exact semantics of the sync/atomic operations are not
written down (that is https://golang.org/issue/5045).

 

I really love go, but this is one thing that has annoyed me for years. This issue is from 2013, and AFAICT, there is still no clear statement of what guarantees the sync/atomic functions make. I know that they should be rarely used. I know that the edge cases for atomic operations are complicated to define, hence all the options for C++ atomics. But on the rare occasion that I do need to use them, I always feel a lot of trepidation, because I can really make few, if any guarantees based on the documentation.  

</rant>

[Ian Lance Taylor]

Yes, the problem is that nobody is entirely happy with the memory
model either. We need to figure out how to improve that first.

Ian