Why "great care" for sync.atomic

[mikey]

I have some counters that are shared between goroutines and I'd like to
updated them safely. Rather than use mutexes, something like the
following seems preferable to me:

var cnt int32 // declared globally
...
// in some goroutine
atomic.AddInt32(&cnt, 1)
...
// in some other goroutine
fmt.Printf("cnt=%d\n", atomic.LoadInt32(&cnt))

However, the following warning appears in the sync.atomic documentation:

These functions require great care to be used correctly

But there is no further description of what care must be taken or why.
Can anyone help me to understand the purpose of this warning and whether
I have actually succeeded in taking "great care" in my example?

[Ian Lance Taylor]

The very short summary is that if you have to ask, you should probably
avoid the package. Or, read the atomic operations chapter of the
C++11 standard; if you understand how to use those operations safely
in C++, then you are more than capable of using Go's sync/atomic
package.

That said, sticking to atomic.AddInt32 and atomic.LoadInt32 is safe as
long as you are just reporting statistical information, and not
actually relying on the values carrying any meaning about the state of
the different goroutines.

Ian

[Konstantin Khomoutov]

IIUC, atomicity only guarantees memory and CPU cache coherence with
regard to the *integrity* of the value being manipulated -- that is, it
only guarantees that if one CPU is writing the value while another one
is reading it, the second one won't get a value which has some of it
bits from the "old" value stored at that memory location before the
first CPU started overwriting it, and other bits from the "new" value
being written. What atomicity does not guarantee, is any ordering of
observability of values. I mean, atomic.AddInt32() does only guarantee
that what this operation stores at &cnt will be exactly *cnt + 1 (with
the value of *cnt being what the CPU executing the active goroutine
fetched from memory when the operation started); it does not provide any
guarantee that if another goroutine will attempt to read this value at
the same time it will fetch that same value *cnt + 1.

On the other hand, mutexes and channels guarantee strict ordering
of accesses to values being shared/passed around (subject to the rules
of Go memory model).

[jonathan....@gmail.com]

Slightly off topic, but are bytes equivalent to atomics in terms of maintaining their integrity? If the cpu operates on whole bytes rather than bits, it seems like this should be the case, although I'm probably missing something important.

[Henrik Johansson]

How are mutexes for example different? There is no guarantee that subsequent reads see the same value even if it is protected by a mutex.

Maybe I am missing something but do you mean that atomic.AddInt32() does not create a real happens before relationship?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Øyvind Teig]

In the ninetees I worked with a Texas signal processor that would either use one cycle for a byte or four cycles (...?), depending on what kind of memory that was configured at the address. But both were bytes. I don't see why a byte is more atomic than anything else, since we really don't know how it's stored, packed or not? 

Øyvind

[Ian Lance Taylor]

On Fri, Mar 28, 2014 at 6:03 AM, <jonathan....@gmail.com> wrote:
>
> Slightly off topic, but are bytes equivalent to atomics in terms of
> maintaining their integrity? If the cpu operates on whole bytes rather than
> bits, it seems like this should be the case, although I'm probably missing
> something important.

I'm not aware of any processor where a byte will not maintain its
integrity.

Functions like atomic.AddInt32 do a little more than ensuring
integrity. On most processors adding a value to memory requires a
read-modify-write memory cycle (this is true even if there is a single
"add to memory" instruction as there on x86). On a multicore machine
if two cores execute the read-modify-write cycle simultaneously it is
possible for one of the additions to be lost. atomic.AddInt32
guarantees that no additions are lost.

Ian

[Ian Lance Taylor]

On Fri, Mar 28, 2014 at 6:17 AM, Henrik Johansson <dahan...@gmail.com> wrote:
>
> How are mutexes for example different? There is no guarantee that subsequent
> reads see the same value even if it is protected by a mutex.
> Maybe I am missing something but do you mean that atomic.AddInt32() does not
> create a real happens before relationship?

The behaviour of mutexes is guaranteed by the Go memory model
(http://golang.org/ref/mem). The behaviour of the sync/atomic
functions is not.

Perhaps the memory model should discuss the sync/atomic functions. A
downside is that that would just encourage people to use them, which
is almost certainly a bad idea. Another downside is that I don't
think it's possible to describe the sync/atomic functions in terms of
a happens-before relationship. I think you have to resort to odd
circumlocutions like "if a LoadInt32 happens to follow a StoreInt32 to
the same memory address, then anything that happens before the
StoreInt32 also happens before the LoadInt32." But there is no way to
say whether the LoadInt32 follows the StoreInt32 or not, and if it
does not come later then you know nothing about the state of memory.

Ian

[Konstantin Khomoutov]

On Fri, 28 Mar 2014 14:17:38 +0100
Henrik Johansson <dahan...@gmail.com> wrote:

> How are mutexes for example different? There is no guarantee that
> subsequent reads see the same value even if it is protected by a
> mutex.

But why? If I acquire a write lock on a mutex in one goroutine and
begin updating a value it protects; then at the same time I acquire a
read lock in another goroutine, the latter will be suspended until the
lock is released. When I'm finished with updating the value and
release the write lock, the suspended goroutine will be eventually
awakened and see the updated value. Of couse, this implies there's no
contention between the reading goroutine and other writing goroutines.

> Maybe I am missing something but do you mean that
> atomic.AddInt32() does not create a real happens before relationship?

That's my understanding, yes. Atomic operations are about CPU
instructions (loads, stores and read-modify-write sequences) while
the "happens before" thing is a high-level concept which does not
discuss CPUs, OS threads etc -- only the observed behaviour of a
program.
Please note that I'm not in the Go core team, and I may be wrong.

[Henrik Johansson]

I think I just assumed it worked like volatile or AtomicInt (or Long) for example in Java.

Basically: A write to a volatile field happens-before every subsequent read of that same field

I see how it can be relaxed to not include ordering but still there is something a bit weird about it.

I normally only use it for dynamic updating of configuration with the intent to ensure not violating the memory model...

I think mentioning it in the memory model spec is better than not. The doc can include the usual caveat about not sharing memory. 

  

[Konstantin Khomoutov]

On Fri, 28 Mar 2014 15:16:16 +0100
Henrik Johansson <dahan...@gmail.com> wrote:

> I think I just assumed it worked like volatile or AtomicInt (or Long)
> for example in Java.
>
> Basically: A write to a volatile field happens-before every
> subsequent read of that same field

I'm not a Java guy, but IIUC in C, declaring a variable as volatile
merely forces the compiler to remove any optimizations which would
otherwise prevent (or delay or reorder) actual writes in it and reads
from it. I don't think it provides any guarantees in the "happens
before" style if threads are present. What I'm leading to is that
different languages seems to consider different guarantees when it
comes to sharing memory. According to [1] it indeed appears that Java
has quite strong definition about how writes to a volatile variable are
synchronized across threads.

> I see how it can be relaxed to not include ordering but still there is
> something a bit weird about it.

You might find this [2] interesting (and other articles it refers to).

> I normally only use it for dynamic updating of configuration with the
> intent to ensure not violating the memory model...
> I think mentioning it in the memory model spec is better than not.
> The doc can include the usual caveat about not sharing memory.

1. http://stackoverflow.com/q/6259745/720999
2. http://preshing.com/20130618/atomic-vs-non-atomic-operations/

[Henrik Johansson]

Java has harder reqs for sure.

http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/package-summary.html#MemoryVisibility

I had forgotten all the lower level possibilities but for my thus far usages I think I am safe.

An entry (which may exist that I didn't read...) in either the memory model or somewhere else might be good.

The sync/atomic doc says "synchronization is better done with channels" is somewhat misleading given this knowledge.

Thanks guys! I (re)learned something today and that is always great!

[Øyvind Teig]

kl. 14:45:40 UTC+1 fredag 28. mars 2014 skrev Ian Lance Taylor følgende:

On Fri, Mar 28, 2014 at 6:03 AM,  <jonathan....@gmail.com> wrote:
>
> Slightly off topic, but are bytes equivalent to atomics in terms of
> maintaining their integrity? If the cpu operates on whole bytes rather than
> bits, it seems like this should be the case, although I'm probably missing
> something important.

I'm not aware of any processor where a byte will not maintain its
integrity.

From a paper of mine:

"The target machine is a TMS320C32 DSP where all (the supported) C's primitive data types are mapped to 32 bit read and writes".

http://www.teigfam.net/oyvind/pub/pub_details.html#AnotherSideOfSPoC

The programmer thinks that the byte is 8 bit but it's 32 bits. Now put an 8-bit memory on that machine (a bad idea).

Then connect two machines to that memory (an even badder idea). I fail to see how that "byte" is atomic. 

I don't buy that the programmer needs to "know" that that "'byte'" is 32 bits long, and therefore it is no "byte". 

A "byte" is what it look like in the code, not on the bus.

What did I miss?

Øyvind

[mikey]

On 28-03-14, 9:52 AM, Ian Lance Taylor wrote:
> On Fri, Mar 28, 2014 at 6:17 AM, Henrik Johansson <dahan...@gmail.com> wrote:
>>
>> How are mutexes for example different? There is no guarantee that subsequent
>> reads see the same value even if it is protected by a mutex.
>> Maybe I am missing something but do you mean that atomic.AddInt32() does not
>> create a real happens before relationship?
>
> The behaviour of mutexes is guaranteed by the Go memory model
> (http://golang.org/ref/mem). The behaviour of the sync/atomic
> functions is not.
>
> Perhaps the memory model should discuss the sync/atomic functions. A
> downside is that that would just encourage people to use them, which
> is almost certainly a bad idea. Another downside is that I don't
> think it's possible to describe the sync/atomic functions in terms of
> a happens-before relationship. I think you have to resort to odd
> circumlocutions like "if a LoadInt32 happens to follow a StoreInt32 to
> the same memory address, then anything that happens before the
> StoreInt32 also happens before the LoadInt32." But there is no way to
> say whether the LoadInt32 follows the StoreInt32 or not, and if it
> does not come later then you know nothing about the state of memory.

In my opinion, inadvertently encouraging their use by documenting them
is worthwhile because it strengthens developers' understanding of key
hardware/software concepts and gives the golang team a chance to
articulate the issues fully rather than letting developers guess their
way thru the issues.

[JT Olds]

Other gotchas with the atomic calls are:

 * the pointer you're updating has to be appropriately aligned in memory.

 * the word size of the architecture must equal or exceed the value you're trying to manipulate atomically. (can't do 64 bit operations on 32 bit systems)

[Ian Lance Taylor]

On Fri, Mar 28, 2014 at 9:27 AM, JT Olds <jto...@gmail.com> wrote:
>
> Other gotchas with the atomic calls are:
> * the pointer you're updating has to be appropriately aligned in memory.
> * the word size of the architecture must equal or exceed the value you're
> trying to manipulate atomically. (can't do 64 bit operations on 32 bit
> systems)

You actually can do 64-bit operations on most current 32-bit
processors, certainly including 32-bit x86 and 32-bit ARM. The code
for doing so can be seen in sync/atomic/asm_386.s and asm_arm.s.

Ian

[Ian Lance Taylor]

I would not recommend that developers guess. I would recommend that
they use channels and mutexes.

Ian

[mikey]

Point taken.

Based on the comments in this thread, it sort of seems like there are
two categories of atomics use:

- Simple/Safe: Add/Load counters for statistical use
- Great Care: Anything else

That info has been helpful to me as reading the C++11 information,
although it has been interesting, hasn't been particularly clear in that
regard.

[Øyvind Teig]

The doc in http://golang.org/pkg/sync/atomic/ as you say causes you to write this comment. 

Could you please elaborate?

[Henrik Johansson]

Well it implies atomics being on par with channels wrt synchronization which it is not it seems.

[Øyvind Teig]

Or more like the doc says don't eat apples, eat bananas - and this thread comparing apples with bananas? (Or did I foul that up completely?)

[Henrik Johansson]

Yeah well I get that that is the gist of it but:

"Except for special, low-level applications, synchronization is better done with channels or the facilities of the sync package."

It seems to say that _synchronization_ is possible. Of course it will require lots of hoops and bounds but if you read it casually and come from let's say Java it can be misread.

I would prefer the doc added something explicit about the functions not establishing a happens-before relationship by simply using them.

[Scott Pakin]

On Friday, March 28, 2014 9:07:11 AM UTC-6, Øyvind Teig wrote:

The programmer thinks that the byte is 8 bit but it's 32 bits. Now put an 8-bit memory on that machine (a bad idea).

Then connect two machines to that memory (an even badder idea). I fail to see how that "byte" is atomic. 

As another example, many RISC processors lacked a mechanism for single-byte accesses.  One would need to do a 32-bit load into a register, replace (via masking) the target byte, and store the 32-bit value back to memory.  Think of what might happen if thread A executed c[i]++ and thread B concurrently executed c[i+1]++.  Either of those increments might get lost.  Hence, just because two stores to the same byte are never going to produce a byte with interleaved bits, that doesn't mean that the programmer can be lax when updating bytes in multithreaded programs.

 

I don't buy that the programmer needs to "know" that that "'byte'" is 32 bits long, and therefore it is no "byte". 

A "byte" is what it look like in the code, not on the bus. 

In a few cases the programmer does need to know.  In grad school I programmed Myricom's LANai 2.3 network processor, which used 16-bit bytes, although this fact wasn't documented.  That violation of the basic assumption that bytes are 8 bits led to a hard-to-track bug in my code when the host CPU memory-mapped the LANai's memory and observed only every other octet containing valid data.

— Scott