Annoying false reports from race detector

[Piotr Narewski]

Race detector is a very nice and useful tool, but sometimes it really pisses me off.

For example, I don't see any reason why it keeps reporting a data race on such a program:

http://play.golang.org/p/yiAa7XQ74U

Unless I don't know about something, bool read/write is an atomic operation - isn't it?

One routine is only reading a bool variable, while the other one is only writing - so what is the problem that it sees here?

Is there any solution to avoid reports about such things, other than wrapping the bool write/read with a useless mutex?

[Jesse McNelis]

On Mon, Jun 16, 2014 at 5:18 PM, Piotr Narewski <piot...@gmail.com> wrote:

> Unless I don't know about something, bool read/write is an atomic operation
> - isn't it?

It's not. Nothing is.
http://golang.org/ref/mem

> One routine is only reading a bool variable, while the other one is only
> writing - so what is the problem that it sees here?
> Is there any solution to avoid reports about such things, other than
> wrapping the bool write/read with a useless mutex?

Yep, you can use a http://golang.org/pkg/sync/#RWMutex
or you can use a channel.
http://play.golang.org/p/19jvP4RFDU





--
=====================
http://jessta.id.au

[Piotr Narewski]

On Monday, June 16, 2014 9:28:57 AM UTC+2, Jesse McNelis wrote:

> Unless I don't know about something, bool read/write is an atomic operation
> - isn't it?

It's not. Nothing is.
http://golang.org/ref/mem

Please explain me how is it possible for a bool read/write to not be an atomic operation?

On the low level: how any existing CPU in the world and read or write a bool value only partially?

Just the idea of that is ridiculous.

 

> One routine is only reading a bool variable, while the other one is only
> writing - so what is the problem that it sees here?
> Is there any solution to avoid reports about such things, other than
> wrapping the bool write/read with a useless mutex?

Yep, you can use a http://golang.org/pkg/sync/#RWMutex
or you can use a channel.
http://play.golang.org/p/19jvP4RFDU

Yes, I know a possible workarounds.

The simplest one would be like this: http://play.golang.org/p/-lh18C8q8u

What I don't understand though is why I need to use workarounds.

[Jon Valdés]

El lunes, 16 de junio de 2014 09:48:47 UTC+2, Piotr Narewski escribió:

On Monday, June 16, 2014 9:28:57 AM UTC+2, Jesse McNelis wrote:

> Unless I don't know about something, bool read/write is an atomic operation
> - isn't it?

It's not. Nothing is.
http://golang.org/ref/mem

Please explain me how is it possible for a bool read/write to not be an atomic operation?

On the low level: how any existing CPU in the world and read or write a bool value only partially?

Just the idea of that is ridiculous.

A bool can be changed in a CPU register but not be flushed to main RAM, where other thread might read it after the original write and find old data.

Bool writes are atomic in the sense that they can't be left half-done, but they can't guarantee sequential consistency, which is needed to avoid data races in many cases.

Kind regards,

   Jon Valdes

[Jesse McNelis]

On Mon, Jun 16, 2014 at 5:48 PM, Piotr Narewski <piot...@gmail.com> wrote:
> Please explain me how is it possible for a bool read/write to not be an
> atomic operation?
> On the low level: how any existing CPU in the world and read or write a bool
> value only partially?
> Just the idea of that is ridiculous.

The size of a bool isn't defined in the language spec.
It's size compared to the word size of any CPU is unknown.
But more importantly, If the two goroutines are running on threads on
different CPUs that don't share a cache then there is no reason for
one to ever see the mutations of the other without synchronisation.

[egon]

On Monday, 16 June 2014 10:48:47 UTC+3, Piotr Narewski wrote:

On Monday, June 16, 2014 9:28:57 AM UTC+2, Jesse McNelis wrote:

> Unless I don't know about something, bool read/write is an atomic operation
> - isn't it?

It's not. Nothing is.
http://golang.org/ref/mem

Please explain me how is it possible for a bool read/write to not be an atomic operation?

On the low level: how any existing CPU in the world and read or write a bool value only partially?

Just the idea of that is ridiculous.

Being atomic doesn't mean it's race free or safe.

https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong

+ egon

[Piotr Narewski]

On Monday, June 16, 2014 9:56:56 AM UTC+2, Jon Valdés wrote:

Please explain me how is it possible for a bool read/write to not be an atomic operation?

On the low level: how any existing CPU in the world and read or write a bool value only partially?

Just the idea of that is ridiculous.

A bool can be changed in a CPU register but not be flushed to main RAM, where other thread might read it after the original write and find old data.

Bool writes are atomic in the sense that they can't be left half-done, but they can't guarantee sequential consistency, which is needed to avoid data races in many cases.

Kind regards,

   Jon Valdes

OK - that makes sense, thank you for explaining.

But then, don't you think that we should have atomic.StoreBool, atomic.StoreByte and atomic.StoreUint16 ?

It's just does not make much sense that there are no such functions, forcing users to wrap single byte writes around complex solutions involving mutexes or channels.

[egon]

On Monday, 16 June 2014 11:04:03 UTC+3, Piotr Narewski wrote:

On Monday, June 16, 2014 9:56:56 AM UTC+2, Jon Valdés wrote:

Please explain me how is it possible for a bool read/write to not be an atomic operation?

On the low level: how any existing CPU in the world and read or write a bool value only partially?

Just the idea of that is ridiculous.

A bool can be changed in a CPU register but not be flushed to main RAM, where other thread might read it after the original write and find old data.

Bool writes are atomic in the sense that they can't be left half-done, but they can't guarantee sequential consistency, which is needed to avoid data races in many cases.

Kind regards,

   Jon Valdes

OK - that makes sense, thank you for explaining.

But then, don't you think that we should have atomic.StoreBool, atomic.StoreByte and atomic.StoreUint16 ?

It wouldn't work that easily... atomic reads/writes require alignment, which means it would be quite difficult to do in a case where:

type A struct {

    X, Y byte

}

If you can properly demonstrate the need for it, with overwhelming evidence, then it probably makes sense... otherwise not.

It's just does not make much sense that there are no such functions, forcing users to wrap single byte writes around complex solutions involving mutexes or channels.

Tell the full story; what are you implementing that needs this?

+ egon

[Piotr Narewski]

On Monday, June 16, 2014 9:58:29 AM UTC+2, Jesse McNelis wrote:

> Please explain me how is it possible for a bool read/write to not be an
> atomic operation?
> On the low level: how any existing CPU in the world and read or write a bool
> value only partially?
> Just the idea of that is ridiculous.

The size of a bool isn't defined in the language spec.  It's size compared to the word size of any CPU is unknown. 

What is there to specify?

Bool carries only one bit of information - it is pretty obvious that no sane person would make it bigger that a single CPU word.

 

But more importantly, If the two goroutines are running on threads on
different CPUs that don't share a cache then there is no reason for
one to ever see the mutations of the other without synchronisation.

But now you are suggesting that inside mutex.Unlock() there is a special code that flushes all the "cached" data of global variables into the memory.

Except that there isn't - I checked it.

The mutex operations are solely about synchronization of routines and they don't do anything about runtime memory management.

[Piotr Narewski]

If you can properly demonstrate the need for it, with overwhelming evidence, then it probably makes sense... otherwise not.

Overwhelming evidence? That's a good one :)

No sorry, I am not going to provide with an overwhelming evidence for the fact that applications where one routine is only there to read a bool value, while another one only writes to it are quite a common in the software development world.

 

It's just does not make much sense that there are no such functions, forcing users to wrap single byte writes around complex solutions involving mutexes or channels.

Tell the full story; what are you implementing that needs this?

I gave you a story with my example.

There is a routine that I want to stop at some point and stopping is by setting a bool variable is the most optimal way to do it.

It is not that it "needs this", but such is the most optimal way to do this specific thing.

Not having atomic.StoreBool, I am forced to replace bool with uint32 and (instead of doing "for !stop") I must be doing "for stop==0".

Which is a good workaround not affecting the performance, but at the same time a pretty stupid thing to do, considering that Go was supposed to be all about better code readability.

[egon]

On Monday, 16 June 2014 11:33:35 UTC+3, Piotr Narewski wrote:

If you can properly demonstrate the need for it, with overwhelming evidence, then it probably makes sense... otherwise not.

Overwhelming evidence? That's a good one :)

No sorry, I am not going to provide with an overwhelming evidence for the fact that applications where one routine is only there to read a bool value, while another one only writes to it are quite a common in the software development world.

What I mean, there should be something that shows, that after adding that functions things will get better... otherwise it's just an additional function that no one will use.

 

 

It's just does not make much sense that there are no such functions, forcing users to wrap single byte writes around complex solutions involving mutexes or channels.

Tell the full story; what are you implementing that needs this?

I gave you a story with my example.

There is a routine that I want to stop at some point and stopping is by setting a bool variable is the most optimal way to do it.

I don't know examples where such place would be performance critical...

One trivial way to accomplish this:

type Loop uint32 // wasn't able to think of a better name

func (v *Loop) Done() bool { return atomic.LoadUint32(v) == 1 }

func (v *Loop) Term()      { atomic.StoreUint32(v, 1) }

for !loop.Done() { ... }

There's also Tomb package http://godoc.org/gopkg.in/tomb.v1 that does something similar.

Depending on your problem there can be other solutions... which is why I'm asking for the specific problem you are having... not the abstract "stop something" problem.

+ egon

[Piotr Narewski]

On Monday, June 16, 2014 10:49:02 AM UTC+2, egon wrote:

I don't know examples where such place would be performance critical...

One trivial way to accomplish this:

type Loop uint32 // wasn't able to think of a better name

func (v *Loop) Done() bool { return atomic.LoadUint32(v) == 1 }

func (v *Loop) Term()      { atomic.StoreUint32(v, 1) }

for !loop.Done() { ... }

There's also Tomb package http://godoc.org/gopkg.in/tomb.v1 that does something similar.

Depending on your problem there can be other solutions... which is why I'm asking for the specific problem you are having... not the abstract "stop something" problem.

Oh please. Why won't you tell me to open a UDP socket and keep the thread running until I see a packet in it? :-)

Just because something is not "performance critical" does not mean that I am going to give up on optimizing it, for a sake of a broken race detector.

I checked it very carefully and for all I see all the bool writes ARE atomic operations - and they all happen instantly. 

And the race detector should be compatible with the compiler - but apparently it isn't and so I am going to insist that IMHO it is broken.

Though, feel free to provide me with an example code that would prove me wrong.

Simply show me a code which stores a global bool and it does not happen instantly - then we I can reconsider my conclusion of a broken race detector.

[Jan Mercl]

On Mon, Jun 16, 2014 at 9:18 AM, Piotr Narewski <piot...@gmail.com> wrote:
> Race detector is a very nice and useful tool, but sometimes it really pisses
> me off.
>
> For example, I don't see any reason why it keeps reporting a data race on
> such a program:
> http://play.golang.org/p/yiAa7XQ74U
>
> Unless I don't know about something, bool read/write is an atomic operation
> - isn't it?

The race (it _is_ a race) is not relevant to (CPU specific) atomic
bool read/write, because according to the memory model the compiler is
free to never write any value whatsoever to the stop variable. The
compiler may keep the value in a register only or even discard the
statement in its entirety.

> One routine is only reading a bool variable, while the other one is only
> writing - so what is the problem that it sees here?

See above.

> Is there any solution to avoid reports about such things, other than
> wrapping the bool write/read with a useless mutex?

It's not useless at all. It forces the write to stop to occur in the
first place.

-j

[Piotr Narewski]

On Monday, June 16, 2014 11:09:23 AM UTC+2, Jan Mercl wrote:

On Mon, Jun 16, 2014 at 9:18 AM, Piotr Narewski <piot...@gmail.com> wrote:
> Race detector is a very nice and useful tool, but sometimes it really pisses
> me off.
>
> For example, I don't see any reason why it keeps reporting a data race on
> such a program:
> http://play.golang.org/p/yiAa7XQ74U
>
> Unless I don't know about something, bool read/write is an atomic operation
> - isn't it?

The race (it _is_ a race) is not relevant to (CPU specific) atomic
bool read/write, because according to the memory model the compiler is
free to never write any value whatsoever to the stop variable. The
compiler may keep the value in a register only or even discard the
statement in its entirety.

So why this _race_ isn't reported by the race detector?

http://play.golang.org/p/-lh18C8q8u

At the low level it does exactly the same thing!

The only difference is the type of the variable in the Go source code.

[andrewc...@gmail.com]

There is an atomic store there. 

[Piotr Narewski]

Exactly.

So is atomic write relevant to something being a race, or isn't it?

[Jesse McNelis]

On Mon, Jun 16, 2014 at 7:13 PM, Piotr Narewski <piot...@gmail.com> wrote:

>> The race (it _is_ a race) is not relevant to (CPU specific) atomic
>> bool read/write, because according to the memory model the compiler is
>> free to never write any value whatsoever to the stop variable. The
>> compiler may keep the value in a register only or even discard the
>> statement in its entirety.
>
>
> So why this _race_ isn't reported by the race detector?
> http://play.golang.org/p/-lh18C8q8u
>
> At the low level it does exactly the same thing!
> The only difference is the type of the variable in the Go source code.

I think that's a bug/missing feature in the race detector.
An atomic.Store() without an atomic.Load() is a race.

[Jan Mercl]

On Mon, Jun 16, 2014 at 11:28 AM, Piotr Narewski <piot...@gmail.com> wrote:
> Exactly.
> So is atomic write relevant to something being a race, or isn't it?

Actually the atomic operations are not mentioned by the memory model - yet.

But their effect is what was discussed before - a write to the memory
"home" of a variable is forced and thus the mutation _may_ be observed
by other goroutines. AFAIK, again not yet guaranteed by the MM. As the
"memory writer" is free to postpone the actual write to a sync
event[0]; a "memory reader" is as free to use (register) cached values
up to yet another sync point occuring.

[0]: http://tip.golang.org/ref/mem#tmp_2

-j

[Jesse McNelis]

On Mon, Jun 16, 2014 at 7:34 PM, Jesse McNelis <jes...@jessta.id.au> wrote:
> I think that's a bug/missing feature in the race detector.
> An atomic.Store() without an atomic.Load() is a race.

yep, https://code.google.com/p/go/issues/detail?id=7621



--
=====================
http://jessta.id.au

[Piotr Narewski]

On Monday, June 16, 2014 11:37:48 AM UTC+2, Jesse McNelis wrote:

On Mon, Jun 16, 2014 at 7:34 PM, Jesse McNelis <jes...@jessta.id.au> wrote:
> I think that's a bug/missing feature in the race detector.
> An atomic.Store() without an atomic.Load() is a race.

yep, https://code.google.com/p/go/issues/detail?id=7621

All right.

Thanks for explained, gents - now it makes much more sense.

Still I'm missing my atomic.Bool operations, in such case.

[Matt Harden]

As has been mentioned, an atomic read/write can't be done to a variable smaller than a certain size. See http://stackoverflow.com/questions/14624776/can-a-bool-read-write-operation-be-not-atomic-on-x86. Use uint32 if you want to use sync/atomic, or go with one of the other methods. The most common one seems to be closing a channel to signal shutdown; this is nice because it plays well with other asynchronous events via the select statement.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Piotr Narewski]

On Monday, June 16, 2014 3:39:17 PM UTC+2, Matt Harden wrote:

As has been mentioned, an atomic read/write can't be done to a variable smaller than a certain size. See http://stackoverflow.com/questions/14624776/can-a-bool-read-write-operation-be-not-atomic-on-x86. Use uint32 if you want to use sync/atomic, or go with one of the other methods. The most common one seems to be closing a channel to signal shutdown; this is nice because it plays well with other asynchronous events via the select statement.

I read that, but I still cannot believe that you guys can do atomic additions to a 64 bit integer (even on a 32 bit platform), but you cannot do atomic read/write of a single byte.

Seriously? 

Anyway, turning my bool into a 32-bit int seems like a good solution to make the race detector quiet.

[Robert Johnstone]

It is a hardware limitation.  I'm certain the compiler writers can give a better answer, but I don't believe that you can do an atomic read/write to less than a memory word. 

[Gustavo Niemeyer]

The atomic package exists primarily to support the construction of
higher-level synchronization primitives, such as sync.Mutex,
sync.RWMutex, sync.Cond, etc. If you have a bool that you want to
synchronize across goroutines, use a sync.Mutex to protect it, or even
better: send it over a channel.

Quoting from the documentation of package atomic:

"These functions require great care to be used correctly. Except for
special, low-level applications, synchronization is better done with
channels or the facilities of the sync package. Share memory by
communicating; don't communicate by sharing memory."

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

gustavo @ http://niemeyer.net

[DV]

>I read that, but I still cannot believe that you guys can do atomic additions to a 64 bit integer (even on a 32 bit platform), but you cannot do atomic read/write of a single byte.

Last time I checked, this is "golang-nuts", not "x86-cpu-hardware-designers-nuts". These atomic operations work because the *hardware* guarantees that they work. You don't like the fact that CPU hardware designers don't provide atomic load/store "boolean" instructions, talk to them about it? It should provide for an interesting conversation. 

[Piotr Narewski]

On Monday, June 16, 2014 7:10:21 PM UTC+2, DV wrote:

>I read that, but I still cannot believe that you guys can do atomic additions to a 64 bit integer (even on a 32 bit platform), but you cannot do atomic read/write of a single byte.

Last time I checked, this is "golang-nuts", not "x86-cpu-hardware-designers-nuts".

That is exactly why I doubt you know what you are talking about.

I do not question that you know more about Go than me.

The thing is that this should be a reason for you to pay more respect towards people who maybe know more about how CPUs work.

Like me, for example :)

I told you already: show me an example code where a bool write would not be atomic operation - and then we can talk.

I hope you understand that the argument of "I am a go nut, so how the hell should I know?" only makes you look pathetic?

[Ian Lance Taylor]

On Mon, Jun 16, 2014 at 10:49 AM, Piotr Narewski <piot...@gmail.com> wrote:
>
> I told you already: show me an example code where a bool write would not be
> atomic operation - and then we can talk.

I think there are two different meanings of atomic floating around.

The meaning in the sync/atomic package is not just that the value is
changed as a unit. The store is also a release operation on other
memory operations. See the ARM implementations.

Ian

[Piotr Narewski]

I understand that there is some reasoning around it, but the claim that there are CPUs which cannot write a bool value in a thread-safe way is ridiculous, IMHO.

If it was true what you are saying, about the memory alignment and so, the attached program would be causing a panic immediately after you run it.

Or even if not immediately - then sooner or later... right? But it doesn't!

Though, feel free to try it on any platform you want and see if it panics there - let me know if it did. I don't think it will.

[Sanjay]

But now you are suggesting that inside mutex.Unlock() there is a special code that flushes all the "cached" data of global variables into the memory.
Except that there isn't - I checked it.
The mutex operations are solely about synchronization of routines and they don't do anything about runtime memory management.

This `code' is implicitly in the sync/atomic functions (which the Mutex.{Lock,Unlock} functions use). It's not really code in those functions, so much as optimization prevention hints to the compiler. The sync/atomic functions also emit assembly that instructs the hardware caches to also do something functionally equivalent to a cross-thread cache flush.

Cheers,

Sanjay

[andrewc...@gmail.com]

Memory ordering and variables that live in registers vs living in memory cause multiple cpu shared memory to more subtle than you realize.

You do realise C and Java have the volatile keyword for this reason (to ensure memory visibility across threads) and that Go does not have a volatile qualifier so cannot guarantee that every store causes a reload of the variable. Perhaps you think you are an expert on CPU's but perhaps not compilers.

Of course its possible for an atomic store to bool to be implemented in Go. It probably just hasn't been needed by the implementation so overlooked. 

[Piotr Narewski]

On Monday, June 16, 2014 11:21:42 PM UTC+2, andrewc...@gmail.com wrote:

Memory ordering and variables that live in registers vs living in memory cause multiple cpu shared memory to more subtle than you realize.

You do realise C and Java have the volatile keyword for this reason (to ensure memory visibility across threads) and that Go does not have a volatile qualifier so cannot guarantee that every store causes a reload of the variable. Perhaps you think you are an expert on CPU's but perhaps not compilers.

 

I know what volatile type is about, and to be honest I'm kind of disappointed that Go does not have such a property of variables, considering that it was created with multi-threading in mind.

I mean, channels are really cool, but often I just don't need such a fancy technology and a regular global variable is just enough to give me what I need. 

Maybe you guys like it, but I hate complicating things that are simple by nature.

And no, I would never call myself "an expert on CPU's", but I am already quite old and I have dealt with plenty of different CPUs in my life.  And one thing I am sure about all of them: they all allow you to update a byte in a memory in a thread-safe matter, even though they have to deal with word alignments while doing it. 

But when I read a patronizing comments of people pretending to be Go experts, explaining me how CPU's allegedly cannot do things that they were designed to do - then you make me thinking that maybe I actually am a CPU expert... :) But no, really I am not - for me it is just a basic knowledge that one-byte store instruction is always atomic, and even if it wasn't the compiler would have not dared to allocate two bool variables inside one memory word.

That a compiler can optimize access to global variables when they are not declared as "volatile" - I know it very well, it's my job to know it when I code C.

And that is why I try to understand what is the right way to implement a volatile bool in Go. I mean: a simpler way than wrapping a one bit of shared information with an advanced synchronization technology, just so I could change it from one thread and check its state in another. If I wanted my CPU to do a thousand things between a bool write and bool read, I would have written my code in bash. :)

[Jesse van den Kieboom]

Looks to me like people are just not agreeing on their definition of atomic. Writing a single byte value is atomic in the sense that you won't observe a half-written value from one thread while writing in another thread. However, it is not atomic in the sense that it's guaranteed that you will read the new value from another thread directly after it has been written.

[andrewc...@gmail.com]

Not to mention cpu caching and out of order cpu operations which probably needs some sort of flush to be visible across threads.

I think if atomic bool store etc are missing you could probably file an issue and then you would get an official response, there is a new release in the works.

The original post is a legitimate race condition though and i'm glad it was detected.

I also think convenience atomic bool operations may be useful, if not critical.

[Michael Jones]

Hi Pitor,

There is a surprising way to think about all of this that you may not have yet noticed about Go. Even though the authors of Go have quite a bit of experience (the UNIX OS, the C language, Plan 9, GCC internals, 100k CPU parallelism at Google, etc.) there are many "grown up ideas" missing from the language. Why is this? That's that subtle point. It is not because they don't know about them, or don't know how, or even don't miss them in some cases. 

My way of saying it is that where time and experience have shown that there are multiple equally workable ways of doing something, the simplest, or more accurately, the one most difficult to hurt yourself with, is the path that Go has followed. The reason is that most of "real" programming is about testing, debugging, extending, and maintaining. Making this easy is something just about every other language never considered. (Well, maybe COBOL, but in a very different way.) Go is simple, but no simpler than is necessary to build sophisticated and rich software systems at industrial scale.

Concurrency is probably the most significant case of this. It is almost impossible to imagine a complex concurrency scheme with asynchronism, allocation ownership floating between threads, and high performance being made to work "the first time" by most developers. On the other hand, Go's primitives make this "just work." (mostly) To achieve this the bulk of the normal ideas (semaphores, thread local storage, threads, atomics, mutexes, critical sections, and everything else) are removed and replaced with a) very general memory semantics (so fast on a range of HW now and future) and, b) the channel model for concurrent communication.

The existence of everything other than channels is minimized so that developers will think in terms of channels wherever that is adequate. This means that sometimes it is too heavyweight but if that is a 1% or 0.1% performance penalty overall, then that's deemed worth the first time perfect property. I would say it is well worth it. Recognizing that it may not suffice, the lower level primitives are exposed but since they are discouraged, they are only there to the extent that they must be.

You may or may not agree with this point of view, but if you consider it, it may help you see why some things that you are used to are missing by design, style, and desire.

Michael

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1 650-335-5765

[Gustavo Niemeyer]

On Mon, Jun 16, 2014 at 7:15 PM, Piotr Narewski <piot...@gmail.com> wrote:
> I know what volatile type is about, and to be honest I'm kind of
> disappointed that Go does not have such a property of variables,
> considering that it was created with multi-threading in mind.

Go was not designed to facilitate access to CPU features, nor with
multi-threading in mind, and volatile won't help you there even in C
[1], so that side of the conversation feels a bit like a dead-end.

Perhaps you can describe what you're trying to achieve at a higher
level, so that people could develop some empathy for your case and try
to help you there?


[1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2016.html


gustavo @ http://niemeyer.net

[andrewc...@gmail.com]

http://en.wikipedia.org/wiki/Memory_barrier
This is very informative, this concerning C especially which shows I was wrong previously:

"The keyword volatile does not guarantee a memory barrier to enforce cache-consistency. Therefore the use of "volatile" alone is not sufficient to use a variable for inter-thread communication on all systems and processors."

Java volatile is better than C volatile for sure, but with channels this stuff is all implicit.

Minus channels, a mutex or atomic operations are critical for inter thread communication not just because of atomicity, but also because of cross thread/cpu visibility. I think this thread highlights the importance of avoiding the lower level things when possible unless you understand the implications of doing so.

[Dmitry Vyukov]

On Mon, Jun 16, 2014 at 12:04 PM, Piotr Narewski <piot...@gmail.com> wrote:
>
>
> On Monday, June 16, 2014 9:56:56 AM UTC+2, Jon Valdés wrote:
>>>
>>> Please explain me how is it possible for a bool read/write to not be an
>>> atomic operation?
>>> On the low level: how any existing CPU in the world and read or write a
>>> bool value only partially?
>>> Just the idea of that is ridiculous.
>>
>>
>> A bool can be changed in a CPU register but not be flushed to main RAM,
>> where other thread might read it after the original write and find old data.
>> Bool writes are atomic in the sense that they can't be left half-done, but
>> they can't guarantee sequential consistency, which is needed to avoid data
>> races in many cases.
>>
>> Kind regards,
>> Jon Valdes
>
>
> OK - that makes sense, thank you for explaining.
>
> But then, don't you think that we should have atomic.StoreBool,
> atomic.StoreByte and atomic.StoreUint16 ?
>
> It's just does not make much sense that there are no such functions, forcing
> users to wrap single byte writes around complex solutions involving mutexes
> or channels.


This is a true report.
You are not forced to use channels/mutexes, use atomic.Store/LoadInt32.

[Piotr Narewski]

Gentlemen, you are debating a topic that you seem to know only from Wikipedia with a guy who has spent half of his life on programming multi-thread software in C.

I don't know nor care about Java, but I can tell you for sure that volatile keyword in C is there only to disable compiler's optimization when dealing with the specific variable.

If you order the compiler to disable optimizations globally (-O0), it should generate you exactly the same code, with or without volatile keywords in front of your variables.

And if Wikipedia tells you something else, then Wikipedia is wrong.

[Piotr Narewski]

Hi Michael,

There is nothing that I would disagree with, in your point of view, but now let me to express mine on the related matter.

Non-experienced programmers perceive their code through a compiler's language. They do not think in the matter of CPU instructions, memory access, stacks, task rescheduling and so on.

As you get older, more experienced, you start perceiving your code like you were the computer which gets to execute the final code. And then the compiler is there mainly to translate your idea into the machine's code. Writing a source code, you imagine the final code already and you make it in the way that would be the most optimal for your target platform. That's natural and that is what makes you a good developer after all.

Go is a suitable compiler for beginners. Things like channels or the hermetic memory model - they help beginners a lot to avoid making mistakes.

But Go is also for experience programmers. That is where you start using things like cgo, unsafe, reflect, atomic, etc.

So far, the authors of this fine compiler have done a great job providing beginners with safety, though still allowing experts to do whatever they want.

And I don't see a reason why it could not be kept like this.

[egon]

On Tuesday, 17 June 2014 09:25:35 UTC+3, Piotr Narewski wrote:

Gentlemen, you are debating a topic that you seem to know only from Wikipedia with a guy who has spent half of his life on programming multi-thread software in C.

Please re-read who wrote this article: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2016.html

 

Alternatively you can also read this: http://www.airs.com/blog/archives/154

Or a quote from Walter Bright: ``You have to give up on volatile. Nobody agrees on what it means. What does "don't optimize" mean? And that's not at all the same thing as "atomic".``

+ egon

[Ibrahim M. Ghazal]

On 17 Jun 2014 09:25, "Piotr Narewski" <piot...@gmail.com> wrote:
>
> Gentlemen, you are debating a topic that you seem to know only from Wikipedia with a guy who has spent half of his life on programming multi-thread software in C.

I'm flabbergasted. How can someone with such experience as yours have such utter lack of knowledge about how *modern* CPUs and compilers work? (As evidenced by all your posts to this thread)

It seems to me that your knowledge is based on empirical and anecdotal evidence not backed by actual facts. People have tried to explain to you why you are wrong multiple times on this thread, and you completely ignore them and continue stating your overly-simplistic view of how things work or how they should work.

I suggest reading the information you are given before lecturing people about *their* knowledge. Starting with this excellent article which was already posted on this thread which you don't seem to have read:

https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong

[andrewc...@gmail.com]

Piotr, I don't doubt that you know what a volatile variable is, I was just trying to help explain what sort of havoc cpu caches can cause when you don't have memory barrier instructions when communicating between threads. This opens up intermittent failures in your code depending on which thread the goroutines are scheduled to.

Its fine to ignore the race detector if you really think there is nothing wrong, it's won't cost me any time or money, but may cost you in the future.

There is not much point in discussing further as I hope we can agree that the builtin atomic load and store operations are what is needed in this case.

[Piotr Narewski]

Or a quote from Walter Bright: ``You have to give up on volatile. Nobody agrees on what it means. What does "don't optimize" mean? And that's not at all the same thing as "atomic".``

To understand what "don't optimize" means, I would first need to learn about what a C compiler is.

Compiler is a tool that translates a source code into a machine code.

Each thing you do in a C source code comes down to a set of primitives, which later get translated into a predefined set of a specific CPU instructions.

When you read a variable - there will be a LOAD instruction.

When you write a variable - there surely is will be a STORE.

Now, as the compilers were advancing, they started to combine the primitives together, allowing to optimize the final code for size or speed.

So "don't optimize"  basically means that you don't combine the primitives and generate the assembly code exactly as then C code tells you to do.

[andrewc...@gmail.com]

That blog post by Dimitry is excellent. I find it funny how one of the examples of code that is broken is exactly what is in the original post.

[Piotr Narewski]

On Tuesday, June 17, 2014 9:10:59 AM UTC+2, Ibrahim M. Ghazal wrote:

It seems to me that your knowledge is based on empirical and anecdotal evidence not backed by actual facts. People have tried to explain to you why you are wrong multiple times on this thread, and you completely ignore them and continue stating your overly-simplistic view of how things work or how they should work.

Which "actual facts"?

If you want to teach people about "facts", you ought to bring in some facts into the discussion.

I believe I have already addressed the alleged facts that CPUs are unable to safely store a byte, because of a memory alignment.

And it was rather me who has proven that "fact" wrong.

[Andrew Gerrand]

On 17 June 2014 16:25, Piotr Narewski <piot...@gmail.com> wrote:

Gentlemen, you are debating a topic that you seem to know only from Wikipedia with a guy who has spent half of his life on programming multi-thread software in C.

I'd like to point out that some of the people commenting in this thread are world experts on compilers and multicore programming.

For a guy that has spent "half his life programming multi-thread software in C" you do seem to have some strange notions that seem incorrect to me.

Since I started working on Go I have learned a lot about modern CPUs that surprised me. They are **very** different from the CPUs I grew up programming against. Maybe this is the case for you, too.

If you want an atomic bool, just use atomic.(Load|Store)Int32. That's what I have done in the past. It's very fast and easy.

If you're not convinced, I challenge you to demonstrate a performance difference between atomically loading/storing an word-sized value and a smaller value. I would suspect that working atomically with smaller values will actually be slower, as you (or the CPU) would need to take care to preserve the adjacent values in the word.

(Your aligns.go program "proves" nothing, by the way. It is possible to write all kinds of programs that behave correctly today—with the current compiler and the current language on your current system—but may be totally incorrect tomorrow. The advice you are receiving in this thread (and from the race detector!) is given to try to help you write robust code that will be always correct.)

Andrew

[egon]

On Tuesday, 17 June 2014 10:15:20 UTC+3, Piotr Narewski wrote:

Or a quote from Walter Bright: ``You have to give up on volatile. Nobody agrees on what it means. What does "don't optimize" mean? And that's not at all the same thing as "atomic".``

To understand what "don't optimize" means, I would first need to learn about what a C compiler is.

You do know who Walter Bright is?

Some more: https://www.kernel.org/doc/Documentation/volatile-considered-harmful.txt

Also, from Scott Meyers and Andrei Alexandrescu: http://www.aristeia.com/Papers/DDJ_Jul_Aug_2004_revised.pdf

First, the Standard’s constraints on (volatile) observable behavior are only for an abstract machine defined by the Standard, and that abstract machine has no notion of multiple threads of execution. As a result, though the Standard prevents compilers from reordering reads and writes to volatile data within a thread, it imposes no constraints at all on such reorderings across threads. At least that’s how most compiler implementers interpret things. As a result, in practice, many compilers may generate thread-unsafe code from the source above. If your multi-threaded code works properly with volatile and doesn’t work without, then either your C++ implementation carefully implemented volatile to work with threads (less likely), or you simply got lucky (more likely). Either case, your code is not portable. 

+ egon

[andrewc...@gmail.com]

Please stop trolling and read the post by the author of the race detector, which explains exactly why your code does indeed have a data race.
https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong

These are the facts people are referring to.

[Ibrahim M. Ghazal]

That's exactly my point. I said that *you* do not provide actual facts. When you "proved" that here:

https://groups.google.com/d/msg/golang-nuts/dy4aImQAPvk/N2DEZQY3EQIJ

You provided empirical and anecdotal evidence, not facts. That might work for the majority of cases, but it will break in subtle ways under different circumstances.

[Piotr Narewski]

On Tuesday, June 17, 2014 9:37:11 AM UTC+2, Andrew Gerrand wrote:

On 17 June 2014 16:25, Piotr Narewski <piot...@gmail.com> wrote:

Gentlemen, you are debating a topic that you seem to know only from Wikipedia with a guy who has spent half of his life on programming multi-thread software in C.

I'd like to point out that some of the people commenting in this thread are world experts on compilers and multicore programming.

This comment wasn't addressed to such people, since the world expert would not need me to explain him what the volatile keyword means for a C compiler.

For a guy that has spent "half his life programming multi-thread software in C" you do seem to have some strange notions that seem incorrect to me.

Since I started working on Go I have learned a lot about modern CPUs that surprised me. They are **very** different from the CPUs I grew up programming against. Maybe this is the case for you, too.

If you want an atomic bool, just use atomic.(Load|Store)Int32. That's what I have done in the past. It's very fast and easy.

You are not adding anything to the topic with this.

That was already said like three times already - and it was myself who first proposed this workaround.

And I keep insisting that it is a workaround, for a lack of atomic bool operations in the package.

 

(Your aligns.go program "proves" nothing, by the way. It is possible to write all kinds of programs that behave correctly today—with the current compiler and the current language on your current system—but may be totally incorrect tomorrow. The advice you are receiving in this thread (and from the race detector!) is given to try to help you write robust code that will be always correct.)

Except that storing a single byte within a wider CPU word has nothing to do with a version of a compiler.

Either CPU can or cannot update a byte within a word in a thread-safe manner.

There is no middle ground and compilers have nothing to do with it.

If you don't believe me, feel free to write the same kind of test in assembler.

It does't take an expert, just a bit of imagination, to know that if CPUs/compilers had not supported such kind of operations, then no traditional software would have been working properly on them.

Such a programming platform would be a developer's hell.

[andrewc...@gmail.com]

The point of argument isn't about a bool storing atomically (it probably does), its about data visibility between threads. By the way, you haven't explained anything to me about what volatile means. You have reiterated what is a common misconception (which I previously held, about the C version of the volatile keyword.), proving you didn't read or understand anything posted AT ALL.

[Piotr Narewski]

On Tuesday, June 17, 2014 10:02:20 AM UTC+2, andrewc...@gmail.com wrote:

The point of argument isn't about a bool storing atomically (it probably does), its about data visibility between threads. By the way, you haven't explained anything to me about what volatile means. You have reiterated what is a common misconception (which I previously held, about the C version of the volatile keyword.), proving you didn't read or understand anything posted AT ALL.

I thought I already did explain you what volatile means in C.

It mans that C compiler does not optimize access to/form the certain variable; whenever there is a reference to a volatile variable in the code, you have it guaranteed that it happens exactly in the place where you wanted.

Sorry, I just cannot phase it any simpler.

Though, I can give you an example - maybe that will stimulate your imagination.

You obviously like studying blog articles, so let me write a short one for you. :)

Let's say we have a code like this:

int a;

void f() {

  a=0;

  a++;

       a+=2; 

}

 

The above code will likely cause a compiler to just store 3 in a. But if you declare a as volatile, then it will first store 0 there, then it will increase it by 1, and then by another 2 - you don't even have it guaranteed to be 3 at the end, since another thread may modify the value in between and before each addition a gets re-fetched from memory. Because the access to a is not optimized, after you defined it as volatile.

Obviously having it volatile it does not mean that operations on a are atomic, because atomic operations is a CPU concept, which has nothing to do with a C compiler.

Saying otherwise: having a volatile a hare in no way assures you that this code would be thread-safe, since the volatility of variables has nothing to do with accessing them in an atomic way. Which is exactly the point your mentor Walter Bright made in the text that you referred me to.

[andrewc...@gmail.com]

What you are saying is all true, and yet you still don't understand that people are talking about something completely different. Dimitry's blog post is talking about compiler assumptions about when variables are live, and the other post was about the lack of memory barrier semantics with the C volatile keyword.

You seriously need to check your ego. I wasn't asking you to explain it, I was saying haven't explained it to me, because I already knew what it does precisely

[roger peppe]

The above operations are all irrelevant to what you're trying
to do with your original code. You were using a variable to communicate
information between goroutines.

The issue being that there is not just one "a" - there are several,
stored in different parts of the processor at different cache levels.
As many people have now pointed out, even if we use a
machine instruction that operates on a single byte, that does not
guarantee that the value that you have now written is flushed to the
other copies of "a" that other goroutines might be reading.

The reason we have channels and explicitly atomic operations
is precisely so that the compiler knows when to flush those caches,
so that *all* goroutines will see the value of "a" that you have
just written.

[Piotr Narewski]

On Tuesday, June 17, 2014 10:49:28 AM UTC+2, rog wrote:

As many people have now pointed out, even if we use a
machine instruction that operates on a single byte, that does not
guarantee that the value that you have now written is flushed to the
other copies of "a" that other goroutines might be reading.

Yes, I know.

That was already said to me yesterday, by Jan and Jessie.

I am not disagreeing with it and I really don't understand why you keep repeating the same thing over and over again.

[andrewc...@gmail.com]

Because you made a lot of technical errors then insulted people who tried to help you.

[Andrew Gerrand]

On 17 June 2014 18:53, Piotr Narewski <piot...@gmail.com> wrote:

I am not disagreeing with it and I really don't understand why you keep repeating the same thing over and over again.

Because you later said "...for me it is just a basic knowledge that one-byte store instruction is always atomic," and "...show me an example code where a bool write would not be atomic operation." With which you now apparently disagree.

On top of that you've been pretty rude and insulting. It is you who started questioning other people's experience and boasted of your own. If you have a sound argument then such claims are totally unnecessary. Please just stick to the pertinent facts next time, please.

Andrew

[Piotr Narewski]

On Tuesday, June 17, 2014 11:03:59 AM UTC+2, Andrew Gerrand wrote:

On 17 June 2014 18:53, Piotr Narewski <piot...@gmail.com> wrote:

I am not disagreeing with it and I really don't understand why you keep repeating the same thing over and over again.

Because you later said "...for me it is just a basic knowledge that one-byte store instruction is always atomic," and "...show me an example code where a bool write would not be atomic operation." With which you now apparently disagree.

No - I still agree with that.

One-byte store instruction is always atomic - that is all true and I have never disagreed with it.

 

On top of that you've been pretty rude and insulting. It is you who started questioning other people's experience and boasted of your own. If you have a sound argument then such claims are totally unnecessary. Please just stick to the pertinent facts next time, please.

Give me a break :)

Half of the people in this topic have been pretty rude and insulting to me and it definitely wasn't me who started.

In fact I've been pretty lenient and tolerant, considering how hostile you guys had been towards me.

[Konstantin Kulikov]

I am not disagreeing with it

Which means topic is done and you can finally correctly rewrite your original loop as

select {

case <-quit:

default:

}

without discussing operations atomicity

[Gustavo Niemeyer]

On Tue, Jun 17, 2014 at 4:56 AM, Piotr Narewski <piot...@gmail.com> wrote:
> This comment wasn't addressed to such people, since the world expert would
> not need me to explain him what the volatile keyword means for a C compiler.

It was addressed to the people that politely informed you that you
were reinforcing a common misconception about the volatile keyword,
associating it with multi-threading, and also suggesting it should be
adopted by Go despite its miserable history in C and C++:

On Mon, Jun 16, 2014 at 7:15 PM, Piotr Narewski <piot...@gmail.com> wrote:
> I know what volatile type is about, and to be honest I'm kind of
> disappointed that Go does not have such a property of variables,
> considering that it was created with multi-threading in mind.

As I said before, this conversation won't take you anywhere nice.

If you do want to solve your problem, there are people here that would
help you with accomplishing your high-level goals, either by
explaining good alternatives or by adapting the compiler or standard
library to accommodate reasonable needs that were not anticipated. The
way to approach that, though, is by taking people for a walk and
having a pleasant chat about the real problem you're trying to solve,
politely explaining why the proposed solutions do not work. Assume
they actually know what they're talking about, too, and focus on
learning from them when possible. It's fine and much more fun to be
experienced and still be learning.


gustavo @ http://niemeyer.net

[egon]

On Tuesday, 17 June 2014 11:27:57 UTC+3, Piotr Narewski wrote:

On Tuesday, June 17, 2014 10:02:20 AM UTC+2, andrewc...@gmail.com wrote:

The point of argument isn't about a bool storing atomically (it probably does), its about data visibility between threads. By the way, you haven't explained anything to me about what volatile means. You have reiterated what is a common misconception (which I previously held, about the C version of the volatile keyword.), proving you didn't read or understand anything posted AT ALL.

I thought I already did explain you what volatile means in C.

It mans that C compiler does not optimize access to/form the certain variable; whenever there is a reference to a volatile variable in the code, you have it guaranteed that it happens exactly in the place where you wanted.

Sorry, I just cannot phase it any simpler.

Though, I can give you an example - maybe that will stimulate your imagination.

You obviously like studying blog articles, so let me write a short one for you. :)

Let's say we have a code like this:

int a;

void f() {

  a=0;

  a++;

       a+=2; 

}

 

The above code will likely cause a compiler to just store 3 in a. But if you declare a as volatile, then it will first store 0 there, then it will increase it by 1, and then by another 2 - you don't even have it guaranteed to be 3 at the end, since another thread may modify the value in between and before each addition a gets re-fetched from memory. Because the access to a is not optimized, after you defined it as volatile.

Yes the order will be preserved, but it doesn't mean the same as "not optimized".

As an example using clang 3.4.1

volatile int a;

void f(){ a += 1; }

With -O3 it generates:

f():                                  # @f()

incl a(%rip)

ret

With -O0 it generates:

f():                                  # @f()

pushq %rbp

movq %rsp, %rbp

movl a, %eax

addl $1, %eax

movl %eax, a

popq %rbp

ret

+ egon

[Piotr Narewski]

On Tuesday, June 17, 2014 11:17:32 AM UTC+2, Konstantin Kulikov wrote:

I am not disagreeing with it

Which means topic is done and you can finally correctly rewrite your original loop as

select {

case <-quit:

default:

}

without discussing operations atomicity

Neee.

I just turned my bool into uint32, and use atomic's LoadUint32/StoreUint32

If I had a channel here, I would not be able to sleep - that much it would have bothered me :)

I have a kind of OCD when it comes to optimizing my software.

[Andrew Gerrand]

On 17 June 2014 19:15, Piotr Narewski <piot...@gmail.com> wrote:

Give me a break :)

Half of the people in this topic have been pretty rude and insulting to me and it definitely wasn't me who started.

In fact I've been pretty lenient and tolerant, considering how hostile you guys had been towards me.

Mate, you came out swinging.

You were "pissed off". The race detector is "broken". The memory model is "ridiculous".

And then it was you who first got personal by saying "I doubt you know what you are talking about."

The source of the bad attitude here is you. You seem like a decent guy and reasonably self-aware guy. Just don't get personal next time and everything will be fine.

Andrew

[andrewc...@gmail.com]

I know the feeling, and i can certainly understand why you want a LoadBool and StoreBool when you don't need an int32.

[Piotr Narewski]

Well, not that I think it will make any difference to you, but for the record.

I never said that the memory model was ridiculous - please don't accuse me of such things.

The claim that CPU cannot write a byte in a thread-safe way - that was ridiculous.

Also, had I known that me saying that something pisses me of would piss you off that much, I would have definitely phrased it otherwise.

Next time I will do better, knowing already how fragile you guys are :)

[Jesper Louis Andersen]

On Mon, Jun 16, 2014 at 10:04 AM, Piotr Narewski <piot...@gmail.com> wrote:

It's just does not make much sense that there are no such functions, forcing users to wrap single byte writes around complex solutions involving mutexes or channels.

This thread has not yet touched on the fact that different architectures have different memory behaviour. Digital Alpha is notorious for having almost no "synchronicity", with ARM being a nice second contender. x86-64 and x86 are in the other end of the spectrum, being very stable about what other processors can "see" once a write from the registers to memory is done.

When designing a language, you have to take into account that your system needs to run on several of these, providing memory barrier instructions as needed. The future might very well hold architectures which weakens the coherency of the shared memory segment further and further, going for a real NUMA style architecture. This is where having an explicit memory model is a necessity in Go, due to its choice of a shared-memory-architecture.

The idiomatic way would be to share memory by communicating, which means sync'ing on a channel. Note that channels in Go serve two purposes: They move data and they synchronize. It also works much better in said NUMA-style architectures, especially if you use a channel to copy data to another goroutine (because cross-memory-segment data grabs are going to be hellishly expensive on such archs).

For a few places, channel passing might be too slow. If you can't algorithmically change your code, then using one of the atomic "hacks" might be handy. But I'd seriously consider using channels instead. It also leads to better decoupled code in many cases.

--
J.

[Piotr Narewski]

Oh, that's interesting, actually.

I did not know that it would use completely different instructions for accessing ints, when you instruct it to not optimize.

But anyway, since you are there already, you can see that for building the code with -O0, it does not matter whether you declared your variable as volatile, or not - in both cases compiler will generate exactly the same code. Only with optimizations enabled, the volatile keyword makes a difference.

[egon]

Also, this is what Walter meant by ``Nobody agrees on what it means. What does "don't optimize" mean?``... i.e. the C specification doesn't exactly say how the "don't optimize" should look like, which means you get inconsistent behavior with different compilers. "INC", "ADD", "MOV/ADD/MOV" are all possible unoptimized translations for "a+=1".

+ egon