map for readonly access will always remain thread-safe ?

6052 views
Skip to first unread message

nicolas...@gmail.com

unread,
Nov 14, 2013, 12:32:57 PM11/14/13
to golan...@googlegroups.com
I want to create a map and fill it with values, in init().

After init(), this map will only be accessed for reading, never for writing.

I imagine that multiple goroutines can access the map to read its content
  - at the same time
  - even if they run on different OS threads
  - without needing to protect access to this map with e.g. sync.Mutex


But I don't find anything in Go specification or other document saying that a map is safe to use like this.

One can imagine that for performance reason, an implementation of the map may make some internal reorganization of its data during a read operation, or add a little cache to store the last value retrieved, or things like that.
In this case, the map won't be safe any more for concurrent read operations.
 
In fact, such behaviour existed and has been fixed, in Issue 5179 "runtime: concurrent map reads grow map and corrupt map internals"  http://code.google.com/p/go/issues/detail?id=5179.

Does such guarantee exists somewhere, that a map will never ever change anything in its internal state during read operations ?




Jesse McNelis

unread,
Nov 14, 2013, 10:06:43 PM11/14/13
to nicolas...@gmail.com, golang-nuts
On Fri, Nov 15, 2013 at 4:32 AM, <nicolas...@gmail.com> wrote:
I want to create a map and fill it with values, in init().

After init(), this map will only be accessed for reading, never for writing.

I imagine that multiple goroutines can access the map to read its content
  - at the same time
  - even if they run on different OS threads
  - without needing to protect access to this map with e.g. sync.Mutex 

But I don't find anything in Go specification or other document saying that a map is safe to use like this.

It's not specified. 
 
One can imagine that for performance reason, an implementation of the map may make some internal reorganization of its data during a read operation, or add a little cache to store the last value retrieved, or things like that.
In this case, the map won't be safe any more for concurrent read operations.

Yep, an implementation of the Go language could make reads from maps unsafe for concurrent access.
 
In fact, such behaviour existed and has been fixed, in Issue 5179 "runtime: concurrent map reads grow map and corrupt map internals"  http://code.google.com/p/go/issues/detail?id=5179.

Does such guarantee exists somewhere, that a map will never ever change anything in its internal state during read operations ?

I believe the Go developers have decided to keep map reads thread-safe in the standard implementation(they have a test for it), but since the spec doesn't guarantee it other implementations could make map reads not thread-safe.
Since they aren't specified to be safe, I assume they are unsafe and use a sync.Mutex to protect my concurrent map accesses.

Ian Lance Taylor

unread,
Nov 15, 2013, 12:19:38 AM11/15/13
to Jesse McNelis, nicolas...@gmail.com, golang-nuts
On Thu, Nov 14, 2013 at 7:06 PM, Jesse McNelis <jes...@jessta.id.au> wrote:
> On Fri, Nov 15, 2013 at 4:32 AM, <nicolas...@gmail.com> wrote:
>>
>> I want to create a map and fill it with values, in init().
>>
>> After init(), this map will only be accessed for reading, never for
>> writing.
>>
>> I imagine that multiple goroutines can access the map to read its content
>> - at the same time
>> - even if they run on different OS threads
>> - without needing to protect access to this map with e.g. sync.Mutex
>>
>>
>> But I don't find anything in Go specification or other document saying
>> that a map is safe to use like this.
>
>
> It's not specified.

A read from a map is like any other read from a variable. Thus it's
fine if multiple goroutines read from a map simultaneously. But if
one goroutine reads from a map while another writes to a map, or if
two goroutines write to a map, then the program must synchronize those
goroutines in some way.

Ian

Dominik Honnef

unread,
Nov 15, 2013, 12:58:13 AM11/15/13
to golan...@googlegroups.com
Ian Lance Taylor <ia...@golang.org> writes:

>> It's not specified.
>
> A read from a map is like any other read from a variable. Thus it's
> fine if multiple goroutines read from a map simultaneously. But if
> one goroutine reads from a map while another writes to a map, or if
> two goroutines write to a map, then the program must synchronize those
> goroutines in some way.
>
> Ian

Does the existing specification actually guarantee this? Because there
are certainly ways to implement it that don't have that guarantee.
https://code.google.com/p/go/source/detail?r=af469280a34b comes to mind
in particular.

I know that this implementation of Go seems to guarantee it, but could
there be an alternate implementation that implements maps that aren't
safe for concurrent reads and still implement the specification?

Kevin Gillette

unread,
Nov 15, 2013, 4:53:12 AM11/15/13
to golan...@googlegroups.com
Implementation details aside, from a usability perspective it's a rather unfortunate thing when concurrent no-write access results in race conditions.

Other such questions about read-only map access have been answered with "yes, it's safe," and it's a widely enough held assumption that concurrent no-write access to anything in Go is safe; it may as well be specified, since any implementation breaking that assumption will break a lot of programs.

Vincent Callanan

unread,
Nov 15, 2013, 5:01:53 AM11/15/13
to golan...@googlegroups.com
Assurances aside, this really needs to be firmed up in the spec as it is so fundamental.

nicolas...@gmail.com

unread,
Nov 15, 2013, 11:11:44 AM11/15/13
to golan...@googlegroups.com, nicolas...@gmail.com
In C, using a pointer to type A to write data, and reading the same data with a pointer to type B was also considered "safe" for decades.
But one day, compilers made more agressive optimizations and such programmatic idiom did not work any more (strict-aliasing problem).
Indeed, this broke a lot of programs, even Linux kernel. The answer was : such thing has never been explicitely allowed by the C specs.

The question is : something works today, but will it work tomorrow ?
The problem that occurred ten years ago in C with this strict-aliasing problem shows that "break a lot of programs" or "widely enough held assumption" are not enough to ensure that something is "safe" to do.

Ian Lance Taylor

unread,
Nov 15, 2013, 1:12:53 PM11/15/13
to golang-nuts
On Thu, Nov 14, 2013 at 9:58 PM, Dominik Honnef <domi...@fork-bomb.org> wrote:
> Ian Lance Taylor <ia...@golang.org> writes:
>
>>> It's not specified.
>>
>> A read from a map is like any other read from a variable. Thus it's
>> fine if multiple goroutines read from a map simultaneously. But if
>> one goroutine reads from a map while another writes to a map, or if
>> two goroutines write to a map, then the program must synchronize those
>> goroutines in some way.
>
> Does the existing specification actually guarantee this? Because there
> are certainly ways to implement it that don't have that guarantee.
> https://code.google.com/p/go/source/detail?r=af469280a34b comes to mind
> in particular.
>
> I know that this implementation of Go seems to guarantee it, but could
> there be an alternate implementation that implements maps that aren't
> safe for concurrent reads and still implement the specification?

The relevant document is here is the memory model
(http://golang.org/ref/mem). The memory model is written in terms of
reads and writes to a variable. You are essentially saying "a map is
not a variable, so we need a separate specification for the behaviour
of a map." I am simply saying: a map is a variable.

I'm not opposed to adding a sentence about maps to the memory model.
I'm just not sure it's required.

You are of course correct that maps can be implemented such that they
do not meet the requirements of the Go memory model. It's possible to
implement non-map variables that way too, e.g., by caching reads and
writes across synchronization boundaries. Either way it would be a
case where the implementation fails to meet the spec, or, in other
words, it would be a bug.

Ian

Ian Lance Taylor

unread,
Nov 15, 2013, 1:19:19 PM11/15/13
to nicolas riesch, golang-nuts
On Fri, Nov 15, 2013 at 8:11 AM, <nicolas...@gmail.com> wrote:
> In C, using a pointer to type A to write data, and reading the same data
> with a pointer to type B was also considered "safe" for decades.
> But one day, compilers made more agressive optimizations and such
> programmatic idiom did not work any more (strict-aliasing problem).
> Indeed, this broke a lot of programs, even Linux kernel. The answer was :
> such thing has never been explicitely allowed by the C specs.

That may well be the perspective of a C programmer, but I don't think
it's an accurate representation of how C developed. During the
initial C standardization process, this issue was discussed. At that
time, the standard writers decided that they would not require that
data written by a pointer to type A be readable using a pointer to
type B. This is explicitly spelled out in the ANSI C89 standard aka
the ISO C90 standard, and similar wording has carried forward into
later C and C++ standards. I believe this decision was made based on
experience with Fortran compilers.

Over time C compiler implementors modified their compilers to follow
the new standard more closely. They also strengthened their
optimizations. This increasingly caused programs to fail due to type
aliasing. At that time programmers complained, but the answer was not
"such thing has never been explicitely allowed by the C specs." The
answer was "the C language standard specifically permit compilers to
make this optimization." And, of course, most production compilers
provided options to disable these optimizations.

None of this has anything to do with Go, which is a different
language with a different philosophy.

Ian

nicolas...@gmail.com

unread,
Nov 15, 2013, 4:07:03 PM11/15/13
to golan...@googlegroups.com, nicolas...@gmail.com
Excuse me, Ian, I did not understand your first answer when I read it.

But is is clear, now : "A MAP IS A VARIABLE", exactly like an int or a string, just a more complex one.

For a Go programmer, a map must be considered "macroscopically" like a very basic type, like int or string, which is provided by Go. The inner mechanism of a map is not relevant for the programmer.

It is obvious that if an "int" variable is assigned a value in init(), and no other assigment ever occurs, many goroutines can read it safely without synchronization.

The fact that in Go, a "map" can be considered like an "int" make it possible to replace the word "int" by "map" in the above sentence, and all becomes clear !

The difficult "philosophical" thing to understand was just : "a map is a VARIABLE" ;-)))

Thank you very much for your explanation, because I think it is something fundamental to understand how Go works.

(and thank you for the explanation about C strict-aliasing, which has bitten me at that time ;-)


nicolas riesch



Vincent Callanan

unread,
Nov 16, 2013, 5:08:02 AM11/16/13
to golan...@googlegroups.com

You are of course correct that maps can be implemented such that they
do not meet the requirements of the Go memory model.  It's possible to
implement non-map variables that way too, e.g., by caching reads and
writes across synchronization boundaries.  Either way it would be a
case where the implementation fails to meet the spec, or, in other
words, it would be a bug.

...not to mention the CPU cache itself which is why we never want to have to use read mutexes for "fixed" map variables.

The "variable" get-out clause is surely a little too "facile" in this instance.

Arrays/slices are special cases (witness race conditions)
So too maps (there is already a statement somewhere about thread-safety of maps) 

Remember, a non-novice poster to this thread has already stated:
"Since they aren't specified to be safe, I assume they are unsafe and use a sync.Mutex to protect my concurrent map accesses".

We wouldn't want that to become the norm.
Go maps very speedy for "const string lookups" even over relatively small data sets.
Having to use mutexes in these situations would be unthinkable.

By firming up the spec, the intent is absolutely clear.

Vincent 

Dmitry Vyukov

unread,
Nov 16, 2013, 6:04:00 AM11/16/13
to Ian Lance Taylor, golang-nuts
On Fri, Nov 15, 2013 at 10:12 PM, Ian Lance Taylor <ia...@golang.org> wrote:
> On Thu, Nov 14, 2013 at 9:58 PM, Dominik Honnef <domi...@fork-bomb.org> wrote:
>> Ian Lance Taylor <ia...@golang.org> writes:
>>
>>>> It's not specified.
>>>
>>> A read from a map is like any other read from a variable. Thus it's
>>> fine if multiple goroutines read from a map simultaneously. But if
>>> one goroutine reads from a map while another writes to a map, or if
>>> two goroutines write to a map, then the program must synchronize those
>>> goroutines in some way.
>>
>> Does the existing specification actually guarantee this? Because there
>> are certainly ways to implement it that don't have that guarantee.
>> https://code.google.com/p/go/source/detail?r=af469280a34b comes to mind
>> in particular.
>>
>> I know that this implementation of Go seems to guarantee it, but could
>> there be an alternate implementation that implements maps that aren't
>> safe for concurrent reads and still implement the specification?
>
> The relevant document is here is the memory model
> (http://golang.org/ref/mem). The memory model is written in terms of
> reads and writes to a variable. You are essentially saying "a map is
> not a variable, so we need a separate specification for the behaviour
> of a map." I am simply saying: a map is a variable.
>
> I'm not opposed to adding a sentence about maps to the memory model.
> I'm just not sure it's required.


Maps are not "as safe as int". "as safe as int" implies that different
goroutines can freely operate on different variables of the type. Maps
are as safe as *int. I am not sure it's clearly and explicitly stated
in the docs.

nicolas...@gmail.com

unread,
Nov 16, 2013, 10:08:36 AM11/16/13
to golan...@googlegroups.com

> Maps are not "as safe as int". "as safe as int" implies that different
> goroutines can freely operate on different variables of the type. Maps
> are as safe as *int. I am not sure it's clearly and explicitly stated
> in the docs.

This time, I think this is correct : a map[int]Type_A is as safe as a pointer to an array of Type_A.


 

Vincent Callanan

unread,
Nov 16, 2013, 2:20:31 PM11/16/13
to golan...@googlegroups.com, nicolas...@gmail.com
de facto or de jure?
:-)

John Nagle

unread,
Nov 16, 2013, 7:12:25 PM11/16/13
to golan...@googlegroups.com
On 11/14/2013 9:32 AM,
nicolas...@gmail.com wrote:
> I want to create a map and fill it with values, in init().
>
> After init(), this map will only be accessed for reading, never for writing.

You could use a read/write lock around it, and when done writing,
lock it for reading and never unlock it again.

John Nagle

nicolas...@gmail.com

unread,
Nov 17, 2013, 9:25:53 AM11/17/13
to golan...@googlegroups.com
Le dimanche 17 novembre 2013 01:12:25 UTC+1, John Nagle a écrit :
> You could use a read/write lock around it, and when done writing,
> lock it for reading and never unlock it again.

                                John Nagle

The answer of Ian seems to be clear. There is no need for a lock in this case.

 As an exemple, the sqlite driver code http://code.google.com/p/gosqlite/source/browse/sqlite/sqlite.go has a map
var errText = map[Errno]string{
        1:   "SQL error or missing database",
        2:   "Internal logic error in SQLite",
        3:   "Access permission denied",
        ,,,
}
which access is not protected by any lock. The map is not initialized in init(), but it makes no difference.


alecb...@gmail.com

unread,
Apr 13, 2020, 1:18:01 PM4/13/20
to golang-nuts
Very old thread I know, but:

> a map is a variable

This seems clear enough, but what doesn't seem obvious to me is that an expression like m["foo"] can be treated as "just a variable". Does the spec also make that clear?

E.g., if I roll my own map type, there's no reason to assume m.Get("foo") is safe for concurrent reads. Naively, I'd assume m["foo"] is analogous to something like m.Get("foo"). Does the spec make it clear that that's not the case, as far as memory semantics go?

Ian Lance Taylor

unread,
Apr 13, 2020, 1:59:47 PM4/13/20
to alecb...@gmail.com, golang-nuts
On Mon, Apr 13, 2020 at 10:17 AM <alecb...@gmail.com> wrote:
>
> Very old thread I know, but:
>
> > a map is a variable
>
> This seems clear enough, but what doesn't seem obvious to me is that an expression like m["foo"] can be treated as "just a variable". Does the spec also make that clear?
>
> E.g., if I roll my own map type, there's no reason to assume m.Get("foo") is safe for concurrent reads. Naively, I'd assume m["foo"] is analogous to something like m.Get("foo"). Does the spec make it clear that that's not the case, as far as memory semantics go?

My opinion from more than six years ago is unchanged.

A map m is a variable. An expression like m["foo"] is a read of that
variable. A statement like m["foo"] = "bar" is a write of that
variable. Yes, m["foo"] is safe for concurrent reads. I don't see
any way to insert a distinction between m and m["foo"] with regard to
whether a map is a variable. I'm not saying that m["foo"] is a
variable (it's not, because it's not addressable). I'm saying that m
is a variable and m["foo"] is a read of that variable m, just as if p
is a variable of pointer type then *p is a read of that variable p.

I agree that if you roll your own map type then m.Get("foo") might not
necessarily be safe for concurrent calls, but I'm asserting that since
in Go a map is variable, m["foo"] must be safe for concurrent
execution. (And, for what it's worth, in the current implementations
that is true.)

Ian
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/75e5430c-855e-479b-b1fa-d8ea196db3b9%40googlegroups.com.

Alec Benzer

unread,
Apr 13, 2020, 2:11:57 PM4/13/20
to Ian Lance Taylor, golang-nuts
I'm not saying that m["foo"] is a
variable (it's not, because it's not addressable).  I'm saying that m
is a variable and m["foo"] is a read of that variable m, just as if p
is a variable of pointer type then *p is a read of that variable p.

Ah, ok, this framing makes sense to me!
Reply all
Reply to author
Forward
0 new messages