concurrent write-only to map ok?

[Alex Buchanan]

Basically, I want to spawn a goroutine per object, objects have unique IDs, and I want each routine to write its results to a shared map. Nothing will be reading from the map while the goroutines are running.

Is this safe?

Thanks

[Ian Lance Taylor]

No. In general, multiple goroutines writing to a single variable is
not safe. You need some sort of synchronization.

Ian

[Shawn Milochik]

On Fri, Oct 13, 2017 at 2:05 PM, Alex Buchanan <buchan...@gmail.com> wrote:

Basically, I want to spawn a goroutine per object, objects have unique IDs, and I want each routine to write its results to a shared map. Nothing will be reading from the map while the goroutines are running.

Is this safe?

Whether this is technically safe today in the reference implementation, it's better to be safe.

I suggest using a waitgroup and a channel to collect the results in an explicitly safe way. Perhaps something like this: https://play.golang.org/p/1NdQbAW65k

[Alex Buchanan]

Thanks for the quick help guys!

I ended up using https://godoc.org/golang.org/x/sync/syncmap

[Shawn Milochik]

On Fri, Oct 13, 2017 at 2:33 PM, Alex Buchanan <buchan...@gmail.com> wrote:

Thanks for the quick help guys!

I ended up using https://godoc.org/golang.org/x/sync/syncmap

Cool, that's what it's there for. Personally I'd avoid using empty interfaces, though.

[Michael Jones]

You can also write to a channel and have the reader of the channel write to the map

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones
michae...@gmail.com

[Marvin Renich]

* Alex Buchanan <buchan...@gmail.com> [171013 14:06]:

> Basically, I want to spawn a goroutine per object, objects have unique IDs,
> and I want each routine to write its results to a shared map. Nothing will
> be reading from the map while the goroutines are running.

You didn't give much detail, but if the map is map[ID]sometype and each
goroutine is only writing to its own element of the map, you could make
it a map[ID]*sometype. Assuming the goroutines are all spawned by a
single "main" goroutine, that main goroutine can create the sometype,
assign its address to the map, and pass the *sometype to the goroutine.
Now, only the main goroutine accesses the map. Accessing the individual
sometype instances does not interfere with any other goroutine, and no
explicit synchronization is needed.

...Marvin

[Slawomir Pryczek]

I think using standard sync'ed map may be bad idea for this use case (write-only access), especially taking into account that it's optimized for read access and stable keys, so each write will acquire mutex anyway.

if nothing is read from that map during the threads run, it should be probably much better to assign single map for every thread and at the end, copy/merge the results to single "output" using mutex or rwmutex. This way you can acquire mutex once per thread, acquiring it for every write from multiple threads (which is what sync/map will do) is a performance killer.

[Alex Buchanan]

So many good answers! Lots of different ways to accomplish this. I'll say that in this specific case, which is does not require every ounce of performance, syncmap is by far the simplest and least amount of code, so readability wins on this one.

[David Collier-Brown]

Like Mr Pryczek, I'd be concerned that
- the performance will be pessimized for this case, and
- the approach is un-go-ish.

Share memory by communicating is a Go idiom: send updates to a goroutine via a pipe, and by construction let the map never be written by others. 

It's more work to think about, but less to actually do (:-)) and a good introduction to the class of gopherish approaches.

--dave

[Jesper Louis Andersen]

An added advantage of a channel solution is that it is far easier to reason about from a correctness point of view. And it needs less information transfer on the side for human beings. Don't underestimate the power of a solution which is easily picked up by later readers of the code base.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Alex Buchanan]

Show me the code! :)

Here's mine: https://play.golang.org/p/ZwAlu5VuYr

[Bryan Mills]

On Sunday, October 15, 2017 at 1:45:06 PM UTC-4, Alex Buchanan wrote:

Show me the code! :)

Here's mine: https://play.golang.org/p/ZwAlu5VuYr

The problem that sync.Map is intended to address is cache contention.

Unfortunately, it doesn't currently address that problem well for stores of disjoint keys (see https://golang.org/issue/21035).

That said, if you're spawning a goroutine per write, you've already got enough contention (on the scheduler) that any other synchronization overhead is unlikely to matter.

As you say: show me the code! The example you gave is simple enough that it's not even clear why you need a map — a slice or channel would suffice.

(Personally, I'd be inclined to use a slice: https://play.golang.org/p/jpS06KFNbv)

So a more realistic use case would help: what do you want to do with this map when you're done with it?

[Alex Buchanan]

Good point, simple examples are almost never enough. I guess I was hoping that we'd end up with 3-5 examples of a really simple case. Maybe I'll come up with a more complex example.

I'm loading under ~5K files in parallel and possibly making an HTTP request for each. I like the slice solution, but I need the files to be accessible later by path (hence the map key). This is a command line utility. Adding parallelism already cut the time down from 1-2 minutes to 5 seconds, so I'm not concerned about map contention. So, my example does kinda capture the core algorithm, but it doesn't explain it very well.