Concurrency fanout pattern

[Mandolyte]

After reading the nice article at http://blog.golang.org/pipelines on various concurrency patterns, I adapted to make a pure fanout pattern where the "workers" will be reading/writing to a database. In case of failures, the entire process will need to stop.

This seems to work the way I need it to. This will be my first foray into serious use of go routines. Suggestions welcomed.

http://play.golang.org/p/z0dzusYfR9

[wkharold]

The `concurrent` channel isn't really doing anything in your code. If you remove it the behavior is the same. A more canonical way of using channels would be to have a channel that all the workers use to receive tasks to do. I don't have time to whip up an example at the moment but I think one of the Go Concurrency talks does something like that.

[Mandolyte]

The concurrent channel is used by the task() method to signal when it's done. As it stands, it constrains how many go routines are running at the same time. If I remove the concurrent channel then all the go routines fire and finish. Not the behavior I need for my use case. I need to carefully control how many are go routines are working.

On the other hand, I could make it a bit more idiomatic by using a channel as a queue for the tasks. But I don't see how to ensure limits on how many go routines are in process without the concurrent channel acting as a throttle on the main routine distributing the work. In the meantime, I have found other pitches and talks, so I'll study some more before I begin coding this up for real.

Thanks for your feedback.

[wkharold]

The task() method isn't signaling anything. On line 18 it receives from the signal channel. The only reason it doesn't wait forever is that you send false down the channel at the beginning of the loop on line 27. The only thing that signaling is accomplishing is making the goroutine running task() wait to exit until the next iteration starts. Since every iteration creates a new goroutine that's pointless.

In a talk he gave in 2010 Rob Pike illustrates the implementation of a load balancer that does fanout: http://talks.golang.org/2010/io/balance.go.