Idioms for a Manager and multiple Worker goroutines?

[tmcd...@gmail.com]

Hi,

I'm trying to learn the best way to set up a "Manager" goroutine with a finite number of "Worker" goroutines. 

The goal is to throttle working on an unknown number of tasks, and then time out cleanly.

I've attempted an implementation here:

• http://play.golang.org/p/SWG05adnjo
  

My questions are:

1. What terms should I search with to find existing writings / answers for similar questions?
2. Am I missing a simpler way to achieve this?
   
3. If there are multiple ways to achieve the goal, which is the most idiomatic?

Thanks for your time,

-Travis

[Dmitry Vyukov]

I would do it in the following way:
http://play.golang.org/p/NDmq3Nkszw
I think it's simpler and provides simpler interface for users and
probably faster.

[William Kennedy]

One of the first things I did when I moved from C# to Go was port my core libs.

My Thread Pool package allows you to define a set of Go routines to perform work on a channel. Just like a thread pool. Maybe it can be of some help.

http://www.goinggo.net/2013/05/thread-pooling-in-go-programming.html?m=0

It also gives you stats of the work that's in queue and the number of active routines.

The final code is in GIT.

[Nguyên Nguyễn Văn Cao]

How about this 300 lines open source pkg: https://github.com/stefantalpalaru/pool

You can take a look at it.

P/s: After do alot of mess with by myself, I decide to use that pkg in my app...

Vào 23:20:41 UTC+7 Thứ năm, ngày 27 tháng sáu năm 2013, Travis McDemus đã viết:

[Dmitry Vyukov]

Looks a bit more complex than it needs to be:

pool.job_wanted_pipe = make(chan chan *Job)
pool.done_pipe = make(chan *Job)
pool.add_pipe = make(chan *Job)
pool.result_wanted_pipe = make(chan chan *Job)
pool.jobs_ready_to_run = list.New()
pool.jobs_completed = list.New()
pool.working_wanted_pipe = make(chan chan bool)
pool.stats_wanted_pipe = make(chan chan stats)
pool.worker_kill_pipe = make(chan bool)
pool.supervisor_kill_pipe = make(chan bool)

Also I do not understand the idea of having a dedicated managing
goroutine that shuffles messages between channels.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[RickyS]

Take a look at this implementation of a pool of worker goroutines.  It's short and sweet.

The number of goroutines should probably not be unlimited, as in the case of a design with a goroutine per user or per task.  More like the number of goroutines would be set by the capacity of your system (number of cpus?) and the number of outstanding requests at any one time.  Better to have large full channels of work to be done, than a huge number of goroutines, each one waiting it's turn.  Or so I think.  But apps with many thousands of goroutines also work well.

And be sure to read up on sync.WaitGroup.  Very useful, especially when the number of goroutines is not known in advance.