Evenly distribute to number of go routines

[Kyle Wolfe]

How would I go this as I can see a lot of my future projects doing this. Say I have one go routine (or main function) reading from a result set or text file, how would I evenly startup x number of go routines and evenly distribute the workload across them? So in other words, unless theres not enough work to go around, there should always be x go routines (no more) executing what my main function is feeding out.

[Aaron Blohowiak]

http://play.golang.org/p/kJJT5wQ63w  Actual fairness not guaranteed, but is good enough.  Uses a waitgroup so you can wait until all the lines are processed.

[Kyle Wolfe]

Can you explain what sync is doing? After posting I realized this was probably just a simple matter of throwing my population at a channel (perhaps buffered) and having x number of go routines read from that same channel.

[Kyle Wolfe]

How can I wait until channel c is closed in this script?

package main

import (
    "fmt"
    "time"
)
var c chan int

func populateChan(c chan int) {
    for i := 0; i < 10; i++ {
        c <- i
        time.Sleep(time.Second * 1)
    }
    close(c)
}

func worker(i int, c chan int) {
    fmt.Println("worker i:", <-c)
}

func main() {
    c := make(chan int)
    go populateChan(c)
    for i := 0; i < 2; i++ {
        go worker(i, c)
    }
}

[al...@lx.lc]

Without figuring out exactly what your script is intended to do, I'll say this -
1 - You are declaring c as a channel globally, then redefining it locally and passing it around.  Don't do this - pick one or the other.
2 - The multiple return value of a channel receive indicates whether a channel is closed or not.  So, value, ok := <-c  .  ok is false on a closed channel.

sync.WaitGroup is one way of making sure all tasks complete.  If the number of items is known, you could instead just do that many receives from the channel.  I prefer the latter way when the number of tasks is known and the function is to return some result in all cases.  Otherwise, I use a WaitGroup.

Thanks,
Alex

[Kyle Wolfe]

Can I check if c is open from main without receiving anything?

[Ian Lance Taylor]

On Fri, Nov 1, 2013 at 10:12 AM, Kyle Wolfe <kyle.a...@gmail.com> wrote:
> Can I check if c is open from main without receiving anything?

No.

(That is because checking whether a channel is closed without
receiving a value is inherently racy. You could check that it is
closed, find that it is not, and then before you can do anything some
other goroutine could receive the last value and the channel could be
closed.)

Ian

[Kyle Wolfe]

Ok, I think I got it, but now I need to know how to avoid a deadlock:

http://play.golang.org/p/yX2NLJ8rQh

[Matthew Kane]

You never actually close the channel after sending all the input. Try this: http://play.golang.org/p/U9mw1uM5E9

On Fri, Nov 1, 2013 at 2:12 PM, Kyle Wolfe <kyle.a...@gmail.com> wrote:

Ok, I think I got it, but now I need to understand how to avoid a deadlock (all routines are asleep)

package main

import (
    "fmt"
    "time"

    "math/rand"
    "sync"
)

var w sync.WaitGroup

func pupulateChan(c chan int) {
    for i := 0; i < 100; i++ {
        c <- i

    }
}

func worker(i int, c chan int) {

    for{
        val, open := <-c
        if(open == false) {
            break
        }
        fmt.Println("worker", i, "recieved", val)
        time.Sleep(time.Millisecond * time.Duration(rand.Intn(10)))
    }
    w.Done()
}

func main() {
    //define worker count
    workers := 10
   
    //create channel
    c := make(chan int)

    //populate the channel
    go pupulateChan(c)
   
    //tell sync how many goroutines are out there
    w.Add(workers)
   
    //start workers to read from channel
    for i := 0; i < workers; i++ {
        go worker(i, c)
    }
   
    //wait for goroutines to complete
    w.Wait()
}

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
matt kane
twitter: the_real_mkb / nynexrepublic
http://hydrogenproject.com

[Nico]

http://play.golang.org/p/D8w5drmEW-

package main

import (
"fmt"

"sync"
"time"
)

var w sync.WaitGroup

func pupulateChan(c chan int) {

for i := 0; i < 10; i++ {
c <- i
time.Sleep(time.Second * 1)
}
close(c)
}

func worker(i int, c chan int) {

for val := range c {

fmt.Println("worker", i, "recieved", val)
}

fmt.Println("worker", i, "done")

w.Done()
}

func main() {
//define worker count

workers := 2

[Kyle Wolfe]

Ha woops, thank you. So is this code example the safe and correct way to split work?

[Edward Muller]

I used something like this to do what I think you are asking: https://gist.github.com/freeformz/7036743

Fairness is not guaranteed, but IMNSHO it doesn't really matter.

On Wed, Oct 30, 2013 at 8:17 AM, Kyle Wolfe <kyle.a...@gmail.com> wrote:

How would I go this as I can see a lot of my future projects doing this. Say I have one go routine (or main function) reading from a result set or text file, how would I evenly startup x number of go routines and evenly distribute the workload across them? So in other words, unless theres not enough work to go around, there should always be x go routines (no more) executing what my main function is feeding out.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Edward Muller
@freeformz