Hi,

I have a program that while processing a data (implies in network I/O) could generate more data to be processed. I would like to use go routines to make this processing faster, but creating a new go routine (and using sync.WaitGroup) for each new data isn't a good solution (to many go routines can cause the error "too many open files"). I thought on making a pool of go routines with buffered channels and a round robin strategy to distribute the data to be processed, but I will never known when the program is finished (all go routines are waiting for more data and there's no data to process). Is there a pattern to solve this kind of problem?

Thanks in advance!
Rafael

Creating a new goroutine per file seems fine.  To avoid opening too many files at once, limit the number of open files using a semaphore (see Effective Go for how to do this using a buffered channel).

And if you have more than 1 request (each resulting in a set of file
operations), then you still can use WaitGroup to track request
completion, even if goroutines are persistent:

var wg sync.WaitGroup
wg.Add(...)
for ... {
    workchan <- &WorkItem{file: f, wg: &wg}
}
wg.Wait()

// workers
for w := range workchan {
  process(w)
  w.wg.Done()
}

I mean that you don't necessary need to terminate goroutines to use
WaitGroup. It can track completion of any abstract task.