WaitGroup Count

8,287 views
Skip to first unread message

Enric Lafont

unread,
Aug 27, 2015, 9:49:57 AM8/27/15
to golang-nuts
I'm a system administrator doing golang development, part as a perl replacement, part because is fun.

I need to know the number of goroutines available in a WaitGroup, but there are no Count() method or any other way to obtain such information from the WaitGroup structure.

I've changed the base libraries to add the method I need, it's fast and simple , but It's not elegant and I don't believe that this must be the way to do it.

The other option is build a wrapper around WaitGroup so I can create the method Count(), but this seem as inelegant as modifying the base libraries and a burden on the system for such small change.

// in sync/waitgroup.go
func (wg *WaitGroup) Count() uint32 {
        return uint32(*(wg.state()) >> 32)
}


The programm I'm doing has a kind of goroutine spool, but the number of goroutines is not fixed, sometimes when channel (that feeds the goroutines) capacity  is full you push more goroutines to work to relieve some channel capacity. Goroutines dies when channel capacity is empty. And yes I know I can develop a more sophisticated architecture that takes care of the spool, but as it is now is extremely simple.

Congrats to the Go Team, 1.5 works wonders in my 32 core machine.

Anyone able to suggest a more right approach to have a WaiGroup.Count() ?
How to suggest to Go team to include the Count() method ? does it make sense ?

Thanks.

Enric

Egon

unread,
Aug 27, 2015, 10:05:03 AM8/27/15
to golang-nuts
Could you provide some more explanation how/when they get the items and how long is the whole system run, how long is it run, etc.? It's hard to say what the correct approach is, because some might work better than others in particular cases.

(Note: following code untested, might contain bugs)

E.g. could you just leave the goroutines let hang around and kill all at once:

func process(v int){
v = v * 2
time.Sleep(1*time.Second)
println(v)
}

func processqueue(queue chan int){
mux := make(chan int, 4)
processor := func(){
for v := range mux {
process(v)
}
}

for v := range queue {
select {
case mux <- v:
// do nothing when was able to send
default:
// add new processor
go processor()
mux <- v
}
}

close(mux)
}

Or an alternative approach:

func process(v int) {
v = v * 2
time.Sleep(1 * time.Second)
println(v)
}

func processqueue(queue chan int) {
mux := make(chan request, 4)

total := uint64(0)

type request struct {
v   uint64
die bool
}

processor := func() {
atomic.AddUint64(&total, 1)
defer atomic.AddUint64(&total, -1)

for v := range mux {
if v.die {
break
}
process(v)
}
}

for v := range queue {
select {
case mux <- request{v}:
// do nothing when was able to send
default:
// add new processor
go processor()
mux <- request{v}
}

if atomic.LoadUint64(&total) > 10 {
mux <- request{die: true}
}
}

close(mux)
}

+ Egon

Jan Mercl

unread,
Aug 27, 2015, 10:16:56 AM8/27/15
to Enric Lafont, golang-nuts


On Thu, Aug 27, 2015, 15:50 Enric Lafont <enr...@gmail.com> wrote:


I need to know the number of goroutines available in a WaitGroup, but there are no Count() method or any other way to obtain such information from the WaitGroup structure.

WaitGroup has no Count method because, in the general case, there's no non-racy way to provide that number - it may get invalidated the moment you know it. The racy value possibly obtained in any other way is thus not much useful.

--

-j

Enric Lafont

unread,
Aug 28, 2015, 4:59:41 AM8/28/15
to golang-nuts
Thanks for answering me.

The solution you propose is quite similar to the one I'm using, well you do a manual count when such number must be equal to the one wg.Count() returns.

What I'm doing is to process a directory in our NAS server badly organized, it has 60 million files and 350.000 subdirectories

The program I'm refactoring is an old PHP script that takes more than 4 days to complete operation, it's badly written and executed slowly, so, as the nature of the problem is a very concurrent one, Golang seem a perfect fit for me, well its the same I would have rewrite it in go even if the problem were not concurrent ... :)

The main problem is that a goroutine for every file overloads the machine and at the end golang complains about too many open files, and "fs.file-max = 4033344"  when the goroutines aproach the milion the program crashes.

I can not overload the machine, it's a server doing business procedures and our NAS too, so I need to limit the number of simultaneous goroutines, but, when I limit the number I find that some subdirectories are deeper than the number of max goroutines  I'm allowed to run, and the feeder channel gets full, so the program stops working, I needed a way to keep the goroutines on order, and this way is limiting the number but allowing more when the deep of the tree structure needs it, something like
            I:
                    for { // Repeat until fullFile name has space in dirch
                        select {
                        case dirch <- fullFile:
                            fmt.Printf("Queue: %3d, %3d  File: %v                         \r", len(dirch), wg.Count(), fullFile)
                            if wg.Count() < (uint32)(len(dirch)) { // len(dirch) is 256 now
                                go readDirectory(dirch, wr, wg) // The goroutine accepts an input read channel, an output  write channel , and a waitgroup
                                runtime.Gosched() // Give time to other goroutines
                            }
                            break I
                        default: // Done when channel capacity is full
                            runtime.Gosched()             // Give time to reduce channel pressure
                            if len(dirch) == cap(dirch) { // if none of the existing routines solved the problem, then
                                go readDirectory(dirch, wr, wg) // run extra routine to relieve channel capacity
                            }
                        }
                    }

This currently works, being able to count the number of goroutines lets me open more goroutines when the need arises, the goroutines dies when the queue (dirch) becomes empty, so I can keep the load on the system as low as possible, using this approach I have a peak of ~900 goroutines for a while and an average of 150 routines running concurrently, the load on the system is about 40%.

Thanks again for your time.

Enric Lafont

unread,
Aug 28, 2015, 5:04:08 AM8/28/15
to golang-nuts, enr...@gmail.com
Thanks Jon for answering

Yes I'm aware that wg.Count() is going to give me a past value the very moment I read it, but as I need to verify an extreme case (max number of goroutines) it is right even if a nanosecond later the value becomes untrue, Count() is more a guide on what should I do that a deterministic value on some computation.

Egon

unread,
Aug 28, 2015, 5:26:15 AM8/28/15
to golang-nuts


On Friday, 28 August 2015 11:59:41 UTC+3, Enric Lafont wrote:
Thanks for answering me.

The solution you propose is quite similar to the one I'm using, well you do a manual count when such number must be equal to the one wg.Count() returns.

What I'm doing is to process a directory in our NAS server badly organized, it has 60 million files and 350.000 subdirectories

The program I'm refactoring is an old PHP script that takes more than 4 days to complete operation, it's badly written and executed slowly, so, as the nature of the problem is a very concurrent one, Golang seem a perfect fit for me, well its the same I would have rewrite it in go even if the problem were not concurrent ... :)

The main problem is that a goroutine for every file overloads the machine and at the end golang complains about too many open files, and "fs.file-max = 4033344"  when the goroutines aproach the milion the program crashes.

I can not overload the machine, it's a server doing business procedures and our NAS too, so I need to limit the number of simultaneous goroutines, but, when I limit the number I find that some subdirectories are deeper than the number of max goroutines  I'm allowed to run, and the feeder channel gets full, so the program stops working, I needed a way to keep the goroutines on order, and this way is limiting the number but allowing more when the deep of the tree structure needs it, something like
            I:
                    for { // Repeat until fullFile name has space in dirch
                        select {
                        case dirch <- fullFile:
                            fmt.Printf("Queue: %3d, %3d  File: %v                         \r", len(dirch), wg.Count(), fullFile)
                            if wg.Count() < (uint32)(len(dirch)) { // len(dirch) is 256 now
                                go readDirectory(dirch, wr, wg) // The goroutine accepts an input read channel, an output  write channel , and a waitgroup
                                runtime.Gosched() // Give time to other goroutines
                            }
                            break I
                        default: // Done when channel capacity is full
                            runtime.Gosched()             // Give time to reduce channel pressure
                            if len(dirch) == cap(dirch) { // if none of the existing routines solved the problem, then
                                go readDirectory(dirch, wr, wg) // run extra routine to relieve channel capacity
                            }
                        }
                    }

This currently works, being able to count the number of goroutines lets me open more goroutines when the need arises, the goroutines dies when the queue (dirch) becomes empty, so I can keep the load on the system as low as possible, using this approach I have a peak of ~900 goroutines for a while and an average of 150 routines running concurrently, the load on the system is about 40%.

Thanks again for your time.

Can you separate your file feeding from the processing? 
Only one goroutine iterates the directories and rest do the processing:

Recursive decomposition (a solution by Dmitry from another thread https://groups.google.com/d/msg/golang-nuts/o2mkC1xsmC4/lDUjP2c1V6gJ):

Solution with busy waiting:

Without busy waiting:


+ Egon

Jan Mercl

unread,
Aug 28, 2015, 5:34:05 AM8/28/15
to Enric Lafont, golang-nuts

On Fri, Aug 28, 2015, 11:04 Enric Lafont <enricl@gmail.com> wrote:


If the need for Count() is actually just a need to limit a number of things, for example goroutines, then use a buffered channel.

        limiter := make(chan struct{}, limit)

Before creating a new thing (starting a new goroutine)

        limiter <- struct{}{}

Before destroying a thing (returning from a goroutine)

        <-limiter

Such limiter is synchronous and safe for concurrent use.

--

-j

mi...@leftronic.com

unread,
Feb 23, 2016, 6:06:18 PM2/23/16
to golang-nuts
I too wanted a way to see the number and was ok to accept nonsafe count. Just wanted to log the count from time to time and on SIGTERM  ("Signal received, waiting on %d jobs.") 


type CountWG struct {
 
sync.WaitGroup
 
Count int // Race conditions, only for info logging.
}


// Increment on Add
func
(cg CountWG) Add(delta int) {
  cg
.Count += delta
  cg
.WaitGroup.Add(delta)
}


// Decrement on Done
func
(cg CountWG) Done() {
  cg
.Count--
  cg
.WaitGroup.Done()
}

Egon

unread,
Feb 23, 2016, 6:31:16 PM2/23/16
to golang-nuts, mi...@leftronic.com
On Wednesday, 24 February 2016 01:06:18 UTC+2, mi...@leftronic.com wrote:
I too wanted a way to see the number and was ok to accept nonsafe count.

 
Just wanted to log the count from time to time and on SIGTERM  ("Signal received, waiting on %d jobs.") 


type CountWG struct {
 
sync.WaitGroup
 
Count int // Race conditions, only for info logging.

atomic.AddInt32(&cg.count, 1)
 

}


// Increment on Add
func
(cg CountWG) Add(delta int) {
  cg
.Count += delta
  cg
.WaitGroup.Add(delta)
}


// Decrement on Done
func
(cg CountWG) Done() {
  cg
.Count--

atomic.AddInt32(&cg.count, -1)

roger peppe

unread,
Feb 24, 2016, 9:41:54 AM2/24/16
to Enric Lafont, golang-nuts
Assuming that it's your business logic that takes the time,
why don't you separate the directory walking from the
processing itself, using one goroutine for traversing the
file tree, and all the others for doing the actual file processing?

Something like this, perhaps, although you'd probably want
to be a bit more careful about the error handling:

http://play.golang.org/p/Lp-Us_At_z

If the directory traversal is also a bottleneck, you could do
a similar thing using the concurrent filepath.Walk implementation
in github.com/MichaelTJones/walk instead of filepath.Walk
itself.

cheers,
rog.
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

nikos....@gmail.com

unread,
Sep 19, 2017, 11:33:03 PM9/19/17
to golang-nuts
This is something super cool :D why couldn't I think of this :)

On Friday, August 28, 2015 at 1:34:05 PM UTC+4, Jan Mercl wrote:
Reply all
Reply to author
Forward
0 new messages