Go Routines, Timeouts, and resource cleanup question

[Shmuel C]

If I am doing something similar to the code below, what happens if in the "doWork" function, the timeout operation fires before the processing goroutine completes - in this case the processing goroutine will be sending data to a remote webservice - does the underlying network socket get closed/cleaned up, or will it start leaking resources?  My aim to to give the processing routine a maximum amount of time to complete it's job and timeout the operation if it cannot succeed (in this case, uploading the results in time).  Just not sure about the consequences of 'abandoning' the network send and possibly creating a resource leak.

// create a bunch of workers and process some tasks

func SomeGeneratorFunc(numWorkers int) {

// create numWorkers:  runtime.NumCPU()

workResults := make (chan workItemResult, numWorkers)

// have each worker perform the task

for i := 0; i < numWorkers; i++ {

go doWork (getWorkData(), workResults)

}

// log the work results

for i := 0; i < numWorkers; i++ {

s := <-workResults

logResult (s)

}

}

// do the work

func doWork (wi workItem, result chan workItemResult) {

timeout := make(chan workItemResult)

ch := make(chan workItemResult)

var dummy workItemResult

// process the work item

go func(w workItemResult) {

x := processData(w)

ch <- sendDataRemote(x) // send the results remotely, return result to channel

}(wi)

// Timeout the work operation if it takes 30 seconds or more

go func (wi workItemResult) {

time.Sleep(time.Duration(30e9))

timeout <- wi

}(dummy)

select {

case result = <- ch:

log.Println ("Processing finished")

case result = <- timeout:

log.Println("Processing took too long")

}

return

}

[James Bardin]

On Tuesday, April 22, 2014 8:45:48 AM UTC-4, Shmuel C wrote:

If I am doing something similar to the code below, what happens if in the "doWork" function, the timeout operation fires before the processing goroutine completes - in this case the processing goroutine will be sending data to a remote webservice - does the underlying network socket get closed/cleaned up, or will it start leaking resources?  My aim to to give the processing routine a maximum amount of time to complete it's job and timeout the operation if it cannot succeed (in this case, uploading the results in time).  Just not sure about the consequences of 'abandoning' the network send and possibly creating a resource leak.

I don't see any network code here, so I can't comment if you are failing to free resources there.

But you are going to leak goroutines in this code...

 

// create a bunch of workers and process some tasks

func SomeGeneratorFunc(numWorkers int) {

// create numWorkers:  runtime.NumCPU()

workResults := make (chan workItemResult, numWorkers)

// have each worker perform the task

for i := 0; i < numWorkers; i++ {

go doWork (getWorkData(), workResults)

}

// log the work results

for i := 0; i < numWorkers; i++ {

s := <-workResults

logResult (s)

}

If any calls were timed out, you're not going to get enough reads to exit this loop, or you're going to block until they are fulfilled regardless of timeouts.

Anyway, use a sync.WaitGroup for this type of resource management. 

 

}

// do the work

func doWork (wi workItem, result chan workItemResult) {

timeout := make(chan workItemResult)

ch := make(chan workItemResult)

var dummy workItemResult

// process the work item

go func(w workItemResult) {

x := processData(w)

ch <- sendDataRemote(x) // send the results remotely, return result to channel

}(wi)

That will block if you never don't read ch. You can add a buffer of 1 to let it finish.

You still need to make sure sendDataRemote has a way to complete, which there's no way to do from this code. Deferring it with a goroutine doesn't make it magically go away.

 

// Timeout the work operation if it takes 30 seconds or more

go func (wi workItemResult) {

time.Sleep(time.Duration(30e9))

timeout <- wi

}(dummy)

The time package has various timers, use them.

Your doWork function could look something like this http://play.golang.org/p/cutMXaItC-

 

[Shmuel C]

Thanks for your comments.  I'm not as concerned with the pseudo code specifically, more so with wondering how cleanup happens when you use a timeout in a select - all the examples I've seen don't make it clear the proper way to handle this cleanup.  For instance, what if the "processing" routine is making a database call, or what I was (poorly) trying to represent making a network call in the processing routine - when the timeout in the select "fires" what happens to the processing routine from a resource cleanup standpoint - does the routine continue running to completion, regardless of the fact that it's output will not be used?  If so, then as long as that processing code does cleanup properly (as if it would never be "short-circuited", then I guess it doesn't matter.  But if the execution of the timeout, and subsequent return from the containing function doesn't allow the function to complete, then I can see resource leaks occurring.  

[James Bardin]

On Tue, Apr 22, 2014 at 7:07 PM, Shmuel C <scohen...@gmail.com> wrote:

Thanks for your comments.  I'm not as concerned with the pseudo code specifically, more so with wondering how cleanup happens when you use a timeout in a select - all the examples I've seen don't make it clear the proper way to handle this cleanup.  

Each goroutine follows its thread of execution until they return, just like any other function. There's no way to GC a goroutine.

 

For instance, what if the "processing" routine is making a database call, or what I was (poorly) trying to represent making a network call in the processing routine - when the timeout in the select "fires" what happens to the processing routine from a resource cleanup standpoint - does the routine continue running to completion, regardless of the fact that it's output will not be used?  

yes.

 

If so, then as long as that processing code does cleanup properly (as if it would never be "short-circuited", then I guess it doesn't matter.  But if the execution of the timeout, and subsequent return from the containing function doesn't allow the function to complete, then I can see resource leaks occurring.  

Yes, if a goroutine is blocked from returning, it will continue to use consume resources. Regardless of whether the code you posted was real or not, the comments I made should still answer your questions. The only "leaks" you could have in that code are goroutines that don't return; so just make sure they can return in all cases.

[Shmuel C]

Thanks!