context.Context and worker pools

[andrey mirtchovski]

While trying to retrofit context.Context within a
worker-pool-patterned package, where work is sent down a channel to be
picked up by number of worker goroutines.

I'm running against the mantra of "Do not store Contexts inside a struct type".

For example, I want to put a timeout on the amount of time spent
waiting to write on a channel, plus the time it takes for my write to
be consumed on the other end (the time the requests spends enqueued on
the channel buffer). How would I go about doing that without embedding
a context.Context inside my write?

A typical example, here is how i would do it without the channel send:

func p(ctx context.Context, req Request) {
ctx, _ = context.WithTimeout(ctx, Timeout)
select {
case <-ctx.Done:
// timed out
default:
worker(ctx, req) // will use ctx to determine if timeout expired
}
}

With a channel I may do something like:

func c(req Request, in chan Request) {
in <- req
<-out
}

func worker(in chan Request) {
for _, v := range in {
// i want to cancel here if 'v' has lived past its timeout
// or do work if not
}
}

And I want to have a meaningful way to measure enqueue time before req
is consumed by a worker in the second example.

Hopefully that makes sense.

[Bruno Albuquerque]

I did something like this here:

https://github.com/brunoga/workerpool

I do not store the context anywhere, I pass it to my Start() method and it passes it down to any place I need it.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bryan Mills]

Pool the resources, not the workers.

For example, if your resource is “memory footprint of in-flight requests”, use a semaphore instead. (There is a “worker pool” example in the package docs.)

If your resource is a fixed-size set of connections, you can store the connections in a buffered channel and select on acquiring it or ctx.Done().

[andrey mirtchovski]

thanks to both for your replies!