how cheap is pool.spawn() ?

[Troy Cauble]

I read somewhere that pool.spawn() is cheap.

Currently I have Pool(size=N) greenlets running a worker function that pulls from a queue.

        def worker(function1, w_id):

            while True:

                item = queue.get()

                do_stuff(item)

So the spawns happen up front and the workers last "forever".

It would be more straightforward to let the greenlets die and spawn new ones.

while True:

  item = q.get()

  pool.spawn(do_stuff, item)   # blocking

But how cheap is spawning?

Thanks!

-troy

[Jason Madden]

> On Nov 17, 2016, at 16:38, Troy Cauble <troyc...@gmail.com> wrote:
>
>
> But how cheap is spawning?

It's likely cheap enough. The overhead is something like two method calls and acquiring a semaphore, which, if uncontended, is also cheap.

Notice I didn't quantify that, because "cheap" is relative, and a lot depends on things like how frequently new items arrive in the queue, how long the tasks run, how often the pool is expected to be full, etc. If you're concerned, my best advice would be to measure the alternatives on the platform(s) you care about and make your decision based on that, taking into account the other factors as needed.