gevent + gipc bounded pool?

[James Addison]

Looking for a reply to https://bitbucket.org/jgehrcke/gipc/issues/24/bounded-process-pool - pasting the question here for gipc familiar folks:


Is gipc compatible with a pool scenario? I want to take web requests to resize images - the resizing of images likely fits well into a separate process, given the GIL.

That said, I do not want uncontrolled process spawning. Is it possible to achieve something like gevent.pool.ThreadPool like upper limit pool sizing? Along the order of:

- On initialization, create a process pool size of 20. 

- Handle incoming web request, ask the pool to perform image resizing function and cooperatively wait for the result

- Once the result is returned from the process, return it to the web client

- Handle many incoming requests, but do not perform more than 20 image resizes at a time because the process pool is bounded

If this is possible, can the processes be reused to save on process initialization overhead? It would be Queue-based, I assume? Could I still wait cooperatively for the result?

The docs make it seem like gipc is great for spawning processes and waiting, but in an uncontrolled fashion? Or am I misunderstanding?

Many thanks.

[ian.t....@gmail.com]

Did you ever get an answer to this question? I can't see the bitbucket issue.

[Matt Billenstein]

Looks like you found the github issue:
https://github.com/jgehrcke/gipc/issues/26

Couldn't you use a regular greenlet pool where each greenlet manages a
single long-running process?

m

> --
> You received this message because you are subscribed to the Google Groups
> "gevent: coroutine-based Python network library" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to gevent+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> gevent/0d47a859-2f1c-4e8f-a2e0-4727a1dd8701%40googlegroups.com.


--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/