gevent + gipc bounded pool?

82 views
Skip to first unread message

James Addison

unread,
Feb 9, 2017, 11:00:39 AM2/9/17
to gevent: coroutine-based Python network library
Looking for a reply to https://bitbucket.org/jgehrcke/gipc/issues/24/bounded-process-pool - pasting the question here for gipc familiar folks:


Is gipc compatible with a pool scenario? I want to take web requests to resize images - the resizing of images likely fits well into a separate process, given the GIL.

That said, I do not want uncontrolled process spawning. Is it possible to achieve something like gevent.pool.ThreadPool like upper limit pool sizing? Along the order of:

- On initialization, create a process pool size of 20. 
- Handle incoming web request, ask the pool to perform image resizing function and cooperatively wait for the result
- Once the result is returned from the process, return it to the web client
- Handle many incoming requests, but do not perform more than 20 image resizes at a time because the process pool is bounded

If this is possible, can the processes be reused to save on process initialization overhead? It would be Queue-based, I assume? Could I still wait cooperatively for the result?

The docs make it seem like gipc is great for spawning processes and waiting, but in an uncontrolled fashion? Or am I misunderstanding?

Many thanks.

ian.t....@gmail.com

unread,
Apr 8, 2020, 12:23:14 PM4/8/20
to gevent: coroutine-based Python network library
Did you ever get an answer to this question? I can't see the bitbucket issue.

Matt Billenstein

unread,
Apr 8, 2020, 1:48:43 PM4/8/20
to gev...@googlegroups.com
Looks like you found the github issue:
https://github.com/jgehrcke/gipc/issues/26

Couldn't you use a regular greenlet pool where each greenlet manages a
single long-running process?

m
> --
> You received this message because you are subscribed to the Google Groups
> "gevent: coroutine-based Python network library" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to gevent+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> gevent/0d47a859-2f1c-4e8f-a2e0-4727a1dd8701%40googlegroups.com.


--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/
Reply all
Reply to author
Forward
0 new messages