Looking for a reply to https://bitbucket.org/jgehrcke/gipc/issues/24/bounded-process-pool
- pasting the question here for gipc familiar folks:
Is gipc compatible with a pool scenario? I want to take web requests to resize images - the resizing of images likely fits well into a separate process, given the GIL.
That said, I do not want uncontrolled process spawning. Is it possible to achieve something like gevent.pool.ThreadPool like upper limit pool sizing? Along the order of:
- On initialization, create a process pool size of 20.
- Handle incoming web request, ask the pool to perform image resizing function and cooperatively wait for the result
- Once the result is returned from the process, return it to the web client
- Handle many incoming requests, but do not perform more than 20 image resizes at a time because the process pool is bounded
If this is possible, can the processes be reused to save on process initialization overhead? It would be Queue-based, I assume? Could I still wait cooperatively for the result?
The docs make it seem like gipc is great for spawning processes and waiting, but in an uncontrolled fashion? Or am I misunderstanding?