subprocess and cherrypy
Rita Morgan
My web application involves many subprocesses to get data back to the
website. I was wondering, if there was a best practices guide or even
better some examples on how to handle this use case?
Daniel Bryan
I don't know about best practises within the CherryPy community, but since CherryPy integrates very well with arbitrary Python libraries, I've dealt with the problem of gathering data from multiple sources (such as third party APIs) by using a message queue framework such as RabbitMQ.
It really depends on what your problem actually is. Are you doing lots of sequential IO work before you can send a response? Then a message queue will mean you can respond at the speed of the slowest IO response and no slower. Are you waiting around in general for some slow IO and hogging precious resources by keeping a process or thread open for that request the whole time? Then a framework for asynchronous IO such as Twisted might be more helpful.
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To post to this group, send email to cherryp...@googlegroups.com.
To unsubscribe from this group, send email to cherrypy-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.
Adam Skutt
As Daniel said, the exact right solution depends on your needs. I
wrote a CherryPy bus plugin to spawn off subprocesses for a commerical
project that worked well enough for long-running daemon processes that
needed to start and stop with my Python application. It was similar
in nature to the BackgroundTask plugin that exists in the standard
distribution, but starts processes instead.
The one tip I can provide is make sure that you spawn your processes
before the threads start, or you may run into some trouble. This
means either using a bus plugin or spawing them before starting the
CherryPy engine.
I'll also note that we were eventually planning to move to Celery to
gain distributed processing support.
Adam