Asynchronous subprocess.Popen

[Neil]

I have a handler that, among other things, resizes an uploaded image
file by piping the file data through GraphicsMagick using
subprocess.Popen. Here's a simplified version:

#!/usr/bin/env python

import subprocess
import tornado.httpserver
import tornado.web


class ImageHandler(tornado.web.RequestHandler):
def get(self):
self.write('<form method="post" enctype="multipart/form-
data">'
'<input type="file" name="image"/>'
'<input type="submit" value="Convert Image"/>'
'</form>')

def post(self):
command = "gm convert - -resize 50 png:-"
pipe = subprocess.Popen(command.split(),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
image = self.request.files["image"][0]["body"]
image = pipe.communicate(image)[0]
self.set_header("Content-Type", "image/png")
self.write(image)


def main():
app = tornado.web.Application([(r"/", ImageHandler)])
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(8080)
tornado.ioloop.IOLoop.instance().start()


if __name__ == "__main__":
main()


I would like to do this asynchronously, so that the imaging processing
does not block, but instead calls a callback when complete.

I found this asynchronous implementation of subprocess.Popen:

PEP: http://www.python.org/dev/peps/pep-3145/
Code: http://code.google.com/p/subprocdev/source/browse/subprocess.py

However, it seems like this could be accomplished with the tornado
ioloop, but I'm still wrapping my head around it and epoll.

Any suggestions?

Thanks in advance,

Neil

[Van]

Check out the autoreload.py file in the tornado folder.

[Russ Ryba]

On Nov 19, 3:54 pm, Van <thegoleff...@gmail.com> wrote:
> Check out the autoreload.py file in the tornado folder.
>
>

All Auto-reload does is close all current connections and restart the
server process. How does this help with the question of how to do non-
blocking sub processes?


I have to do this a lot in my system so I've looked at it a bit, but
not implemented anything using Tornado yet. I use twisted-parallels
right now and it works pretty well.

The interesting asynchronous part of tornado appears to be in tornado/
httpclient.py, but in appears the background processing is done by
pycurl.CurlMulti() which is a wrapper to a module written in C. I
didn't see anything easily usable for monitoring generic processes.

I started writing something in Tornado, but was redirected to other
projects. My plan was to make an @asychronous call that lauched a
process via os.process, get the pid, then loop until os.waitpid(pid,
os.WNOHANG) returned indicating that the process was complete. I am
currently upgrading my system from 2.5 to 2.6 so I don't have any
working code at the moment.

Other good sources are twisted process protocol, and twisted spread.
They work really well and you may be able to use them directly with
the tornado on twisted code base someone has created. Or keep trying
to roll your own. ;o)

Hope this helps Neil.

Regards,
Russ Ryba

[Neil]

Thanks, Russ. I will try that approach when I get the chance and see
how it goes.

What I have now is a little, one-file Tornado app--basically the one
posted above--running on a different port from my main app. When I
need to do image processing, I send a post request to it using the
asynchronous Tornado httpclient, just as one would to a third-party
web service. It seems to be working well enough, and has the added
benefit that if I needed to increase capacity, I could move it onto a
separate machine.

Thanks again,

Neil

[Van]

> All Auto-reload does is close all current connections and restart the
server process. How does this help with the question of how to do
non-
blocking sub processes?

It also "plugs" into IOLoop and polls every 500 ms to do a non-
blocking file-modification-date check - which is related to one of the
OP's original questions/thoughts.

Alternatively, you could also use a queuing system and schedule it out
to slave machines or processes. Lots of options.

[Don]

> I would like to do this asynchronously, so that the imaging processing
> does not block, but instead calls a callback when complete.

Neil:

10 days after you posted your question, an answer appears in a blog
here:

http://brianglass.wordpress.com/2009/11/29/asynchronous-shell-commands-with-tornado/

I haven't checked it out in detail but since I need this functionality
too, I will check it out soon.

- Don

[Neil]

Very cool. Thanks.

On Dec 22, 2:14 am, Don <don.bart...@gmail.com> wrote:
> > I would like to do this asynchronously, so that the imaging processing
> > does not block, but instead calls a callback when complete.
>
> Neil:
>
> 10 days after you posted your question, an answer appears in a blog
> here:
>

> http://brianglass.wordpress.com/2009/11/29/asynchronous-shell-command...