How can I serve a (never ending) system call with Tornado

[Tzury Bar Yochay]

For instance, suppose I have this code:

def dump():
    tcpdump = subprocess.Popen("tcpdump -nli any", 
        stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)

    outputfile = tcpdump.stdout
    for line in outputfile:
        print line,

How can I serve the output of such to the browser? Since there is no stopping point, I have no idea where to hook with the polling loop. More than that, as print line works (I see lines dumped on the terminal), browser do not get the very same lines, see below:

class TCPDumpHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("<form method='post' action='/log'><input type='submit'></form>")

@tornado.web.asynchronous
def post(self):
    tcpdump = subprocess.Popen("tcpdump -nli any",
            stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
    outputfile = tcpdump.stdout

    for line in outputfile:
        print line,
        self.write(line)

    self.finish()

[Phil Plante]

You might try adding a self.flush() after the self.write() call.

Another thing to consider is using a http long polling method via client-side javascript.  But you should be able to at least get some data sent along using flush()

[Ben Darnell]

Calling flush() is necessary but not sufficient - flushed output still won't actually be sent if the IOLoop is blocked reading from the subprocess's pipe.  You need to either move the readline loop to a separate thread (probably the simplest approach) or make it asynchronous.  Making it asynchronous would require using fcntl to set the O_NONBLOCK flag, adding a handler to the IOLoop to get notified when there is new data, and using read() instead of readline() (see http://stackoverflow.com/questions/375427/non-blocking-read-on-a-stream-in-python).  It shouldn't be too hard to adapt IOStream to work with pipes in addition to sockets.

-Ben

[Tzury Bar Yochay]

On Wed, Feb 23, 2011 at 9:45 PM, Ben Darnell <b...@bendarnell.com> wrote:

Calling flush() is necessary but not sufficient - flushed output still won't actually be sent if the IOLoop is blocked reading from the subprocess's pipe.  You need to either move the readline loop to a separate thread (probably the simplest approach) or make it asynchronous.  Making it asynchronous would require using fcntl to set the O_NONBLOCK flag, adding a handler to the IOLoop to get notified when there is new data, and using read() instead of readline() (see http://stackoverflow.com/questions/375427/non-blocking-read-on-a-stream-in-python).  It shouldn't be too hard to adapt IOStream to work with pipes in addition to sockets.

@Ben,

You are absolutely right about the flush in-effectiveness. It was there actually and removed after I discovered that it is not helping.

Anyway, I did what I was not planning to do and typed in my editor the following statement:

from threading import Thread ...

Yep. it works like a charm, though I would have been happier getting it done without an extra thread, but I can't fancy that this time.

many thanks,

tzury