long poll / comet

[Bill Moseley]

I have a web app (Catalyst) that processes uploaded files.   It's a multi-stage process where a file is uploaded, analyzed in the background, then requires additional user input. 

Currently what happens after the upload is the web client (the browser) does normal HTTP requests to poll for when analysis is done.   It's polling the main web app, which is behind Apache and is not a great use of limited resources.  If the analysis engine gets bogged down I can end up with a large number of clients polling -- which can compound the original problem.

I'd prefer to do this with some push operation -- e.g. long pool to an event-based web server that's listening on a per-user message queue.  There's many other types of events I'd like to notify the web client about asynchronously, too, so I'd like to look at a more generic message-pushing system.   I'd want to support a "large" number of concurrent clients -- maybe 5k to 10k -- or at least in a scalable way.

Is Twiggy an appropriate option as a server?   I briefly tried to run the example chat client but it didn't work for me out of the box (using Safari and Chrome).  I wonder if this might be a related this RT ticket, too.

127.0.0.1 - - [08/Jul/2012:10:51:16 -0700] "GET /chat/room HTTP/1.1" 200 3770 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2"

err: Unexpected end-of-file at chat.psgi line 66, <> line 9.

Or would PocketIO be a better choice?

I'm just starting to look at this, so please excuse my poking around in the dark.   Any pointers (and pointers to examples) would be very welcome.

Thanks,

--
Bill Moseley
mos...@hank.org

[Steve McDonald]

Hi Bill,

I'm no expert here, but I may have done a little more poking around in
the dark than you ;-)

PocketIO (and socket.io the node.js project it's based off of) mostly
handle the browser stuff for you in a way that works for most
browsers. I've not used it in production, but it looks like an awesome
tool that will save you a lot of effort.

That said, it still needs a web server to run under, one that
implements the psgi nonblocking api. Last time I looked at the
PocketIO docs they suggested Twiggy (or at least used it in an
example), but other servers should work… Feersum comes to mind, I
think there are a couple of others that would work too.

Getting your head around event based programming if you haven't done
it before is probably the biggest challenge here.

I can't help with your specific error… though you might start by
trying to implement the example in the docs for the Tatsumaki module,
it's a more complete example than what's in the Twiggy docs alone.

Regards,
Steve

--
Steven McDonald
Berkeley Electronic Press
smcd...@bepress.com

http://www.bepress.com/