Python task/job queue implementation for pylons

[karik...@gmail.com]

We need schduler to do certain tasks basically downloading and parsing
html files & RSS from many other sites.

We would like to have a job scheduler with db backend to add and
remove the tasks using pylons.

Have you used any specific job queue with pylons? Could you please
recommend some?

Thanks & Regards,

Krish

[Hans Lellelid]

I would recommend taking a look at Apache ActiveMQ [1] as a message
broker. It's not written in Python (more on that below), but it's easy
to use and you can interface with it in Python. We use the STOMP
protocol w/ ActiveMQ from Pylons to push messages (usually
JSON-structured) onto worker queues or broadcast messages to topics that
our GUI clients subscribe to (to be notified of changes).

STOMP protocol has a few client implementations in python. There's
stomp.py [2] which we're using from Pylons right now (though we
simplified it to remove the "listener" parts). There's also stomper [3]
which seems to be better architected, in that the protocol is separate
from the transport. We ended up using Stomper with Twisted for our
worker daemons (which subscribe to the queues), since stomp.py does some
stuff under the hood with spawning off another thread that doesn't
necessarily play nice with other frameworks.

Anyway, there are other message brokers out there, but ActiveMQ was
*extremely* easy to just get up & running and supported the concept of
topics and queues. In default mode it will persist queues to files, but
can be configured to use a database. There's also MorbidQ (written in
Python, uses Twisted), but it was both difficult to get working and also
seems to hack around with message bodies such that it is not possible to
send binary data, etc. I also don't think it supported persisting the
messages to a database. In short, it seemed to be a somewhat
experimental implementation and didn't feel at all like something I'd
want to trust with anything important.

I'd be interested in learning the best way to manage socket connections
from within a pylons app. In our case, we're setting up & tearing down
the sockets with every request. This probably isn't optimum, but we
only open those connections somewhat rarely; also I'm honestly not sure
where the *right* place to put that would be. app_globals.py?
(Globals() is a stacked object proxy, right?)

Anyway, good luck in your search! (And let us know what you end up using.)

Cheers,
Hans

[1] http://activemq.apache.org/
[2] http://code.google.com/p/stomppy/
[3] http://code.google.com/p/stomper/

[karikrishni]

Thank you Hans.. I found celery which works with django.

Here is my post. http://groups.google.com/group/celery-users/browse_thread/thread/4d7bed4e381ab964/69f1238fae45944f

First, I will try porting celery to work with pylons and then try with
other solutions.

Regards,

Krish

On Sep 12, 5:28 pm, Hans Lellelid <h...@velum.net> wrote: