RabbitMQ & Python

[Stefan Arentz]

Hey, first post :)

I'm planning on using RabbitMQ to distribute some tasks for Firefox Home. I just want to check if we have some standard libraries or setup that we use for that.

If I understand correctly the right Python lib to use with RabbitmQ is Celery?

Anything I need to know?

Also, do any of you know a library in Python that will implement all the right things to behave as a daemon? And also to maintain a bunch of child processes?

Ideally I start some master process that I can configure with "start N client processes, keep an eye on those and if they crash of hang, restart them". I think Twisted comes with something like that, but I don't want to do a whole Twisted thing.

I've looked at multiprocessing, but I prefer to keep things simple and simply have a bunch of child processes that each maintain their own connection to RabbitMQ. So the role of the master would be just to start/stop them. And to keep an eye on their status.

 S.

[Erik Rose]

> If I understand correctly the right Python lib to use with RabbitmQ is Celery?
>
> Anything I need to know?
>
> Also, do any of you know a library in Python that will implement all the right things to behave as a daemon? And also to maintain a bunch of child processes?

[snip]

> I've looked at multiprocessing, but I prefer to keep things simple and simply have a bunch of child processes that each maintain their own connection to RabbitMQ.

Celery itself comes with a daemon, celeryd, which sites around executing deferred tasks. It autospawns processes or threads or greenlets or whatever you choose. Was that all you needed, or do you need to also have your own process pool for something else?

> So the role of the master would be just to start/stop them. And to keep an eye on their status.

supervisord is pretty popular, if you have something that isn't capable of taking care of itself.

Cheers,
Erik

[Stefan Arentz]

On 2011-05-03, at 4:39 PM, Erik Rose wrote:

>> If I understand correctly the right Python lib to use with RabbitmQ is Celery?
>>
>> Anything I need to know?
>>
>> Also, do any of you know a library in Python that will implement all the right things to behave as a daemon? And also to maintain a bunch of child processes?
> [snip]
>> I've looked at multiprocessing, but I prefer to keep things simple and simply have a bunch of child processes that each maintain their own connection to RabbitMQ.
>
> Celery itself comes with a daemon, celeryd, which sites around executing deferred tasks. It autospawns processes or threads or greenlets or whatever you choose. Was that all you needed, or do you need to also have your own process pool for something else?

That sounds exactly like what I need. Thanks for hint, I had not seen that tool yet!

S.

[bear]

On Tue, May 3, 2011 at 16:26, Stefan Arentz <sar...@mozilla.com> wrote:
> Hey, first post :)
> I'm planning on using RabbitMQ to distribute some tasks for Firefox Home. I
> just want to check if we have some standard libraries or setup that we use
> for that.
> If I understand correctly the right Python lib to use with RabbitmQ is
> Celery?
> Anything I need to know?

The library I think is actually Carrot (or the one I use a lot -
Kombu) - I would look at what Christian uses for Pulse to get some
idea of what lib and toolset are used:

https://hg.mozilla.org/users/clegnitto_mozilla.com/mozillapulse/

>
> Also, do any of you know a library in Python that will implement all the
> right things to behave as a daemon? And also to maintain a bunch of child
> processes?

I use a combination of Python's Subprocess module and a daemon.py
script that can be found in many flavours on PyPI.

>
> Ideally I start some master process that I can configure with "start N
> client processes, keep an eye on those and if they crash of hang, restart
> them". I think Twisted comes with something like that, but I don't want to
> do a whole Twisted thing.
>
> I've looked at multiprocessing, but I prefer to keep things simple and
> simply have a bunch of child processes that each maintain their own
> connection to RabbitMQ. So the role of the master would be just to
> start/stop them. And to keep an eye on their status.
>  S.
>

That's the rub about doing this kind of coding - it's not simple even
at the simple side of the scale :) Twisted and multiprocessing all do
things to keep the messy details at arms length but when you get into
edge cases and error handling you will appreciate their flexibility

--
Bear

be...@xmpp.org (email)
bea...@gmail.com (xmpp, email)
be...@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111  D770 9331 E822 40B3 CD29

[Shane Caraveo]

celery is one lib you can use, the same author also has kombu (formerly carrot).  If I recall correctly, celery put more of an rpc type layer on top of amqp (or maybe that was a different kind of rabbit food).  If you want to go low-level, the python amqp library is pretty basic.  A couple years ago I worked a bit on https://github.com/gmr/rejected but I think celery is more advanced at this time. 

Shane