Integrating RabbitMQ with Django

[Erol Merdanović]

Hi all

I'm using Django to develop a service. Service connects to RabbitMQ and receives events that are translated to creating/updating/deleting of the database records. 

What is the best way to run RabbitMQ consumer with Django? I'm using Pika library and I run consumer as a management command (python manage.py listen_to_changes). Is this a good solution? I don't see it so. I'm running Django as WSGI so I presume there are no start/stop Django events that I could plugin in to start/stop consumer.

I'm aware of the Celery and how it nicely integrates into Django, but for me it's not an option.

What do you suggest? How to run RabbitMQ with Django (stability is the main objective).

[Vijay Khemlani]

Celery would be the usual choice for this, why is it not an option? Seems far easier and more stable than trying to use a webserver as a queue worker.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/e2b0cfd3-f33f-485c-9497-98fd281c9cd2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Erol Merdanović]

Hello Vijay

The differences are

1. Celery message and our message has different format

2. Celery uses custom queues

Is this correct?

[Sadaf Noor]

Instead you can try using celery. For demonstration purpose I have tried an image processing app using rabbitmq, celery with flask. It worked great. It can be found at my blog ( http://www.sadafnoor.com/blog/rabbitmq-celery-demonstration-using-image-processing-app-on-flask/ ).

--

[Vijay Khemlani]

Is the origin of the messages something different than your Django application? Seems overkill to have a custom message format.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/249b4f0e-b817-4a1f-8ad0-4dabc64d554d%40googlegroups.com.

[Erol Merdanović]

@Vijay

Yes, source of the messages is outside of the Django application. I could use some transformers, but again, it's extra moving part.

@Sadaf

Ok, thank you for the blog. I will check it out. But again, Celery is great but I'm not really sure it solves my problem.

[Felipe Arenhardt Tomaz]

For manual manipulation recommend Pika[1]

[1] https://pypi.python.org/pypi/pika

--

Felipe Arenhardt Tomaz

Systems Developer

Curitiba - Paraná

www.felipetomaz.com

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a1b78c97-eddb-4330-96e9-b442fe6609bd%40googlegroups.com.

[Tom Evans]

On Wed, Sep 16, 2015 at 2:34 PM, Erol Merdanović <zase...@gmail.com> wrote:
> Hello Vijay
>
> The differences are
> 1. Celery message and our message has different format
> 2. Celery uses custom queues
>
> Is this correct?
>

Yep. Celery is a distributed task system which can use AMQP as
transport, but by default it just uses simple direct exchanges. If you
are using AMQP for something more specialized, like a fanout or topic
exchange, you can configure this for celery. It is still all about
running tasks though.

This page in the celery docs describes how celery uses AMQP, and
briefly discusses the format:

http://celery.readthedocs.org/en/latest/userguide/routing.html

Implementing your worker processes as a management command is
sensible, I would use something like supervisord or circusd to manage
the processes. You should integrate restarting the workers in to the
deployment scripts that update your web server code.

Cheers

Tom

[Erol Merdanović]

@Tom

Yes, it was also thinking about using supervisord. But I'm interested if there is anything else that nicely integrates into Django.

@Felipe

Yes, I'm using pika. Afaik there are no decent tutorials how to implement consumers into Django.

[Felipe Arenhardt Tomaz]

I create and use this class to manipulate the queues

https://gist.github.com/Arenhardt/8ac7e7667134c28eceed

--

Felipe Arenhardt Tomaz

Systems Developer

Curitiba - Paraná

www.felipetomaz.com

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/ec65ffec-0b14-4a17-a15f-6899a3575cca%40googlegroups.com.

[Erol Merdanović]

@Felipe

Yes, I use very similar approach. Thank you for the example.

I found a great project https://github.com/rfk/django-supervisor that enables me to run Django commands under supervisor. I will test it and let you know how it goes.

[Felipe Arenhardt Tomaz]

I don't like to put the commands on supervisor. You can create an init script.

--

Felipe Arenhardt Tomaz

Systems Developer

Curitiba - Paraná

www.felipetomaz.com

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/4907005c-5272-4f0a-8048-749874f4d755%40googlegroups.com.

[Erol Merdanović]

@Felipe

Can you show an example what you mean by init script?

[Felipe Arenhardt Tomaz]

I update my gist and put 2 other files [1]

An init script (/etc/init.d/script) and an Django Command.

[1] https://gist.github.com/Arenhardt/8ac7e7667134c28eceed

--

Felipe Arenhardt Tomaz

Systems Developer

Curitiba - Paraná

www.felipetomaz.com

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/0155f46c-ff45-48aa-8c5a-97da1558c5ef%40googlegroups.com.