Django with rabbit (but without async celery) - can be done?

[Alon Nisser]

I need to implement a quite simple Django server that server some http requests and listens to a rabbitmq message queue that streams information into the Django app (that should be written to the db). the data must be written to the db in a synchronized order , So I can't use the obvious celery/rabbit configuration.  I was told that there is no way to do this in the same Django project. since Django would listen to http requests on It's process. and It can't handle another process to listen for Rabbit - forcing me to to add Another python/django project for the rabbit/db writes part - working with the same models The http bound django project works with.. You can smell the trouble with this config from here. .. Any Ideas how to solve this?

Thanks!

[Jonathan Baker]

Could you write a RESTful API in your Django project, and then issue POST requests to that from a middleman script that the mq could communicate with?

JDB

On Tue, Feb 25, 2014 at 8:28 AM, Alon Nisser <alon...@gmail.com> wrote:

I need to implement a quite simple Django server that server some http requests and listens to a rabbitmq message queue that streams information into the Django app (that should be written to the db). the data must be written to the db in a synchronized order , So I can't use the obvious celery/rabbit configuration.  I was told that there is no way to do this in the same Django project. since Django would listen to http requests on It's process. and It can't handle another process to listen for Rabbit - forcing me to to add Another python/django project for the rabbit/db writes part - working with the same models The http bound django project works with.. You can smell the trouble with this config from here. .. Any Ideas how to solve this?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/794bf8ca-a15b-40da-97c2-b3a416a35fe0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Carlos Daniel Ruvalcaba Valenzuela]

I'm still not sure of what you need, so there is something (a process) sending a message through rabbitmq with data that needs to be written to the DB, now, this needs to be synchronized, sync with what? an http request? ordered execution of writes?

Regards,

Carlos Ruvalcaba

On Tue, Feb 25, 2014 at 8:28 AM, Alon Nisser <alon...@gmail.com> wrote:

I need to implement a quite simple Django server that server some http requests and listens to a rabbitmq message queue that streams information into the Django app (that should be written to the db). the data must be written to the db in a synchronized order , So I can't use the obvious celery/rabbit configuration.  I was told that there is no way to do this in the same Django project. since Django would listen to http requests on It's process. and It can't handle another process to listen for Rabbit - forcing me to to add Another python/django project for the rabbit/db writes part - working with the same models The http bound django project works with.. You can smell the trouble with this config from here. .. Any Ideas how to solve this?

Thanks!

--

[Alon Nisser]

thanks for your response: to clarify:
ordered execution of writes
and the sending process isn't python (so I can't use eager celery for that)

[Alon Nisser]

I could do that (and would fall back to this if needed) but it's suboptimal to say the least. celery does this somehow (run inside djano context but from without the django process. listen to rabbitmq, etc) so I guess it can be done. Looking for an Idea how

[Camilo Torres]

On Tuesday, February 25, 2014 10:58:47 AM UTC-4:30, Alon Nisser wrote:

I need to implement a quite simple Django server that server some http requests and listens to a rabbitmq message queue that streams information into the Django app (that should be written to the db). the data must be written to the db in a synchronized order , So I can't use the obvious celery/rabbit configuration.  I was told that there is no way to do this in the same Django project. since Django would listen to http requests on It's process. and It can't handle another process to listen for Rabbit - forcing me to to add Another python/django project for the rabbit/db writes part - working with the same models The http bound django project works with.. You can smell the trouble with this config from here. .. Any Ideas how to solve this?

Hello,

What you has been told seems to be the right way of solving this. You should have another python (not Django) application listening to the queue to process incoming messages. You can use Django models to insert the data into the database. That queue does not really streams data to _the application_ but to _the database_ via a third running application. I don't have any smell here, only curious of what 'written to the db in a _synchronized order_' means ;)

If you are going to receive a stream of data, split in messages in a queue, then I assume you may need to store the data in the DB in an ordered fashion, in a way that can then easily be retrieved in the right order or sequence to play or use the data. That is a different problem with many solutions with the tools you have, and has nothing to do with Django, but programming in general :) You can have an order field in your models. You can cache the data in your queue listener application and write to the DB only portions that make sense according to order and keep the data that is out of order in memory until it can be written.

Regards,

Camilo