Alternative to celery

615 views
Skip to first unread message

guettli

unread,
Apr 20, 2017, 5:16:37 AM4/20/17
to Django users
Some years ago I used celery for a project.

Today I face roughly the same use case again.

The last time I used it, celery felt to heavy weighted, too complicated.

I found this alternative: http://python-rq.org/

There are two things which speak agains python-rq:

 - redis is a cache. I need a persistent queue. If a power-failure happens, no job must get lost.

 - AFAIK python-rq uses python pickle to serialize data. I want a language neutral data exchange format.

At the moment my favourite protocol is gRPC (protocol-buffers). I never used it up to now, but this would be a nice use case.

Maybe I am missing the right terms. What does celery implement? According to wikipedia "asynchronous task queue". But
my favorite search engine could not reveal a gRPC based server implemented in Python ....

What do you think?


Andréas Kühne

unread,
Apr 20, 2017, 5:26:07 AM4/20/17
to django...@googlegroups.com
I don't know anything about the python-rq stuff - however redis can be configured to save to disk (see https://redis.io/topics/persistence) - so you wouldn't loose anything on a power-failure (if configured correctly). Redis is so much more than a cache nowadays.

Regards,

Andréas

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d43d2cef-f1a1-418d-9d62-ecce85961fbb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Avraham Serour

unread,
Apr 20, 2017, 8:26:45 AM4/20/17
to django-users

marcin....@gmail.com

unread,
May 11, 2017, 5:00:34 AM5/11/17
to Django users
RQ is simple, stable and reliable. Redis is also a persistent storage, but you should read about this and use the best solution for your case: https://redis.io/topics/persistence   
You may also consider deploying a fault-tolerant Redis cluster (min. 3 nodes AFAIR)

RQ has different architecture than Celery. It is based on fork(), which is just slower. So you should ask yourself how many tasks per sec you need to handle. RQ may not be enough for all use cases.
Some things aren't OOTB, like repeating failed tasks (you should handle it manually, i.e. by using own worker class), nor  tasks scheduling (you should use rq-scheduler). 

RQ uses pickle, but there is a feature request about handling json and other serializers - https://github.com/nvie/rq/issues/369 


Reply all
Reply to author
Forward
0 new messages