Alternatives to celery

[Isaac XXX]

Hi folks,

I need a solution to create cron-like jobs into my django projects. I've a server with several projects.

I've testes celery, and it seems to work well with only 1 project, but when I need to set up several projects with celery support, I can't figure how to do it (because daemonization of celeryd process, described here http://celery.readthedocs.org/en/latest/cookbook/daemonizing.html#daemonizing only describes how to start workers for a single process because this directive CELERYD_CHDIR="/opt/Myproject/").

Well, my question is the following:

Is there any good tutorial about deploying django with celery on several projects, sharing celery support? if not, is there any alternative viable to celery for django?

I've found APScheduler, that seems to be a good scheduler for python, but integration with django seems a bit tricky. Is it a good alternative?

I will be pleased to read all your feedback about it.

Thanks in advance,

Isaac

[Carlos Daniel Ruvalcaba Valenzuela]

Hi, so far I have been using one celery instance per project and seems
to work ok, I have setup scripts (fabric) to start and reload the
project celery server (via manage.py celeryd), some notes en backends:

* Shared redis instance does not seem to work well, unless you use
different db numbers.
* I have not tested multiple celerys with one rabbitmq but should be
possible with some routing configuration.
* DB backend works ok and is easy to setup per project (no extra setup
or daemon to run apart from celeryd), this backend is the least
performing.

If all you projects are sharing tasks you may even run celeryd on its
own and configure all apps to send, that tasks it works ok, I have one
celeryd instance accepting shared tasks from many apps using redis
backend (rabbitmq should work as well) and seems to scale ok.

If you are managing multiple celery instances you may want to give a
look at supervisord:

http://supervisord.org/

Regards,
Carlos Daniel Ruvalcaba Valenzuela

[Arun P]

@Isaac, How about gearman?

Gearman seems to support multiple projects through one gearman server.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
Digitally,
Arun Prabhakar

[Markus Gattol]

Have a look at https://github.com/dmgctrl/django-ztask It's based on ZeroMQ, you can schedule and background tasks in various ways, same machine or even across the network.

[Alec Taylor]

Looking for an alternative to celery? — Have you considered carrot?

:P

[Cherian Thomas]

Second Arun’s suggestion on gearman. Pretty solid too.

 

Regards,

Cherian

On Mon, Jan 16, 2012 at 3:52 PM, Alec Taylor <alec.t...@gmail.com> wrote:

Looking for an alternative to celery? — Have you considered carrot?

:P

--
You received this message because you are subscribed to the Google Groups "Django users" group.

[bryan hunt]

Gearman only supports a MySql backend for persistent queues - it
doesn't support a real message queue like Rabbitmq.

[Cherian Thomas]

Gearman supports sqlite. Gearman is for jobs, not just data. 

Regards,

Cherian

[Tom Evans]

On Mon, Jan 16, 2012 at 3:05 PM, bryan hunt <sentimen...@gmail.com> wrote:
> Gearman only supports a MySql backend for persistent queues - it
> doesn't support a real message queue like Rabbitmq.
>
>
>
> On Jan 16, 10:33 am, Cherian Thomas <cherian...@gmail.com> wrote:
>> Second Arun’s suggestion on gearman. Pretty solid too.

Gearman also would require a separate process for each project that
you have operating in it.

If you think about it, that is a bit of a non-requirement, since every
distributed task queue will require a daemon process to attach to it
to receive work, and if they are different projects, they will clearly
require separate processes.

With celery/rabbitmq, you can run as many django projects, with as
many celeryd instances per project, as you want. You don't need a 1:1
mapping between projects and celeryd instances, we run many celeryd
instances for a single project (across many boxes). Each project
should use a separate vhost on rabbitmq, which you configure with
settings.BROKER_VHOST.

Cheers

Tom