Is there any good tutorial on how to daemonize celery?

[Manu]

Hi all,

I have had no success so far with daemonizing celery on my server. I raised two stack overflow questions and have received only a few answers. Could someone please point me to a solid tutorial or blog post on how to make it work. I'm using celery with django in a virtual environment.

Btw, the links to those two questions

1. http://stackoverflow.com/questions/16896686/daemonizing-celery-process-celeryd-multi-not-found

2. http://stackoverflow.com/questions/16915034/is-there-a-simple-manage-py-command-that-i-run-with-django-celery-to-daemonize-c

Thanks in advance for your time! 

Regards,
Manu

[Vineet Naik]

Hi Manu, 

Have you seen the "Running the worker as a daemon" section of celery docs - http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html

It also includes an example with Django and virtualenv. 

Regards,

Vineet

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[manu...@gmail.com]

Yeah Vineet,

I've gone through that. I mentioned it in the first question on SO. You will even see my settings there.

Regards,

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/F0dH6WdwFBA/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Manu Ganji,
Mobile: +91-8500719336.

[Vineet Naik]

A few months back, as far as I remember, I had used the same examples and got it working without much trouble. 

I am only guessing here but are you sure that celery and django-celery both are installed in the same virtualenv and not globally? 

Vineet Naik

[manu...@gmail.com]

Yeah Vineet, both of them are installed in the same virtualenv.

[Mike]

Have you tried supervisord?  I installed supervisord from my linux distro on my server and put a config file in the conf.d directory.  Its working fine.

[manu...@gmail.com]

Thanks for the reply Mike. I've been trying to setup supervisord for 4 hours now. Just for this celery. I don't know where I'm going wrong. Looks like nobody else is having a problem.

--

You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/F0dH6WdwFBA/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Mike]

Thats why I installed it using the package provided in my Ubuntu linux server.  Then I used the template .conf file (I think from the celery documentation) and customized it for my project.  I did have some trouble to get it running the first time but I think there is a delay between when you restart supervisord and when the daemons it controls restart.  In fact, I don't use the 'restart' command to restart supervisord, I use to 'stop' and then confirm that the celery processes have stopped by using 'ps'.  then I use '/etc/init.d/supervisord start' to start it up again.

Is your worker running fine when you execute:

    python manage.py celery worker

?

Are you using a virtualenv?

[manu...@gmail.com]

Mike,

Please find my replies below.

On 5 June 2013 20:27, Mike <mike...@gmail.com> wrote:

Thats why I installed it using the package provided in my Ubuntu linux server.  Then I used the template .conf file (I think from the celery documentation) and customized it for my project.  I did have some trouble to get it running the first time but I think there is a delay between when you restart supervisord and when the daemons it controls restart.  In fact, I don't use the 'restart' command to restart supervisord, I use to 'stop' and then confirm that the celery processes have stopped by using 'ps'.  then I use '/etc/init.d/supervisord start' to start it up again.

Ok, I'll look into that. 

Is your worker running fine when you execute:

    python manage.py celery worker

?

 I'm getting an error that it's unable to import some of the files in my django project. This, even when I run after activating my virtual environment.

Are you using a virtualenv?

Yeah as I said above, I'm using a virtualenv. 

 

--

Manu Ganji,

[Mike]

In your SO question you say that you have set up  /etc/default/celeryd but you don't need this for running celery in a Django project.  In your settings.py file you should have this:

import djcelery

djcelery.setup_loader()

BROKER_URL = 'amqp://guest:guest@localhost:5672//'

(BROKER_URL should be set to whatever broker you're using)

your INSTALLED_APPS should also have 'djcelery'

get the command 'python manage.py celery worker' working first, before you mess around with supervisor or init scripts.  If you can't run a worker on the command line it probably won't work in a daemon either.

 

[manu...@gmail.com]

Mike, 

Thanks for the last reply! That gives some clarity. So I put the following settings in my settings file.

_________________________________________________________________________________

BROKER_URL = "amqp://<usrname>:<passwd>@localhost:5672/<vhost>"

CELERY_RESULT_BACKEND = "database"

CELERY_RESULT_DBURI = "postgresql://"+get_env_setting('DB_USER')+":"+get_env_setting('DB_PASSWD')+"@localhost/"+get_env_setting('DB_NAME') 

# CELERY_IMPORTS = ('tasks','users.tasks',)

# put these two lines at the very bottom of the settings file

import djcelery

djcelery.setup_loader()

from celery import Celery

celery = Celery(broker=BROKER_URL)

___________________________________________________________________________________

Now I tried to run it with this manage.py command

python manage.py celery worker --app=project.settings.production

It's starting up succesfully but I'm seeing several KeyErrors that celery received an unregistered task. I tried inserting the solutions I found when I researched. For example, adding a setting CELERY_IMPORTS hasn't solved it. Restarting celery also didn't help. Any idea why this could be happening?

Regards,

 

--

You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/F0dH6WdwFBA/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Michael Thon]

> CELERY_RESULT_BACKEND = "database"
> CELERY_RESULT_DBURI = "postgresql://"+get_env_setting('DB_USER')+":"+get_env_setting('DB_PASSWD')+"@localhost/"+get_env_setting('DB_NAME')

> Now I tried to run it with this manage.py command
>
> python manage.py celery worker --app=project.settings.production
>
> It's starting up succesfully but I'm seeing several KeyErrors that celery received an unregistered task. I tried inserting the solutions I found when I researched. For example, adding a setting CELERY_IMPORTS hasn't solved it. Restarting celery also didn't help. Any idea why this could be happening?
>

What happens if you don't set CELERY_RESULT_DBURI and CELERY_RESULT_BACKEND ?

[manu...@gmail.com]

Mike, with those settings removed I'm still getting the same errors.

And I'm now trying to do everything from scratch in a new server without virtualenv. Could somebody please help with the last issue I need to fix? I raised a stack overflow question here - http://stackoverflow.com/questions/16970379/environment-variables-with-supervisor

I'm just amazed by the community support here. Thanks a lot, everyone!

Regards,