Daemonizing Celery when using django-celery

[Michael Lissner]

I'm new to Celery and utterly confused about how to daemonize Celery when using django-celery. From what I can tell, there are four things named celeryd:

• the command called by manage.py
• the script that can be placed in /etc/init.d/celeryd (though it's not installed there by the installer)
• the config file in /etc/default/celeryd, that is read by the init script
• the celeryd program (I think)

There's also three places that daemon configurations can apparently live:

• /etc/init.d/celeryd
  
• /etc/default/celeryd
• myApp/settings.py

I've read and re-read the daemonizing instructions: http://docs.celeryq.org/en/latest/cookbook/daemonizing.html, but I'm still baffled.

Can we take a look at the documentation here and sort out exactly what people need to do to daemonize?

My impression is that I need to:
1. Download the daemon script since it's not included in the installation: https://github.com/ask/celery/blob/master/contrib/generic-init.d/celeryd
2. Put it in /etc/init.d/celeryd
3. Make a file in /etc/default/celeryd, and put the variables here into it: http://docs.celeryq.org/en/latest/cookbook/daemonizing.html#example-django-configuration
4. Start the script

But is that really the right way to do this? I don't want to have settings in two places, and I want to use the correct commands to start django-celery.

Help?

[Michael Lissner]

I didn't get a response here, but for others that might also need help on this, I sorted out what deploying Celery really means in a production environment. In the end, I ended up with an init script at /etc/init.d/celeryd, a settings file inside my django project, and a log file inside my django project.

I realized that the documentation doesn't discuss log rotation for Celery. Maybe it's obvious, but it might be something to add, so others don't have logs piling up.

To get this all running, I had to do the following as root:

        # I prefer init scripts to be checked into version control, so
        # they live in my django project and need to be linked.
        ln -s myproject/init-scripts/celeryd /etc/init.d/celeryd

        # The init script needs to be executable:
        chmod +x /etc/init.d/celeryd
       
        # Make an unprivileged, non-password-enabled user and group to run celery
        useradd celery
       
        # make a spot for the logs and the pid files
        mkdir /var/log/celery
        mkdir /var/run/celery
        chown celery:celery /var/log/celery
        chown celery:celery /var/run/celery
       
        # I also keep log scripts in version control, so they need to get linked as well.
        ln -s myproject/log-scripts/celery /etc/logrotate.d/celery

All in all, not too difficult, but it took me some sorting out to make sure I understood everything. The bottom line is that the init script imports from its own settings file, which in turn imports from the django settings file. I'm still confused why there are two places for settings - curious if others know why.

[Dan Julius]

Great!

Any chance you can post your logrotate config file?

Also, any ideas about how to implement centralized logging?

Dan

--
You received this message because you are subscribed to the Google Groups "celery-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/7MUQCHlJ0kEJ.

To post to this group, send email to celery...@googlegroups.com.
To unsubscribe from this group, send email to celery-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.