Celery daemonization best practice?

[ben.e...@gmail.com]

Hi Ed (and others!),

I've set up celery as a daemon which seems to be working well. One query I have though is how certain folder permissions should be set for a production set up using Apache. As per the documentation the static and media directories are owned by www-data. However, following the celery documentation: http://docs.celeryproject.org/en/latest/userguide/daemonizing.html#daemonizing it recommends using an unprivileged user for running the celery worker process. This worker will then need read/write access to /media/celery and /media/exports. In this case I made a "celery" user for this purpose. However, it appears www-root will also need read/write access to these directories e.g. for deleting exported files. The simple answer might be to make celery run as www-data, but this user has no shell so can't execute the python -m celery worker command.

Any thoughts on the best way to set up other than giving read/write access to all users to these directories?

Thanks,

Ben

[Ed McDonagh]

Hi Ben

I start the celery daemon as the www-data user - why can't you do this? 

Interactively, I do this by becoming www-data:

sudo -s -u www-data

Does that help?

Ed

--
You received this message because you are subscribed to the Google Groups "OpenREM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrem+unsubscribe@googlegroups.com.
To post to this group, send an email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/openrem.
For more options, visit https://groups.google.com/d/optout.

[ben.e...@gmail.com]

If in /etc/default/celeryd I set:

CELERYD_USER="www-data"
CELERYD_GROUP="www-data"

The celeryd service fails to start when I run:

sudo service celeryd start

I get the following error:

Job for celeryd.service failed because the control process exited with error code. See "systemctl status celeryd.service
" and "journalctl -xe" for details.

Getting the service status (showing the key lines):

Apr 27 13:33:23 rtubuntu su[12448]: pam_unix(su:session): session opened for user www-data by (uid=0
Apr 27 13:33:23 rtubuntu celeryd[12428]: This account is currently not available.
Apr 27 13:33:23 rtubuntu systemd[1]: celeryd.service: Control process exited, code=exited status=1

From researching I think this is from the case that www-data has no shell, from:

more /etc/passwd | grep www-data

Returns:

www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin

I think running sudo -s -u www-data is different than running as a service as this uses the current shell with the rights of the chosen user.

Do you see what I mean? Have you added a shell to www-data for running a celeryd service?

Thanks,

Ben

On Thursday, 27 April 2017 12:30:03 UTC+1, Ed McDonagh wrote:
> Hi Ben
>
>
> I start the celery daemon as the www-data user - why can't you do this? 
>
>
> Interactively, I do this by becoming www-data:
> sudo -s -u www-data
>
>
>
> Does that help?
>
>
> Ed
>
>
> On 27 April 2017 at 08:49, Ben wrote:
> Hi Ed (and others!),
>
>
>
> I've set up celery as a daemon which seems to be working well. One query I have though is how certain folder permissions should be set for a production set up using Apache. As per the documentation the static and media directories are owned by www-data. However, following the celery documentation: http://docs.celeryproject.org/en/latest/userguide/daemonizing.html#daemonizing it recommends using an unprivileged user for running the celery worker process. This worker will then need read/write access to /media/celery and /media/exports. In this case I made a "celery" user for this purpose. However, it appears www-root will also need read/write access to these directories e.g. for deleting exported files. The simple answer might be to make celery run as www-data, but this user has no shell so can't execute the python -m celery worker command.
>
>
>
> Any thoughts on the best way to set up other than giving read/write access to all users to these directories?
>
>
>
> Thanks,
>
>
>
> Ben
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "OpenREM" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to openrem+u...@googlegroups.com.

[Ed McDonagh]

I do see what you mean. 

Confession time: I've never gotten around to daemonising celery, so I've not hit your issue.

However, I do run Gunicorn  using  supervisor to daeomonise it - I followed the guide here: http://www.robgolding.com/blog/2011/11/12/django-in-production-part-1---the-stack/

It might be that this might work as supervisor is the daemon, which forks off Celery?

There are some example configs in https://github.com/celery/celery/tree/master/extra/supervisord

[ben.e...@gmail.com]

I guess you're running behind nginx? If on ubuntu it looks like nginx installs www-data with a shell: https://serverfault.com/questions/559290/does-www-data-user-need-a-real-shell

Is that the case?

Otherwise it might be because supervisor runs sub processes with set rights from root for services (rather than creating a separate process). So it might be inheriting root's shell.

[Ed McDonagh]

I am running behind nginx,and www-data on that machine does seem to to have a shell. Interestingly, on an Ubuntu 16.04 I have just installed with nginx, it has /usr/sbin/nologin as the shell.

However, I haven't run Celery as a daemon or with supervisord - I just run it manually each time with multi. I will daemonise it when I get around to building a new server for it at some point soon though!

Ed

[Tim de Wit]

I'm running on Ubuntu 16.04 and got everything to work as follows:   

- add group to /etc/group and add relevant members to it, e.g.   

openrem:x:1007:www-data,_conquest,celery

- output of "ls -al /var/openrem/media/"

drwxrwxr-x 2 celery   celery      4096 Apr 14 18:49 celery

drwxrwxr-x 3 www-data celery      4096 Apr 14 18:50 exports

-rw-rw-r-- 1 www-data openrem  9179264 Apr 20 11:42 openrem.log

-rw-rw-r-- 1 www-data openrem  6956569 Apr 19 09:41 openrem_qr.log

-rw-rw-r-- 1 www-data openrem        0 Sep 23  2016 openrem_store.log

drwxrwxrwx 4 www-data openrem     4096 Mar  1 16:56 skin_maps

- celeryd is running as user celery.

regards, Tim

[Tim de Wit]

Something else I remembered... initially I also didn't get the ubuntu version of celeryd to work (the one installed through apt). When starting the service it reported "successfully started" but the daemons were not actually running. After lots of trial-and-error I finally removed it without purging the configs (leaving only /etc/default/celeryd and the startup scripts, since they are very convenient).

Next I installed celeryd through pip (in /usr/local/bin/celery). Make sure /usr/local/bin is in your path; which it probably already is. That should do the trick (also change permissions according to my post above).

[Ed McDonagh]

Thanks a lot for this Tim - I shall have to rebuild my server and set this up properly and document it so we can all benefit! I shall certainly be coming back to this email chain when I do!

On 3 May 2017 at 14:18, Tim de Wit <t.c....@gmail.com> wrote:

Something else I remembered... initially I also didn't get the ubuntu version of celeryd to work (the one installed through apt). When starting the service it reported "successfully started" but the daemons were not actually running. After lots of trial-and-error I finally removed it without purging the configs (leaving only /etc/default/celeryd and the startup scripts, since they are very convenient).

Next I installed celeryd through pip (in /usr/local/bin/celery). Make sure /usr/local/bin is in your path; which it probably already is. That should do the trick (also change permissions according to my post above).

--

You received this message because you are subscribed to the Google Groups "OpenREM" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrem+unsubscribe@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

[ben.e...@gmail.com]

Thanks Tim that's exactly what I was after! Applied it to the setup here and it works well.

Ed - Yeah I think that would be good for the documentation, the setup isn't really too hard but there's a few gotchas along the way which could be avoided as they aren't covered in the documentation of the individual apps like celery.

Thank you both for your help,

Ben

On Wednesday, 3 May 2017 16:04:43 UTC+1, Ed McDonagh wrote:
> Thanks a lot for this Tim - I shall have to rebuild my server and set this up properly and document it so we can all benefit! I shall certainly be coming back to this email chain when I do!
>
>
> On 3 May 2017 at 14:18, Tim de Wit <t.c....@gmail.com> wrote:
>
> Something else I remembered... initially I also didn't get the ubuntu version of celeryd to work (the one installed through apt). When starting the service it reported "successfully started" but the daemons were not actually running. After lots of trial-and-error I finally removed it without purging the configs (leaving only /etc/default/celeryd and the startup scripts, since they are very convenient).
>
> Next I installed celeryd through pip (in /usr/local/bin/celery). Make sure /usr/local/bin is in your path; which it probably already is. That should do the trick (also change permissions according to my post above).
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "OpenREM" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to openrem+u...@googlegroups.com.