Help setting up systemd for nemo

[Kevin McPeak]

Dear NEMO users,

Here at LSU we have NEMO running on a CentOS 7 server. Currently we have to manually start nemo if the server goes down. I would like to setup nemo and other dependencies, e.g. gunicorn, Nginx to automatically restart with systemd in the case of a power outage, etc.. Has anyone on the list configured systemd to manage nemo? If so, your input would be greatly appreciated. Thanks!

Regards,

Kevin

[Dylan Klomparens]

Hi Kevin,

There is a convenience script you can use to do this. NEMO comes with an "entry point script" which helps. (This advice is pretty much only applicable when NEMO is installed from the Python Package Index using pip). After you've run `pip install nemo`, you can run the `nemo` binary file. Here is the code for this file: https://github.com/usnistgov/NEMO/blob/master/NEMO/provisioning.py#L210

The command you're looking for is nemo install_systemd_service. I'll admit, this code is untested, and was around before we had fully committed to Docker (at NIST). Furthermore, the user and group parameters are hardcoded as "nemo"... you may wish to change this for your infrastructure.

Looking at the source code of the link above will give you an idea of what the systemd target should look like.

You can also run nemo uninstall_systemd_service.

-- Dylan

[kmc...@gmail.com]

Dear Dylan and Mathieu,

I did what you suggested above for installing nemo as a service, but when I check on the status of the nemo service I get the following response:

[kmcpeak@nemo system]$ systemctl status nemo.service -l

● nemo.service - NEMO is a laboratory logistics web application. Use it to schedule reservations, control tool access, track maintenance issues, and more.

   Loaded: loaded (/etc/systemd/system/nemo.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Mon 2020-09-07 13:18:46 CDT; 15min ago

  Process: 10519 ExecStart=/home/nemo/python/bin/gunicorn NEMO.wsgi:application (code=exited, status=3)

 Main PID: 10519 (code=exited, status=3)

Sep 07 13:18:46 nemo.che.lsu.edu gunicorn[10519]: self._setup(name)

Sep 07 13:18:46 nemo.che.lsu.edu gunicorn[10519]: File "/home/nemo/python/lib/python3.6/site-packages/django/conf/__init__.py", line 64, in _setup

Sep 07 13:18:46 nemo.che.lsu.edu gunicorn[10519]: % (desc, ENVIRONMENT_VARIABLE))

Sep 07 13:18:46 nemo.che.lsu.edu gunicorn[10519]: django.core.exceptions.ImproperlyConfigured: Requested setting LOGGING_CONFIG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

Sep 07 13:18:46 nemo.che.lsu.edu gunicorn[10519]: [2020-09-07 13:18:46 -0500] [10526] [INFO] Worker exiting (pid: 10526)

Sep 07 13:18:46 nemo.che.lsu.edu gunicorn[10519]: [2020-09-07 13:18:46 -0500] [10519] [INFO] Shutting down: Master

Sep 07 13:18:46 nemo.che.lsu.edu gunicorn[10519]: [2020-09-07 13:18:46 -0500] [10519] [INFO] Reason: Worker failed to boot.

Sep 07 13:18:46 nemo.che.lsu.edu systemd[1]: nemo.service: main process exited, code=exited, status=3/NOTIMPLEMENTED

Sep 07 13:18:46 nemo.che.lsu.edu systemd[1]: Unit nemo.service entered failed state.

Sep 07 13:18:46 nemo.che.lsu.edu systemd[1]: nemo.service failed.

I am running v3.3.0. Any suggestions for resolving this issue?

Regards,

Kevin

[Dylan Klomparens]

This appears to be the source of the problem:

gunicorn[10519]: django.core.exceptions.ImproperlyConfigured: Requested setting LOGGING_CONFIG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

Check that the settings file is properly configured according to Django’s documentation, and make sure the LOGGING_CONFIG is set to something reasonable in settings.py.

— Dylan

--


You received this message because you are subscribed to the Google Groups "nemo" group.


To unsubscribe from this group and stop receiving emails from it, send an email to nemo+uns...@list.nist.gov.


To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/nemo/3f388c92-9c33-400b-839b-f1730321f048n%40list.nist.gov.

[kmc...@gmail.com]

Dylan,

Thank you for your reply. In my settings.py I do not have a
parameter LOGGING_CONFIG, but I do have the following set for LOGGING: (I believe these override the default LOGGING_CONFIG settings)

LOGGING = {
        'version': 1,
        'disable_existing_loggers': False,
        'formatters': {
            'default': {
            # exact format is not important, this is the minimum information
                'format': '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
            },
        },
        'handlers': {
            'mail_admins': {
                'level': 'INFO',
                'class': 'django.utils.log.AdminEmailHandler'
            },
            'error_file': {
                'level': 'WARNING',
                'class': 'logging.FileHandler',
                'filename': '/home/nemo/logs/django_error.log',
                'formatter': 'default',
            },
            'security_file': {
                'level': 'INFO',
                'class': 'logging.FileHandler',
                'filename': '/home/nemo/logs/django_security.log',
                'formatter': 'default',
            },
        },
        'loggers': {
                'django.request': {
                        'handlers': ['error_file'],
                        'level': 'WARNING',
                        'propagate': True,
                },
                'django.security': {
                        'handlers': ['mail_admins', 'security_file'],
                        'level': 'WARNING',
                        'propagate': True,
                },
        }
}  

I think this is all correct? Do you see an issue?

Regards,

Kevin

[Mathieu Rampant]

Hi Kevin,

I would add a default root logger in there as well (otherwise you will only see logs for django requests and django security) and a NEMO logger:

'loggers': {
    '': {
        'handlers': ['error_file'],
        'level': 'ERROR',
        'propagate': True,
    },

    'NEMO': {
        'handlers': ['error_file'],

        'level': 'INFO',
        'propagate': True,
    },

    'django.requests': { ...}

}

The root logger will catch anything not caught by the other loggers. Having a NEMO logger will allow you to set it to DEBUG for NEMO logs only if you are having trouble etc.

[Sergi Lendínez]

Dear Dylan and Mathieu,

I am trying to continue Kevin's efforts of setting up NEMO as a systemd service here at LSU. However, I'm finding the same error Kevin described above even after adding the suggested loggers to "settings.py".

Any ideas/suggestions?

Thank you,

Sergi

[mathieu...@gmail.com]

Hi Sergi,

Try adding Environment in [SERVICE] in the systemd file that is created (/etc/systemd/system/nemo.service) like this:

[Service]
Environment="DJANGO_SETTINGS_MODULE=NEMO.settings"

...

then run "systemctl daemon-reload" and try again

[mathieu...@gmail.com]

I tested by adding this to nemo.service:

Environment="DJANGO_SETTINGS_MODULE=settings"

Environment="PYTHONPATH=$PYTHONPATH:/<directory_where_settings_are>/"

and it worked for me. I will try to fix the install_systemd_service  in NEMO

[Sergi Lendínez]

Mathieu, thanks a lot for your help.

I have implemented the changes but now a new error shows up.

[kmcpeak@nemo ~]$ systemctl status nemo.service -l

● nemo.service - NEMO is a laboratory logistics web application. Use it to schedule reservations, control tool access, track maintenance issues, and more.
   Loaded: loaded (/etc/systemd/system/nemo.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Tue 2021-05-04 15:54:48 CDT; 6s ago
  Process: 12441 ExecStart=/home/nemo/python/bin/gunicorn NEMO.wsgi:application (code=exited, status=3)
 Main PID: 12441 (code=exited, status=3)

May 04 15:54:48 nemo.che.lsu.edu gunicorn[12441]: check_sqlite_version()
May 04 15:54:48 nemo.che.lsu.edu gunicorn[12441]: File "/home/nemo/python/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 63, in check_sqlite_version
May 04 15:54:48 nemo.che.lsu.edu gunicorn[12441]: raise ImproperlyConfigured('SQLite 3.8.3 or later is required (found %s).' % Database.sqlite_version)
May 04 15:54:48 nemo.che.lsu.edu gunicorn[12441]: django.core.exceptions.ImproperlyConfigured: SQLite 3.8.3 or later is required (found 3.7.17).
May 04 15:54:48 nemo.che.lsu.edu gunicorn[12441]: [2021-05-04 15:54:48 -0500] [12452] [INFO] Worker exiting (pid: 12452)
May 04 15:54:48 nemo.che.lsu.edu gunicorn[12441]: [2021-05-04 15:54:48 -0500] [12441] [INFO] Shutting down: Master
May 04 15:54:48 nemo.che.lsu.edu gunicorn[12441]: [2021-05-04 15:54:48 -0500] [12441] [INFO] Reason: Worker failed to boot.
May 04 15:54:48 nemo.che.lsu.edu systemd[1]: nemo.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
May 04 15:54:48 nemo.che.lsu.edu systemd[1]: Unit nemo.service entered failed state.
May 04 15:54:48 nemo.che.lsu.edu systemd[1]: nemo.service failed.

When I check the SQlite version I get

[kmcpeak@nemo ~]$ sqlite3 --version
3.29.0 2019-07-10 17:32:03 fc82b73eaac8b36950e527f12c4b5dc1e147e6f4ad2217ae43ad82882a88bfa6

But python3 in this root user is seeing an older version:

[kmcpeak@nemo ~]$ python3 -c "import sqlite3; print(sqlite3.sqlite_version)"
3.7.17

Any ideas on how to fix this?

Thank you,

Sergi

[mathieu...@gmail.com]

First of all, depending on the amount of traffic you expect on your instance of NEMO, you might want to make sure SQlite is right for you. There are other alternatives, like PostgreSQL that could be better suited.

If you want to keep using SQlite, find where your system sqlite3 library is by running something like: find / -name "libsqlite3.so"

then you can set the environment variable in nemo.service:

Environment="LD_LIBRARY_PATH=/<folder_where_lib_is>/"

that *should* do it

[Sergi Lendínez]

Hi Mathieu,

Thanks again, that did actually solve the issue and now we can start the NEMO service. However, that doesn't start nginx, so I tried to also do a similar systemd service for nginx.

In this case, the nginx.service file looks like this:

[Unit]

Description=NGINX HTTP and reverse proxy server

After=syslog.target netweork-online.target remote-fs.target nss-lookup.target

Wants=network-online.target

 

[Service]

Type=forking

PIDFile=/tmp/nginx.pid

ExecStartPre=/home/nemo/nginx/nginx -t

ExecStart=/home/nemo/nginx/nginx

ExecReload=/home/nemo/nginx/nginx -s reload

ExecStop=/bin/kill -s QUIT $MAINPID

PrivateTmp=true

 

[Install]

WantedBy=multi-user.target

However, the service fails to start with the following error:

● nginx.service - NGINX HTTP and reverse proxy server
   Loaded: loaded (/etc/systemd/system/nginx.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2021-05-05 09:39:41 CDT; 3h 45min ago
  Process: 51226 ExecStartPre=/home/nemo/nginx/nginx -t (code=exited, status=1/FAILURE)

May 05 09:39:40 nemo.che.lsu.edu systemd[1]: Starting NGINX HTTP and reverse proxy server...
May 05 09:39:41 nemo.che.lsu.edu nginx[51226]: nginx: [emerg] open() "./configuration" failed (2: No such file or directory)
May 05 09:39:41 nemo.che.lsu.edu nginx[51226]: nginx: configuration file ./configuration test failed
May 05 09:39:41 nemo.che.lsu.edu systemd[1]: nginx.service: control process exited, code=exited status=1
May 05 09:39:41 nemo.che.lsu.edu systemd[1]: Failed to start NGINX HTTP and reverse proxy server.
May 05 09:39:41 nemo.che.lsu.edu systemd[1]: Unit nginx.service entered failed state.
May 05 09:39:41 nemo.che.lsu.edu systemd[1]: nginx.service failed.

Again, thank you very much for your help!

Regards,

Sergi

[mathieu...@gmail.com]

Hi Sergi,

You probably just need to specify the configuration file for nginx, 

ExecStart=/home/nemo/nginx/nginx -c <absolute_path_to_nginx.conf>

[Sergi Lendínez]

Hi Mathieu,

Thank you so much for your help, I finally got both nemo and nginx to run as systemd services. The nginx.service file looks like this:

[Unit]

Description=NGINX HTTP and reverse proxy server

After=syslog.target netweork-online.target remote-fs.target nss-lookup.target

Wants=network-online.target

 

[Service]

Type=forking

ExecStart=/home/nemo/nginx/nginx -c /home/nemo/nginx/configuration

ExecReload=/home/nemo/nginx/nginx -s reload

ExecStop=/bin/kill -s TERM $MAINPID

 

[Install]

WantedBy=multi-user.target

For some reason, I had to remove the definition of the PIDFile in the Service section.

Thanks again for helping me in setting this up!

Regards,

Sergi

[mathieu...@gmail.com]

Glad it's working!

Thanks for posting your working service file, I am sure it will be useful to others!