Error Module Django not found

[Matt Bozeman]

In a CentOs 7 installation of a netbox 2.6.4 instance using gunicorn and nginx the command
'gunicorn -c /opt/netbox/gunicorn_config.py netbox.wsgi' continually comes back with the error

"File "/usr/local/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
File "/usr/local/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
File "/usr/local/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
File "/usr/local/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
File "/usr/local/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
File "/opt/netbox-2.6.4/netbox/netbox/wsgi.py", line 9, in
from django.core.wsgi import get_wsgi_application
ModuleNotFoundError: No module named 'django'
[2019-09-20 14:52:38 -0500] [30731] [INFO] Worker exiting (pid: 30731)
[2019-09-20 14:52:38 -0500] [30706] [INFO] Shutting down: Master
[2019-09-20 14:52:38 -0500] [30706] [INFO] Reason: Worker failed to boot."

the /opt/netbox/gunicorn_config.py file contains:
"
pythonpath = "/opt/netbox/netbox"
bind = "127.0.0.1:8001"
workers = 3
user = "nginx"
"

and the /etc/supervisor.d/netbox.conf file contains:
"
program:netbox]
command = gunicorn -c /opt/netbox/gunicorn_config.py netbox.wsgi
directory = /opt/netbox/netbox/
user = nginx
"
if I comment out the user in /etc/supervisor.d/netbox.conf or if I run the command

"/usr/local/bin/gunicorn --pythonpath "/opt/netbox/netbox" --bind "127.0.0.1:8001" --workers 3 netbox.wsgi"

the page will load. However, the terminal will not run as nginx it will only run it as root, but other files are being run as nginx as if I run: ps aux | grep nginx

the output is:
"
root 4868 0.0 0.0 112708 976 pts/0 S+ 16:27 0:00 grep --color=auto nginx
root 29731 0.0 0.0 125084 2292 ? Ss 14:38 0:00 nginx: master process /usr/sbin/nginx
nginx 29732 0.0 0.0 127692 4052 ? S 14:38 0:00 nginx: worker process
nginx 29733 0.0 0.0 127692 4052 ? S 14:38 0:00 nginx: worker process
nginx 29734 0.0 0.0 127692 4048 ? S 14:38 0:00 nginx: worker process
nginx 29735 0.0 0.0 127692 3596 ? S 14:38 0:00 nginx: worker process
nginx 29736 0.0 0.0 127692 3804 ? S 14:38 0:00 nginx: worker process
nginx 29737 0.0 0.0 127692 3596 ? S 14:38 0:00 nginx: worker process
nginx 29738 0.0 0.0 127692 3596 ? S 14:38 0:00 nginx: worker process
nginx 29739 0.0 0.0 127692 3800 ? S 14:38 0:00 nginx: worker process
"
Any way to run it as nginx instead of root?

[Brian Candler]

The packages that Netbox depends on (e.g. django) ought to have been installed globally - e.g. under /usr/local/lib/python3.5/

Perhaps what's happened is they ended up under /root/.local/lib/python3.5 or somewhere like that instead.

If that's true, then you need to start from scratch with the installation, and work out where it went wrong.  If you logged in as a normal user first, then try "sudo su -l" to get a root login shell.

[Matt Bozeman]

Django and these packages are located at : /usr/local/lib/python3.6/site-packages/django

[Brian Candler]

We're into the realms of general system debugging now, not really anything to do with Netbox.  Try:

sudo su - nginx -s /bin/bash

echo "import django" | python3

If that fails, try again with strace:

echo "import django" | strace -f python3

Does it try to open the files under /usr/local/lib/python3.6? Maybe a permissions problem?

[Brian Candler]

Can you show the output of:

ls -l /usr/local/lib/python3.6/site-packages/

[Matt Bozeman]

I started over on a new machine and I got passed the Django module not found error, but I know get this error:

2019-09-26 11:56:12,144 INFO RPC interface 'supervisor' initialized

2019-09-26 11:56:12,146 INFO daemonizing the supervisord process

2019-09-26 11:56:12,147 INFO supervisord started with pid 1281

2019-09-26 11:56:13,152 INFO spawned: 'netbox' with pid 1283

2019-09-26 11:56:13,156 INFO spawned: 'netbox-rqworker' with pid 1284

2019-09-26 11:56:14,553 INFO success: netbox entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

2019-09-26 11:56:14,553 INFO success: netbox-rqworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)"

then netbox-rqworker exits with (exit status 1; not expected) so netbox starts, but the rqworker dies and takes netbox with it. Anyone encouontered this/know how to fix it?

On Tuesday, September 24, 2019 at 10:45:40 AM UTC-5, Brian Candler wrote:

[Brian Candler]

Can you show the output from the netbox-rqworker process?  If necessary, start it manually in the same way that supervisord does.

netbox-rqworker halting shouldn't "take netbox with it".  They are separate processes, and I believe rqworker is only required if you are using webhooks.

[Matt Bozeman]

On Monday, September 30, 2019 at 11:20:44 AM UTC-5, Brian Candler wrote:

Can you show the output from the netbox-rqworker process?  If necessary, start it manually by python manage.py -c /opt/netbox/gunicorn.conf I get:

 

ImportError: No module named django.core.management

[Brian Candler]

So same issue then - Django libraries not installed in a place where the netbox-rqworker, running as uid X, can find them.  Or not readable due to permissions.

Running it under strace will show you what files/directories it is trying to open.

And to be clear, "python manage.py" is running the same version of python as you were running when you installed netbox - this isn't python2 by any chance?

[Matt Bozeman]

solution was found, I simply put SELinux in permissive mode and it works now. 

[Brian Candler]

Ah yes, SELinux - the universal Linux system breaker :-)  Glad it's sorted.

For future reference, it could be helpful if you could post any 'deny' message you saw in dmesg.

As to how python/gunicorn ended up in a constrained security context which can't even read standard python library locations, I have no idea.