NetBox HTTP - nginx

[Jason Kopacko]

I can access NetBox directly via the testing with:

(venv) ubuntu@####:/etc/nginx/sites-enabled$ sudo python3 /opt/netbox/netbox/manage.py runserver 0.0.0.0:8000 --insecure
Performing system checks...


System check identified no issues (0 silenced).
June 05, 2020 - 13:41:53
Django version 3.0.7, using settings 'netbox.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

I copied over the netbox.conf file to nginx (which is working and serving other websites) and I get this:

(venv) ubuntu@####:/etc/nginx/sites-enabled$ sudo systemctl status netbox
● netbox.service - NetBox WSGI Service
   Loaded: loaded (/etc/systemd/system/netbox.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2020-06-05 13:40:11 CDT; 7s ago
     Docs: https://netbox.readthedocs.io/en/stable/
  Process: 22341 ExecStart=/opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi (code=exited, status=203/EXEC)
 Main PID: 22341 (code=exited, status=203/EXEC)

Any ideas?

Also, my nginx configuration is this:

server {
    listen 80;


    # CHANGE THIS TO YOUR SERVER'S NAME
    server_name ####;


    client_max_body_size 25m;


    location /static/ {
        alias /opt/netbox/netbox/static/;
    }


    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

I removed the 443 stuff to validate everything is working before installing a cert.

[Jason Kopacko]

Here is what I see when accessing the test server via port 8000:

[Jason Kopacko]

Found this:

Jun 05 14:14:02 #### systemd[23466]: netbox.service: Failed at step EXEC spawning /opt/netbox/venv/bin/gunicorn: No such file or directory

I suspect I missed a step in the document for installing gunicorn, but I can't seem to find it.

[Jason Kopacko]

Can't seem to post full copy but I re-ran the install where gunicorn is installed:

Re-ran the install where gunicorn is installed:

(venv) ubuntu@####:/opt/netbox$ sudo pip3 install -r requirements.txt

The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Requirement already satisfied: Django<3.1,>=3.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 1))

Requirement already satisfied: django-cacheops==4.2 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 2))

Requirement already satisfied: django-cors-headers==3.2.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 3))

Requirement already satisfied: django-debug-toolbar==2.2 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 4))

Requirement already satisfied: django-filter==2.2.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 5))

Requirement already satisfied: django-mptt==0.11.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 6))

Requirement already satisfied: django-pglocks==1.0.4 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 7))

Requirement already satisfied: django-prometheus==2.0.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 8))

Requirement already satisfied: django-rq==2.3.2 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 9))

Requirement already satisfied: django-tables2==2.3.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 10))

Requirement already satisfied: django-taggit==1.2.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 11))

Requirement already satisfied: django-taggit-serializer==0.1.7 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 12))

Requirement already satisfied: django-timezone-field==4.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 13))

Requirement already satisfied: djangorestframework==3.11.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 14))

Requirement already satisfied: drf-yasg[validation]==1.17.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 15))

Requirement already satisfied: gunicorn==20.0.4 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 16))

Requirement already satisfied: Jinja2==2.11.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 17))

Requirement already satisfied: Markdown==3.2.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 18))

Requirement already satisfied: netaddr==0.7.19 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 19))

Requirement already satisfied: Pillow==7.1.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 20))

Requirement already satisfied: psycopg2-binary==2.8.5 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 21))

Requirement already satisfied: pycryptodome==3.9.7 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 22))

Requirement already satisfied: PyYAML==5.3.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 23))

Requirement already satisfied: redis==3.4.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 24))

Requirement already satisfied: svgwrite==1.4 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 25))

Requirement already satisfied: asgiref~=3.2 in /usr/local/lib/python3.6/dist-packages (from Django<3.1,>=3.0->-r requirements.txt (line 1))

Requirement already satisfied: pytz in /usr/lib/python3/dist-packages (from Django<3.1,>=3.0->-r requirements.txt (line 1))

Requirement already satisfied: sqlparse>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from Django<3.1,>=3.0->-r requirements.txt (line 1))

Requirement already satisfied: six>=1.4.0 in /usr/lib/python3/dist-packages (from django-cacheops==4.2->-r requirements.txt (line 2))

Requirement already satisfied: funcy<2.0,>=1.8 in /usr/local/lib/python3.6/dist-packages (from django-cacheops==4.2->-r requirements.txt (line 2))

Requirement already satisfied: django-js-asset in /usr/local/lib/python3.6/dist-packages (from django-mptt==0.11.0->-r requirements.txt (line 6))

Requirement already satisfied: prometheus-client>=0.7 in /usr/local/lib/python3.6/dist-packages (from django-prometheus==2.0.0->-r requirements.txt (line 8))

Requirement already satisfied: rq>=1.2 in /usr/local/lib/python3.6/dist-packages (from django-rq==2.3.2->-r requirements.txt (line 9))

Requirement already satisfied: coreapi>=2.3.3 in /usr/local/lib/python3.6/dist-packages (from drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: coreschema>=0.0.4 in /usr/local/lib/python3.6/dist-packages (from drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: ruamel.yaml>=0.15.34 in /usr/local/lib/python3.6/dist-packages (from drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: uritemplate>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: inflection>=0.3.1 in /usr/local/lib/python3.6/dist-packages (from drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: swagger-spec-validator>=2.1.0; extra == "validation" in /usr/local/lib/python3.6/dist-packages (from drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: setuptools>=3.0 in /usr/lib/python3/dist-packages (from gunicorn==20.0.4->-r requirements.txt (line 16))

Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib/python3/dist-packages (from Jinja2==2.11.1->-r requirements.txt (line 17))

Requirement already satisfied: click>=5.0 in /usr/lib/python3/dist-packages (from rq>=1.2->django-rq==2.3.2->-r requirements.txt (line 9))

Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from coreapi>=2.3.3->drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: itypes in /usr/local/lib/python3.6/dist-packages (from coreapi>=2.3.3->drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /usr/local/lib/python3.6/dist-packages (from ruamel.yaml>=0.15.34->drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: pyparsing>=2.0.2 in /usr/lib/python3/dist-packages (from packaging->drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

Requirement already satisfied: jsonschema in /usr/lib/python3/dist-packages (from swagger-spec-validator>=2.1.0; extra == "validation"->drf-yasg[validation]==1.17.1->-r requirements.txt (line 15))

[Jason Kopacko]

My directory list where gunicorn should be:

(venv) ubuntu@####:/opt/netbox/venv/bin$ ll
total 40
drwxr-xr-x 2 root root 4096 Jun  5 12:42 ./
drwxr-xr-x 6 root root 4096 Jun  5 12:42 ../
-rw-r--r-- 1 root root 2192 Jun  5 12:42 activate
-rw-r--r-- 1 root root 1248 Jun  5 12:42 activate.csh
-rw-r--r-- 1 root root 2412 Jun  5 12:42 activate.fish
-rwxr-xr-x 1 root root  246 Jun  5 12:42 easy_install*
-rwxr-xr-x 1 root root  246 Jun  5 12:42 easy_install-3.6*
-rwxr-xr-x 1 root root  218 Jun  5 12:42 pip*
-rwxr-xr-x 1 root root  218 Jun  5 12:42 pip3*
-rwxr-xr-x 1 root root  218 Jun  5 12:42 pip3.6*
lrwxrwxrwx 1 root root    7 Jun  5 12:42 python -> python3*
lrwxrwxrwx 1 root root   16 Jun  5 12:42 python3 -> /usr/bin/python3*

[Jason Kopacko]

So not sure if I did something wrong but I did this:

ubuntu@####:/opt/netbox$ source venv/bin/activate


(venv) ubuntu@####:/opt/netbox$ sudo su


root@####:/opt/netbox-2.8.5# cd /opt/netbox


root@####:/opt/netbox# source venv/bin/activate


(venv) root@####:/opt/netbox# pip3 install -r requirements.txt
GUNICORN SPECIFIC LINES
Collecting gunicorn==20.0.4 (from -r requirements.txt (line 16))
  Using cached https://files.pythonhosted.org/packages/69/ca/926f7cd3a2014b16870086b2d0fdc84a9e49473c68a8dff8b57f7c156f43/gunicorn-20.0.4-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=3.0 in ./venv/lib/python3.6/site-packages (from gunicorn==20.0.4->-r requirements.txt (line 16))
Installing collected packages: gunicorn
Successfully installed gunicorn-20.0.4


(venv) root@####:/opt/netbox# cd venv/bin


(venv) root@####:/opt/netbox/venv/bin# ll
total 88
drwxr-xr-x 3 root root 4096 Jun  5 14:28 ./

drwxr-xr-x 6 root root 4096 Jun  5 12:42 ../
-rw-r--r-- 1 root root 2192 Jun  5 12:42 activate
-rw-r--r-- 1 root root 1248 Jun  5 12:42 activate.csh
-rw-r--r-- 1 root root 2412 Jun  5 12:42 activate.fish

-rwxr-xr-x 1 root root  237 Jun  5 14:28 chardetect*
-rwxr-xr-x 1 root root  279 Jun  5 14:28 django-admin*
-rwxr-xr-x 1 root root  137 Jun  5 14:28 django-admin.py*

-rwxr-xr-x 1 root root  246 Jun  5 12:42 easy_install*
-rwxr-xr-x 1 root root  246 Jun  5 12:42 easy_install-3.6*

-rwxr-xr-x 1 root root  233 Jun  5 14:28 gunicorn*
-rwxr-xr-x 1 root root  229 Jun  5 14:28 jsonschema*
-rwxr-xr-x 1 root root  230 Jun  5 14:28 markdown_py*
-rwxr-xr-x 1 root root 1167 Jun  5 14:28 netaddr*

-rwxr-xr-x 1 root root  218 Jun  5 12:42 pip*
-rwxr-xr-x 1 root root  218 Jun  5 12:42 pip3*
-rwxr-xr-x 1 root root  218 Jun  5 12:42 pip3.6*

drwxr-xr-x 2 root root 4096 Jun  5 14:28 __pycache__/

lrwxrwxrwx 1 root root    7 Jun  5 12:42 python -> python3*
lrwxrwxrwx 1 root root   16 Jun  5 12:42 python3 -> /usr/bin/python3*

-rwxr-xr-x 1 root root  221 Jun  5 14:28 rq*
-rwxr-xr-x 1 root root  221 Jun  5 14:28 rqinfo*
-rwxr-xr-x 1 root root  225 Jun  5 14:28 rqworker*
-rwxr-xr-x 1 root root  232 Jun  5 14:28 sqlformat*

[Brian Candler]

Look carefully at that output:

On Friday, 5 June 2020 20:24:02 UTC+1, Jason Kopacko wrote:

Requirement already satisfied: gunicorn==20.0.4 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 16))

You are installing into the global environment (/usr/local/lib), not into the netbox venv (/opt/netbox/venv)

You need to "activate" the venv before using pip.

cd /opt/netbox

source venv/bin/activate

pip3 install -r requirements.txt

(as documented at https://netbox.readthedocs.io/en/stable/installation/3-netbox/#set-up-python-environment )

This "activation" sets environment variables in your shell, and only lasts for as long as you're logged in.  So if you logout and login again, you'll need to re-activate.

[Jason Kopacko]

sudo journalctl -eu netbox returns this:

Jun 05 14:28:40 #### gunicorn[24698]: [2020-06-05 14:28:40 -0500] [24698] [INFO] Starting gunicorn 20.0.4
Jun 05 14:28:40 #### gunicorn[24698]: [2020-06-05 14:28:40 -0500] [24698] [INFO] Listening at: http://127.0.0.1:8001 (24698)
Jun 05 14:28:40 #### gunicorn[24698]: [2020-06-05 14:28:40 -0500] [24698] [INFO] Using worker: threads
Jun 05 14:28:40 #### gunicorn[24698]: [2020-06-05 14:28:40 -0500] [24698] [INFO] Unhandled exception in main loop
Jun 05 14:28:40 #### gunicorn[24698]: Traceback (most recent call last):
Jun 05 14:28:40 #### gunicorn[24698]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 202, in run
Jun 05 14:28:40 #### gunicorn[24698]:     self.manage_workers()
Jun 05 14:28:40 #### gunicorn[24698]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
Jun 05 14:28:40 #### gunicorn[24698]:     self.spawn_workers()
Jun 05 14:28:40 #### gunicorn[24698]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
Jun 05 14:28:40 #### gunicorn[24698]:     self.spawn_worker()
Jun 05 14:28:40 #### gunicorn[24698]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 565, in spawn_worker
Jun 05 14:28:40 #### gunicorn[24698]:     self.cfg, self.log)
Jun 05 14:28:40 #### gunicorn[24698]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/workers/gthread.py", line 69, in __init__
Jun 05 14:28:40 #### gunicorn[24698]:     super().__init__(*args, **kwargs)
Jun 05 14:28:40 #### gunicorn[24698]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 63, in __init__
Jun 05 14:28:40 #### gunicorn[24698]:     self.tmp = WorkerTmp(cfg)
Jun 05 14:28:40 #### gunicorn[24698]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/workers/workertmp.py", line 29, in __init__
Jun 05 14:28:40 #### gunicorn[24698]:     util.chown(name, cfg.uid, cfg.gid)
Jun 05 14:28:40 #### gunicorn[24698]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/util.py", line 153, in chown
Jun 05 14:28:40 #### gunicorn[24698]:     os.chown(path, uid, gid)
Jun 05 14:28:40 #### gunicorn[24698]: PermissionError: [Errno 1] Operation not permitted: '/tmp/wgunicorn-or7om2ha'
Jun 05 14:28:40 #### systemd[1]: netbox.service: Main process exited, code=exited, status=255/n/a
Jun 05 14:28:40 #### systemd[1]: netbox.service: Failed with result 'exit-code'.
Jun 05 14:29:10 #### systemd[1]: netbox.service: Service hold-off time over, scheduling restart.
Jun 05 14:29:10 #### systemd[1]: netbox.service: Scheduled restart job, restart counter is at 57.
Jun 05 14:29:10 #### systemd[1]: Stopped NetBox WSGI Service.
Jun 05 14:29:10 #### systemd[1]: Started NetBox WSGI Service.

[Brian Candler]

What OS are you under?  If it's CentOS and it has SELinux enabled then that could be it - try setting SELinux to permissive mode.

[Jason Kopacko]

Fresh install of Ubuntu 18.04.

This is the latest error info:

Jun 05 14:45:34 #### systemd[1]: Started NetBox WSGI Service.
Jun 05 14:45:34 #### gunicorn[27011]: [2020-06-05 14:45:34 -0500] [27011] [INFO] Starting gunicorn 20.0.4
Jun 05 14:45:34 #### gunicorn[27011]: [2020-06-05 14:45:34 -0500] [27011] [INFO] Listening at: http://127.0.0.1:8001 (27011)
Jun 05 14:45:34 #### gunicorn[27011]: [2020-06-05 14:45:34 -0500] [27011] [INFO] Using worker: threads
Jun 05 14:45:34 #### gunicorn[27011]: [2020-06-05 14:45:34 -0500] [27011] [INFO] Unhandled exception in main loop
Jun 05 14:45:34 #### gunicorn[27011]: Traceback (most recent call last):
Jun 05 14:45:34 #### gunicorn[27011]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 202, in run
Jun 05 14:45:34 #### gunicorn[27011]:     self.manage_workers()
Jun 05 14:45:34 #### gunicorn[27011]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
Jun 05 14:45:34 #### gunicorn[27011]:     self.spawn_workers()
Jun 05 14:45:34 #### gunicorn[27011]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
Jun 05 14:45:34 #### gunicorn[27011]:     self.spawn_worker()
Jun 05 14:45:34 #### gunicorn[27011]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 565, in spawn_worker
Jun 05 14:45:34 #### gunicorn[27011]:     self.cfg, self.log)
Jun 05 14:45:34 #### gunicorn[27011]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/workers/gthread.py", line 69, in __init__
Jun 05 14:45:34 #### gunicorn[27011]:     super().__init__(*args, **kwargs)
Jun 05 14:45:34 #### gunicorn[27011]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 63, in __init__
Jun 05 14:45:34 #### gunicorn[27011]:     self.tmp = WorkerTmp(cfg)
Jun 05 14:45:34 #### gunicorn[27011]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/workers/workertmp.py", line 29, in __init__
Jun 05 14:45:34 #### gunicorn[27011]:     util.chown(name, cfg.uid, cfg.gid)
Jun 05 14:45:34 #### gunicorn[27011]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/util.py", line 153, in chown
Jun 05 14:45:34 #### gunicorn[27011]:     os.chown(path, uid, gid)
Jun 05 14:45:34 #### gunicorn[27011]: PermissionError: [Errno 1] Operation not permitted: '/tmp/wgunicorn-nz47igsc'
Jun 05 14:45:34 #### systemd[1]: netbox.service: Main process exited, code=exited, status=255/n/a
Jun 05 14:45:34 #### systemd[1]: netbox.service: Failed with result 'exit-code'.

[Brian Candler]

 Can you show output of:

ls -ld /tmp

cat /etc/systemd/system/netbox.service

cat /opt/netbox/gunicorn_config.py

[Jason Kopacko]

Here you go:

ubuntu@####:~$ ls -ld /tmp
drwxrwxrwt 11 root root 4096 Jun  5 14:54 /tmp


ubuntu@####:~$ cat /etc/systemd/system/netbox.service
[Unit]
Description=NetBox WSGI Service
Documentation=https://netbox.readthedocs.io/en/stable/
After=network-online.target
Wants=network-online.target

[Service]
Type=simple

User=netbox
Group=netbox
PIDFile=/var/tmp/netbox.pid
WorkingDirectory=/opt/netbox

ExecStart=/opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi

Restart=on-failure
RestartSec=30
PrivateTmp=true

[Install]
WantedBy=multi-user.target

ubuntu@####:~$ cat /opt/netbox/gunicorn_config.py
cat: /opt/netbox/gunicorn_config.py: No such file or directory

ubuntu@####:~$ cd /opt/netbox

ubuntu@####:/opt/netbox$ ll
total 100
drwxrwxr-x  8 root root  4096 Jun  5 14:41 ./
drwxr-xr-x  4 root root  4096 Jun  5 12:38 ../
-rw-rw-r--  1 root root  2580 May 26 15:17 base_requirements.txt
-rw-rw-r--  1 root root   117 May 26 15:17 CHANGELOG.md
drwxrwxr-x  2 root root  4096 Jun  5 14:05 contrib/
-rw-rw-r--  1 root root  8081 May 26 15:17 CONTRIBUTING.md
drwxrwxr-x 13 root root  4096 May 26 15:17 docs/
-rw-rw-r--  1 root root    17 May 26 15:17 .gitattributes
drwxrwxr-x  3 root root  4096 May 26 15:17 .github/
-rw-rw-r--  1 root root   287 May 26 15:17 .gitignore
-rw-r--r--  1 root root   575 Jun  5 14:09 gunicorn.py
-rw-rw-r--  1 root root 10174 May 26 15:17 LICENSE.txt
-rw-r--r--  1 root root     7 Jun  5 14:40 local_requirements.txt
-rw-rw-r--  1 root root  3994 May 26 15:17 mkdocs.yml
drwxrwxr-x 18 root root  4096 Jun  5 13:11 netbox/
-rw-rw-r--  1 root root    72 May 26 15:17 NOTICE
-rw-rw-r--  1 root root  2597 May 26 15:17 README.md
-rw-rw-r--  1 root root   517 May 26 15:17 requirements.txt
drwxrwxr-x  3 root root  4096 May 26 15:17 scripts/
-rw-rw-r--  1 root root   339 May 26 15:17 .travis.yml
-rwxrwxr-x  1 root root  3125 May 26 15:17 upgrade.sh*
drwxr-xr-x  6 root root  4096 Jun  5 12:42 venv/

[Jason Kopacko]

Here is this:

ubuntu@####:/opt/netbox$ cat /opt/netbox/gunicorn.py
# The IP address (typically localhost) and port that the Netbox WSGI process should listen on
bind = '127.0.0.1:8001'


# Number of gunicorn workers to spawn. This should typically be 2n+1, where
# n is the number of CPU cores present.
workers = 5


# Number of threads per worker process
threads = 3


# Timeout (in seconds) for a request to complete
timeout = 120


# The maximum number of requests a worker can handle before being respawned
max_requests = 5000
max_requests_jitter = 500


command = '/usr/local/bin/gunicorn'
pythonpath = '/opt/netbox/netbox'
user = 'www-data'

[Jason Kopacko]

These lines I copied over from my Ubuntu 16.04 installation:

[Jason Kopacko]

So I manually ran this:

sudo /opt/netbox/venv/bin/gunicorn --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi

And got this:

ValueError: Unable to configure handler 'file': [Errno 13] Permission denied: '/var/log/netbox.log'

So I ran this:

sudo chown www-data:www-data netbox.log

And now see this:

-rw-r--r--   1 www-data  www-data          7517 Jun  5 14:53 netbox.log

Now when I run it manually, I see this:

ubuntu@####:/var/log$ sudo /opt/netbox/venv/bin/gunicorn --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi


[2020-06-05 15:15:55 -0500] [31971] [INFO] Starting gunicorn 20.0.4
[2020-06-05 15:15:55 -0500] [31971] [INFO] Listening at: http://127.0.0.1:8001 (31971)
[2020-06-05 15:15:55 -0500] [31971] [INFO] Using worker: threads
[2020-06-05 15:15:55 -0500] [31974] [INFO] Booting worker with pid: 31974
[2020-06-05 15:15:55 -0500] [31975] [INFO] Booting worker with pid: 31975
[2020-06-05 15:15:55 -0500] [31978] [INFO] Booting worker with pid: 31978
[2020-06-05 15:15:55 -0500] [31979] [INFO] Booting worker with pid: 31979
[2020-06-05 15:15:55 -0500] [31981] [INFO] Booting worker with pid: 31981

I opened another session and checked the status and see this:

ubuntu@####:~$ sudo systemctl status netbox

● netbox.service - NetBox WSGI Service
   Loaded: loaded (/etc/systemd/system/netbox.service; enabled; vendor preset: enabled)

   Active: inactive (dead) (Result: exit-code) since Fri 2020-06-05 15:12:52 CDT; 5min ago
     Docs: https://netbox.readthedocs.io/en/stable/
  Process: 31775 ExecStart=/opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi (code=exited, status=255)
 Main PID: 31775 (code=exited, status=255)

Jun 05 15:12:52 #### systemd[1]: Stopped NetBox WSGI Service.

Even though the site is actually responding right now.

[Jason Kopacko]

I keep seeing this:

Jun 05 15:30:05 #### gunicorn[1778]: PermissionError: [Errno 1] Operation not permitted: '/tmp/wgunicorn-b_q0vshf'

Or some variation as it is trying to create a temp file. I noticed, on one attempt this existed:

ubuntu@####:/tmp$ ll

-rw-------  1 ubuntu ubuntu    0 Jun  5 15:10 wgunicorn-7hsakcja

I suspect this is a permissions problem and I cannot seem to resolve it.

[Jason Kopacko]

I am not even sure what I did anymore but it appears to be working. I found one more permission problem.

Jun  5 15:55:20 #### gunicorn[5461]: ValueError: Unable to configure handler 'file': [Errno 13] Permission denied: '/var/log/netbox.log'

So I did [chmod 777 netbox.log] and see this:

ubuntu@####:/var/log$ ll | grep netbox

-rwxrwxrwx   1 www-data  www-data              0 Jun  5 15:50 netbox.log*

And now everything seems to be working.

ubuntu@####:/var/log$ sudo systemctl status netbox

● netbox.service - NetBox WSGI Service
   Loaded: loaded (/etc/systemd/system/netbox.service; enabled; vendor preset: enabled)

   Active: active (running) since Fri 2020-06-05 15:55:50 CDT; 1min 52s ago
     Docs: https://netbox.readthedocs.io/en/stable/
 Main PID: 5552 (gunicorn)
    Tasks: 6 (limit: 4915)
   CGroup: /system.slice/netbox.service
           ├─5552 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
           ├─5572 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
           ├─5574 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
           ├─5575 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
           ├─5578 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
           └─5582 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi


Jun 05 15:55:50 #### systemd[1]: Started NetBox WSGI Service.
Jun 05 15:55:51 #### gunicorn[5552]: [2020-06-05 15:55:51 -0500] [5552] [INFO] Starting gunicorn 20.0.4
Jun 05 15:55:51 #### gunicorn[5552]: [2020-06-05 15:55:51 -0500] [5552] [INFO] Listening at: http://127.0.0.1:8001 (5552)
Jun 05 15:55:51 #### gunicorn[5552]: [2020-06-05 15:55:51 -0500] [5552] [INFO] Using worker: threads
Jun 05 15:55:51 #### gunicorn[5552]: [2020-06-05 15:55:51 -0500] [5572] [INFO] Booting worker with pid: 5572
Jun 05 15:55:51 #### gunicorn[5552]: [2020-06-05 15:55:51 -0500] [5574] [INFO] Booting worker with pid: 5574
Jun 05 15:55:51 #### gunicorn[5552]: [2020-06-05 15:55:51 -0500] [5575] [INFO] Booting worker with pid: 5575
Jun 05 15:55:51 #### gunicorn[5552]: [2020-06-05 15:55:51 -0500] [5578] [INFO] Booting worker with pid: 5578
Jun 05 15:55:51 #### gunicorn[5552]: [2020-06-05 15:55:51 -0500] [5582] [INFO] Booting worker with pid: 5582

[Jason Kopacko]

ubuntu@####:/var/log$ sudo lsof -nPi

COMMAND     PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python3    5512          netbox    7u  IPv4 227002      0t0  TCP 127.0.0.1:33724->127.0.0.1:6379 (ESTABLISHED)
python3    5512          netbox    8u  IPv4 227003      0t0  TCP 127.0.0.1:33726->127.0.0.1:6379 (ESTABLISHED)
gunicorn   5552          netbox    5u  IPv4 227108      0t0  TCP 127.0.0.1:8001 (LISTEN)
gunicorn   5572          netbox    5u  IPv4 227108      0t0  TCP 127.0.0.1:8001 (LISTEN)
gunicorn   5574          netbox    5u  IPv4 227108      0t0  TCP 127.0.0.1:8001 (LISTEN)
gunicorn   5575          netbox    5u  IPv4 227108      0t0  TCP 127.0.0.1:8001 (LISTEN)
gunicorn   5578          netbox    5u  IPv4 227108      0t0  TCP 127.0.0.1:8001 (LISTEN)
gunicorn   5582          netbox    5u  IPv4 227108      0t0  TCP 127.0.0.1:8001 (LISTEN)
postgres  13068        postgres    7u  IPv4  86779      0t0  TCP 127.0.0.1:5432 (LISTEN)
postgres  13068        postgres   11u  IPv4  86785      0t0  UDP 127.0.0.1:49365->127.0.0.1:49365
postgres  13078        postgres   11u  IPv4  86785      0t0  UDP 127.0.0.1:49365->127.0.0.1:49365
postgres  13079        postgres   11u  IPv4  86785      0t0  UDP 127.0.0.1:49365->127.0.0.1:49365
postgres  13080        postgres   11u  IPv4  86785      0t0  UDP 127.0.0.1:49365->127.0.0.1:49365
postgres  13081        postgres   11u  IPv4  86785      0t0  UDP 127.0.0.1:49365->127.0.0.1:49365
postgres  13083        postgres   11u  IPv4  86785      0t0  UDP 127.0.0.1:49365->127.0.0.1:49365
postgres  13085        postgres   11u  IPv4  86785      0t0  UDP 127.0.0.1:49365->127.0.0.1:49365
redis-ser 14456           redis    6u  IPv4  89518      0t0  TCP 127.0.0.1:6379 (LISTEN)
redis-ser 14456           redis    7u  IPv6  89519      0t0  TCP [::1]:6379 (LISTEN)
redis-ser 14456           redis    8u  IPv4 225035      0t0  TCP 127.0.0.1:6379->127.0.0.1:33724 (ESTABLISHED)
redis-ser 14456           redis    9u  IPv4 225037      0t0  TCP 127.0.0.1:6379->127.0.0.1:33726 (ESTABLISHED)
nginx     21603            root    8u  IPv4 105622      0t0  TCP *:80 (LISTEN)
nginx     21603            root    9u  IPv6 105623      0t0  TCP *:80 (LISTEN)
nginx     21603            root   10u  IPv4 105624      0t0  TCP *:443 (LISTEN)
nginx     21606        www-data    8u  IPv4 105622      0t0  TCP *:80 (LISTEN)
nginx     21606        www-data    9u  IPv6 105623      0t0  TCP *:80 (LISTEN)
nginx     21606        www-data   10u  IPv4 105624      0t0  TCP *:443 (LISTEN)
nginx     21608        www-data    8u  IPv4 105622      0t0  TCP *:80 (LISTEN)
nginx     21608        www-data    9u  IPv6 105623      0t0  TCP *:80 (LISTEN)
nginx     21608        www-data   10u  IPv4 105624      0t0  TCP *:443 (LISTEN)
nginx     21610        www-data    8u  IPv4 105622      0t0  TCP *:80 (LISTEN)
nginx     21610        www-data    9u  IPv6 105623      0t0  TCP *:80 (LISTEN)
nginx     21610        www-data   10u  IPv4 105624      0t0  TCP *:443 (LISTEN)
nginx     21612        www-data    8u  IPv4 105622      0t0  TCP *:80 (LISTEN)
nginx     21612        www-data    9u  IPv6 105623      0t0  TCP *:80 (LISTEN)
nginx     21612        www-data   10u  IPv4 105624      0t0  TCP *:443 (LISTEN)

So my outstanding questions are:

- Where did the permissions fall apart?

- And do I really have to leave that log file with chmod 777?

[Brian Candler]

I notice a hangover your gunicorn.py. You have:

command = '/usr/local/bin/gunicorn'

when I'd have thought it should be

command = '/opt/netbox/venv/bin/gunicorn'

The sample configs at https://github.com/netbox-community/netbox/tree/master/contrib don't have this setting at all, so it's simplest just to remove it.

Why permissions fell apart for writing to /tmp, I'm not entirely sure.  It's clear that you were running netbox as the "ubuntu" user at least some of the time (because the /tmp/wgunicorn_XXXXXX file was owned by "ubuntu" , whereas systemd runs it as the "netbox" user; but each run should use a different filename.

Also your /tmp directory has the correct permissions: mode 1777 (rwx + sticky bit).  This means that any user can create files and directories there, but only the same user can delete them.

However on my system, I don't see these tmp files at all.

As for the permissions on /var/log/netbox, the real question is why it's trying to write to /var/log/netbox.log in the first place.

It looks like you've applied this "example" Django logging configuration verbatim:

https://netbox.readthedocs.io/en/stable/configuration/optional-settings/#logging

On most systems /var/log is owned by root, group syslog.  If you want Netbox to be able to log there, then create directory /var/log/netbox owned by netbox, and configure it log to /var/log/netbox/netbox.log

It might be better if the sample config pointed at /opt/netbox/log/netbox.log instead.

[Jason Kopacko]

Yea I removed those command option lines. They were from older install configs.

I will have to adjust my install config to do what you suggested for the netbox folder within the /var/log path.

Thank you for everything sir.