Netbox broken since intro of VENV

[Dirk Melvin]

I have been having issues with Netbox since the introduction of the VENV.

I attempted an upgrade at first, and my install did not like it. Worked through troubleshooting, then big dumb me eventually deleted our 'configuration.py' file, with no backup. It took me a while to finally get a successful way to back up our database, but once I had that I started from scratch. Fresh install of Ubuntu 18.04, fresh install of (at the time Netbox 2.7.11 following the Netbox install here: https://netbox.readthedocs.io/en/stable/installation/3-netbox/ ) but during the DJANGO installs I ran into the same issues that Adam Batey did here (https://groups.google.com/forum/#!topic/netbox-discuss/SGEfbCHDpzc), with errors during the 'building wheels' processes of various parts.

I get a '502 bad gateway' when I go my netbox URL, and my netbox services fail

# systemctl status netbox*

● netbox-rq.service - NetBox Request Queue Worker
   Loaded: loaded (/etc/systemd/system/netbox-rq.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2020-04-20 13:27:41 UTC; 3s ago
     Docs: https://netbox.readthedocs.io/en/stable/
  Process: 132100 ExecStart=/opt/netbox/venv/bin/python3 /opt/netbox/netbox/manage.py rqworker (code=exited, status=1/FAILURE)
 Main PID: 132100 (code=exited, status=1/FAILURE)

Apr 20 13:27:41 netbox systemd[1]: netbox-rq.service: Main process exited, code=exited, status=1/FAILURE
Apr 20 13:27:41 netbox systemd[1]: netbox-rq.service: Failed with result 'exit-code'.

● netbox.service - NetBox WSGI Service
   Loaded: loaded (/etc/systemd/system/netbox.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2020-04-20 13:27:26 UTC; 22s ago
     Docs: https://netbox.readthedocs.io/en/stable/
  Process: 132071 ExecStart=/opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi (code=exited, status=1/FAILURE)
 Main PID: 132071 (code=exited, status=1/FAILURE)

journalctl -eu netbox shows:

Apr 20 13:48:16 netbox gunicorn[134549]:   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Apr 20 13:48:16 netbox gunicorn[134549]:   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Apr 20 13:48:16 netbox gunicorn[134549]:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/netbox/netbox/wsgi.py", line 7, in <module>
Apr 20 13:48:16 netbox gunicorn[134549]:     application = get_wsgi_application()
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
Apr 20 13:48:16 netbox gunicorn[134549]:     django.setup(set_prefix=False)
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/django/__init__.py", line 19, in setup
Apr 20 13:48:16 netbox gunicorn[134549]:     configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
Apr 20 13:48:16 netbox gunicorn[134549]:     self._setup(name)
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
Apr 20 13:48:16 netbox gunicorn[134549]:     self._wrapped = Settings(settings_module)
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
Apr 20 13:48:16 netbox gunicorn[134549]:     mod = importlib.import_module(self.SETTINGS_MODULE)
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
Apr 20 13:48:16 netbox gunicorn[134549]:     return _bootstrap._gcd_import(name[level:], package, level)
Apr 20 13:48:16 netbox gunicorn[134549]:   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
Apr 20 13:48:16 netbox gunicorn[134549]:   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
Apr 20 13:48:16 netbox gunicorn[134549]:   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
Apr 20 13:48:16 netbox gunicorn[134549]:   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Apr 20 13:48:16 netbox gunicorn[134549]:   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Apr 20 13:48:16 netbox gunicorn[134549]:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/netbox/netbox/settings.py", line 394, in <module>
Apr 20 13:48:16 netbox gunicorn[134549]:     "Required parameter AUTH_LDAP_SERVER_URI is missing from ldap_config.py."
Apr 20 13:48:16 netbox gunicorn[134549]: django.core.exceptions.ImproperlyConfigured: Required parameter AUTH_LDAP_SERVER_URI is missing from ldap_config.py.
Apr 20 13:48:16 netbox gunicorn[134549]: [2020-04-20 13:48:16 +0000] [134571] [INFO] Worker exiting (pid: 134571)
Apr 20 13:48:16 netbox gunicorn[134549]: Traceback (most recent call last):
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 209, in run
Apr 20 13:48:16 netbox gunicorn[134549]:     self.sleep()
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 357, in sleep
Apr 20 13:48:16 netbox gunicorn[134549]:     ready = select.select([self.PIPE[0]], [], [], 1.0)
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Apr 20 13:48:16 netbox gunicorn[134549]:     self.reap_workers()
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Apr 20 13:48:16 netbox gunicorn[134549]:     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Apr 20 13:48:16 netbox gunicorn[134549]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Apr 20 13:48:16 netbox gunicorn[134549]: During handling of the above exception, another exception occurred:
Apr 20 13:48:16 netbox gunicorn[134549]: Traceback (most recent call last):
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/bin/gunicorn", line 11, in <module>
Apr 20 13:48:16 netbox gunicorn[134549]:     sys.exit(run())
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 58, in run
Apr 20 13:48:16 netbox gunicorn[134549]:     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/app/base.py", line 228, in run
Apr 20 13:48:16 netbox gunicorn[134549]:     super().run()
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/app/base.py", line 72, in run
Apr 20 13:48:16 netbox gunicorn[134549]:     Arbiter(self).run()
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 229, in run
Apr 20 13:48:16 netbox gunicorn[134549]:     self.halt(reason=inst.reason, exit_status=inst.exit_status)
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 342, in halt
Apr 20 13:48:16 netbox gunicorn[134549]:     self.stop()
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 393, in stop
Apr 20 13:48:16 netbox gunicorn[134549]:     time.sleep(0.1)
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Apr 20 13:48:16 netbox gunicorn[134549]:     self.reap_workers()
Apr 20 13:48:16 netbox gunicorn[134549]:   File "/opt/netbox/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Apr 20 13:48:16 netbox gunicorn[134549]:     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Apr 20 13:48:16 netbox gunicorn[134549]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Apr 20 13:48:16 netbox systemd[1]: netbox.service: Main process exited, code=exited, status=1/FAILURE
Apr 20 13:48:16 netbox systemd[1]: netbox.service: Failed with result 'exit-code'.

Does this give anyone clues as to what I am missing? What do I need to do to get this back up and running? Then is it safe to import my database?

Thanks,

Dirk

[Brian Candler]

You have to read the backtrace carefully, but the error is in there:

[Dirk Melvin]

Good catch on that error.

But why would that stop NETBOX from running? That should be a component problem, not a complete application stopper, IMO.

But I will work on fixing that, I am not quite ready for LDAP auth yet anyway.

Will report back...

Thanks,

Dirk

[Dirk Melvin]

Well, that did indeed get me further. I commented out all the lines I had commented out in the LDAP config file, just to have something there and NETBOX services are running now.

Now, I am seeing the error 111 connection refused messages, and I know I have seen those before, so I will go research those now and be back to fill in the details.

Thanks,

Dirk

[Dirk Melvin]

I changed my gunicorn config from 'locahost:8001' to my actual IP:8001, and now I get "STATIC MEDIA FAILURE" errors on different files (so far looks like .css files). I think I was here before on a previous install before I gave up. Maybe permission issues, but there is nothing I found in the install documents about permissions other than the netbox user, and I did that for the media folder.

Static Media Failure

The following static media file failed to load: bootstrap-3.4.1-dist/css/bootstrap.min.css

Check the following:

• manage.py collectstatic was run during the most recent upgrade. This installs the most recent iteration of each static file into the static root path.
• The HTTP service (e.g. nginx or Apache) is configured to serve files from the STATIC_ROOT path. Refer to the installation documentation for further guidance.
  • STATIC_URL: /static/
• The file bootstrap-3.4.1-dist/css/bootstrap.min.css exists in the static root directory and is readable by the HTTP process.

Click here to attempt loading NetBox again.

[Dirk Melvin]

I verified that if I run this

python3 manage.py runserver 0.0.0.0:8000 --insecure

I do successfully get to the NETBOX site and it looks as it should, has my header customization too.

[Dirk Melvin]

Really, nothing for the Static Media Failure error?

[Brian Candler]

It's been answered many times before.  Check your apache/nginx configs and logs, check "collectstatic", check permissions, check browser console logs.

[Dirk Melvin]

I ran the collectstatic command (3x now), the last 2 with zero changes.

I have just changed the /opt/netbox/netbox/static folder permissions (recursively) to 777.

I have restarted NGINX, and NETBOX services, even rebooted the VM itself.

I am still getting the STATIC MEDIA FAILURE errors. 

[Brian Candler]

Check your nginx logs.  Are you receiving any requests for /static/ ?  If so, what response code is being given to these requests? Are there any errors shown?

Check your browser's Javascript console.  It will show all the requests your page makes, including requests for static assets.  Are they going to /static/xxx ?  What response code does it show?

Are you using HTTPS?  It's possible that if you haven't configured things right, the main page will be HTTPS but it will be fetching the static assets over HTTP, and browsers frown on this sort of "mixed content".  If so, you need to add a header in your proxy to tell Netbox to generate https://... links instead of http://...

Anyway: I'm pretty sure the logs from server side and client side will show you what's going on.

[Dirk Melvin]

I am now able to get to http://netbox.my.local (it is if I go to :8001, that I get the STATIC MEDIA FAILURE errors). So I am confused on the purposes/differences between what is being used/served on port 80 vs 8001 and why. I get the above command for 'testing port 8000' as a temporary measure. But why the seeming 'rabbit hole' of 8001 if it is giving the media error, but 80 works? Still a config issue, maybe (probably) but why have both in the first place?

I now have a different issue, involving login, I will start a new thread about that one, I am SURE that is config related as there is very little to go on in the Netbox install documents for the django ldap setup.

[Brian Candler]

You don't need both to be accessible.

One of these (typically 80 or 443) is the nginx proxy frontend which users connect to.  The other (typically 8001) is the gunicorn application server.  You can bind this to 127.0.0.1, so it's not accessible to the outside world - only nginx needs to talk to it.

Therefore, if everything is working through the nginx frontend, then all is good.  gunicorn isn't supposed to serve the static assets.

If you do want to run a standalone webserver just for testing without nginx, then there's "python3 manage.py runserver" (which *will* serve the static assets as well - however it's a python dev webserver which is less efficient than gunicorn)

[Dirk Melvin]

Just wanted to post that our install of Netbox is back up and running now.

Still not entirely sure what the issue was that fixed my issue with port 80 giving me my original bad gateway error, and 8001 giving the static media failure errors.

I think changing the permissions on the static folder might have fixed port 80, but that doesn't make sense to me.

I disabled LDAP auth for now, but will revisit that soon, and I was able to restore my previous database without issue.

I have to say, the introduction of the VENV really threw us for a loop, I don't know why, and I don't understand what it really changed or what it really improved for us end users, but IMO a functionality change like this would have been better kept for a MAJOR version update, not a minor (and incremental) update.

Thank you to everyone in the group that helped us along the way. Look forward to participating and sharing more in the future.

[Emanuel Haine]

Hello guys,

I have the same problem with the "Static Media Failure" message.

Python 3.6.8

Netbox: v2.8.7

This is my vhost file:

server {
    listen 80;
    listen [::]:80;
    server_name server;
   
    return 301 https://$host$request_uri;

}

server {
    listen       443 ssl;
    listen       [::]:443 ssl;

    server_name server;

    ssl_certificate           ssl_file_bundle;
    ssl_certificate_key       ssl_file;

    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;

    access_log            <file>;
    error_log            <file>;

    client_max_body_size 25m;

    location /static/ {
        alias /opt/netbox/netbox/static/;
    }

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
    
    }
}

I performed chown nginx:nginx /opt/netbox -R and some SELinux contexts too.

I just open the page successfuly when a change the user to root on /etc/nginx/nginx.conf

To me it's doesn't make sense, because nginx has permissions to all the files.

I'm running netbox on a Centos7.8 VM and I tryed to run wihout ssl, but I got the same error.

Do you guys have an idea?