2.3.3 Does not run after installation from instructions

[Dan Cotruta]

Since my bug report was closed, I am posting this here. 2.3.3 does not work, if one follows the instructions to install.

We're using 2.3.0 and there are several bug fixes that would be very useful.

I fail to see how this is an acceptable bug to close, since it means that there is _something_ wrong with either the documentation or a package in requires.txt

### Issue type
[x] Bug report

### Environment

(netbox-venv) ***:/opt/netbox-2.3.3$ python --version
Python 3.5.2
(netbox-venv) ***:/opt/netbox-2.3.3$ pip --version
pip 10.0.1 from /opt/netbox-venv/lib/python3.5/site-packages/pip (python 3.5)

[[[ AT THIS STAGE, THE ENVIRONMENT IS COMPLETELY CLEAN ]]]

Following pip install -r requirements.txt package list:
asn1crypto==0.24.0
bcrypt==3.1.4
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
click==6.7
coreapi==2.3.3
coreschema==0.0.4
cryptography==2.2.2
Django==1.11.13
django-cors-headers==2.2.0
django-debug-toolbar==1.9.1
django-filter==1.1.0
django-js-asset==1.1.0
django-mptt==0.9.0
django-tables2==1.21.2
django-timezone-field==2.1
djangorestframework==3.8.2
drf-yasg==1.7.4
flex==6.13.2
future==0.16.0
graphviz==0.8.3
idna==2.6
inflection==0.3.1
itypes==1.1.0
Jinja2==2.10
jsonpointer==1.14
jsonschema==2.6.0
lxml==4.2.1
Markdown==2.6.11
MarkupSafe==1.0
natsort==5.3.2
ncclient==0.5.3
netaddr==0.7.18
openapi-codec==1.3.2
paramiko==2.4.1
Pillow==5.1.0
psycopg2-binary==2.7.4
py-gfm==0.1.3
pyasn1==0.4.3
pycparser==2.18
pycryptodome==3.6.1
PyNaCl==1.2.1
pytz==2018.4
PyYAML==3.12
requests==2.18.4
rfc3987==1.3.7
ruamel.yaml==0.15.37
six==1.11.0
sqlparse==0.2.4
strict-rfc3339==0.7
swagger-spec-validator==2.1.0
uritemplate==3.0.0
urllib3==1.22
validate-email==1.3
xmltodict==0.11.0

Gunicorn setup
gunicorn (version 19.8.1)

gunicorn_config.py
command = '/usr/bin/gunicorn'
pythonpath = '/opt/netbox-2.3.3/netbox'
bind = '127.0.0.1:8001'
workers = 3
user = 'www-data'

Trying to run:

(netbox-venv) ***:~$ sudo gunicorn -c /opt/netbox-2.3.3/netbox/gunicorn_config.py netbox.wsgi

Results in:

[2018-05-30 09:47:19 +0000] [7104] [INFO] Starting gunicorn 19.8.1
[2018-05-30 09:47:19 +0000] [7104] [INFO] Listening at: http://127.0.0.1:8001 (7104)
[2018-05-30 09:47:19 +0000] [7104] [INFO] Using worker: sync
[2018-05-30 09:47:19 +0000] [7107] [INFO] Booting worker with pid: 7107
[2018-05-30 09:47:19 +0000] [7108] [INFO] Booting worker with pid: 7108
[2018-05-30 09:47:19 +0000] [7111] [INFO] Booting worker with pid: 7111
[2018-05-30 09:47:20 +0000] [7111] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.5/dist-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/opt/netbox-2.3.3/netbox/netbox/wsgi.py", line 8, in 
application = get_wsgi_application()
File "/usr/local/lib/python3.5/dist-packages/django/core/wsgi.py", line 13, in get_wsgi_application
django.setup(set_prefix=False)
File "/usr/local/lib/python3.5/dist-packages/django/init.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.5/dist-packages/django/apps/registry.py", line 108, in populate
app_config.import_models()
File "/usr/local/lib/python3.5/dist-packages/django/apps/config.py", line 202, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/opt/netbox-2.3.3/netbox/dcim/models.py", line 21, in 
from extras.rpc import RPC_CLIENTS
File "/opt/netbox-2.3.3/netbox/extras/rpc.py", line 6, in 
import paramiko
File "/usr/local/lib/python3.5/dist-packages/paramiko/init.py", line 22, in 
from paramiko.transport import SecurityOptions, Transport
File "/usr/local/lib/python3.5/dist-packages/paramiko/transport.py", line 57, in 
from paramiko.ed25519key import Ed25519Key
File "/usr/local/lib/python3.5/dist-packages/paramiko/ed25519key.py", line 22, in 
import nacl.signing
File "/usr/local/lib/python3.5/dist-packages/nacl/signing.py", line 19, in 
import nacl.bindings
File "/usr/local/lib/python3.5/dist-packages/nacl/bindings/init.py", line 17, in 
from nacl.bindings.crypto_box import (
File "/usr/local/lib/python3.5/dist-packages/nacl/bindings/crypto_box.py", line 27, in 
crypto_box_SEEDBYTES = lib.crypto_box_seedbytes()
AttributeError: cffi library '_sodium' has no function, constant or global variable named 'crypto_box_seedbytes'
[2018-05-30 09:47:20 +0000] [7111] [INFO] Worker exiting (pid: 7111)
[2018-05-30 09:47:20 +0000] [7107] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.5/dist-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/opt/netbox-2.3.3/netbox/netbox/wsgi.py", line 8, in 
application = get_wsgi_application()
File "/usr/local/lib/python3.5/dist-packages/django/core/wsgi.py", line 13, in get_wsgi_application
django.setup(set_prefix=False)
File "/usr/local/lib/python3.5/dist-packages/django/init.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.5/dist-packages/django/apps/registry.py", line 108, in populate
app_config.import_models()
File "/usr/local/lib/python3.5/dist-packages/django/apps/config.py", line 202, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/opt/netbox-2.3.3/netbox/dcim/models.py", line 21, in 
from extras.rpc import RPC_CLIENTS
File "/opt/netbox-2.3.3/netbox/extras/rpc.py", line 6, in 
import paramiko
File "/usr/local/lib/python3.5/dist-packages/paramiko/init.py", line 22, in 
from paramiko.transport import SecurityOptions, Transport
File "/usr/local/lib/python3.5/dist-packages/paramiko/transport.py", line 57, in 
from paramiko.ed25519key import Ed25519Key
File "/usr/local/lib/python3.5/dist-packages/paramiko/ed25519key.py", line 22, in 
import nacl.signing
File "/usr/local/lib/python3.5/dist-packages/nacl/signing.py", line 19, in 
import nacl.bindings
File "/usr/local/lib/python3.5/dist-packages/nacl/bindings/init.py", line 17, in 
from nacl.bindings.crypto_box import (
File "/usr/local/lib/python3.5/dist-packages/nacl/bindings/crypto_box.py", line 27, in 
crypto_box_SEEDBYTES = lib.crypto_box_seedbytes()
AttributeError: cffi library '_sodium' has no function, constant or global variable named 'crypto_box_seedbytes'
[2018-05-30 09:47:20 +0000] [7107] [INFO] Worker exiting (pid: 7107)
[2018-05-30 09:47:20 +0000] [7108] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.5/dist-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/opt/netbox-2.3.3/netbox/netbox/wsgi.py", line 8, in 
application = get_wsgi_application()
File "/usr/local/lib/python3.5/dist-packages/django/core/wsgi.py", line 13, in get_wsgi_application
django.setup(set_prefix=False)
File "/usr/local/lib/python3.5/dist-packages/django/init.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.5/dist-packages/django/apps/registry.py", line 108, in populate
app_config.import_models()
File "/usr/local/lib/python3.5/dist-packages/django/apps/config.py", line 202, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/opt/netbox-2.3.3/netbox/dcim/models.py", line 21, in 
from extras.rpc import RPC_CLIENTS
File "/opt/netbox-2.3.3/netbox/extras/rpc.py", line 6, in 
import paramiko
File "/usr/local/lib/python3.5/dist-packages/paramiko/init.py", line 22, in 
from paramiko.transport import SecurityOptions, Transport
File "/usr/local/lib/python3.5/dist-packages/paramiko/transport.py", line 57, in 
from paramiko.ed25519key import Ed25519Key
File "/usr/local/lib/python3.5/dist-packages/paramiko/ed25519key.py", line 22, in 
import nacl.signing
File "/usr/local/lib/python3.5/dist-packages/nacl/signing.py", line 19, in 
import nacl.bindings
File "/usr/local/lib/python3.5/dist-packages/nacl/bindings/init.py", line 17, in 
from nacl.bindings.crypto_box import (
File "/usr/local/lib/python3.5/dist-packages/nacl/bindings/crypto_box.py", line 27, in 
crypto_box_SEEDBYTES = lib.crypto_box_seedbytes()
AttributeError: cffi library '_sodium' has no function, constant or global variable named 'crypto_box_seedbytes'
[2018-05-30 09:47:20 +0000] [7108] [INFO] Worker exiting (pid: 7108)
[2018-05-30 09:47:20 +0000] [7104] [INFO] Shutting down: Master
[2018-05-30 09:47:20 +0000] [7104] [INFO] Reason: Worker failed to boot.

[Brian Candler]

You haven't specified which OS you installed under, and you haven't specified where you got gunicorn from (it's not in your pip list)

It appears that you are mixing a locally-built python (/usr/local/lib) with a system-provided gunicorn (/usr/bin).  If you're building python3 from source, you need to build a corresponding gunicorn as well.  Maybe just "pip install gunicorn" inside your virtualenv will fix your problem.

If you were using Ubuntu/Debian and python3, then you would need "gunicorn3" not "gunicorn".  But in that case you should be using the system python3/pip3 anyway.

[Brian Candler]

http://docs.gunicorn.org/en/stable/deploy.html#using-virtualenv

[Dan Cotruta]

Yes, I should have mentioned this information.

ubuntu@***:/opt$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description:    Ubuntu 16.04.4 LTS

Release:        16.04

Codename:       xenial

(netbox-venv) ***:/opt$ which gunicorn

/opt/netbox-venv/bin/gunicorn

(netbox-venv) ***:/opt$ gunicorn -v

gunicorn (version 19.7.1)

Installed into the venv via "pip install gunicorn"

[Brian Candler]

I see you have "command = '/usr/bin/gunicorn'" in your config - does that reference your one in /opt?  Also, is there a particular reason not to use the Ubuntu system python3?

[Brian Candler]

OK, I think I see the problem.

You said you installed packages in a virtualenv:

$ pip --version
pip 10.0.1 from /opt/netbox-venv/lib/python3.5/site-packages/pip

But you actually have packages in /usr/local/lib/python3.5/dist-packages/ in your backtrace.

At first I thought this had meant you'd installed python3 from source, but now I think you have installed packages in your system pip rather than your virtualenv pip.  Or maybe you have them installed in both; but when gunicorn runs it's using the system packages.

So if you're going to use a virtualenv, it's important that everything goes into the virtualenv (both packages from requirements.txt and anything else you install like gunicorn), so it must be activated before doing those pip installs; and it must be activated when you run gunicorn too.

And it's also important to check for package installation errors when pip runs, because unfortunately it does tend to skip over failures.  In particular, you need all the right development libraries installed so that packages which link to C libraries can build successfully:

apt-get install -y python3 python3-dev python3-setuptools build-essential libxml2-dev libxslt1-dev libffi-dev graphviz libpq-dev libssl-dev zlib1g-dev