Receiveing a proxy error when atremptog to login to netbox after upgrading from 2.7.12 to 2.9.11

[Travis Johnson]

Hello,

I am taking over an existing netbox implementation and performed an upgrade from 2.7.12 to 2.9.11. The server is CentOS 7.9 running supervisord.

I ran supervisorctl stop netbox

cd'd to /opt/netbox

ran git pull

ran git checkout tags/v2.9.11

ran supervisorctl start netbox

I attempted to navigate to the page and got this error:
"Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

Reason: Error reading from remote server"

I checked the supervisord logs and the netbox service cannot stay running:

Supervisord logs
=================
2021-02-04 15:56:03,469 INFO spawned: 'netbox' with pid 17734
2021-02-04 15:56:04,897 INFO success: netbox entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-02-04 15:56:05,221 INFO exited: netbox (exit status 1; not expected)
2021-02-04 15:56:06,224 INFO spawned: 'netbox' with pid 17745
2021-02-04 15:56:07,656 INFO success: netbox entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-02-04 15:56:08,040 INFO exited: netbox (exit status 1; not expected)
2021-02-04 15:56:09,043 INFO spawned: 'netbox' with pid 17756
2021-02-04 15:56:10,560 INFO success: netbox entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-02-04 15:56:10,811 INFO exited: netbox (exit status 1; not expected)
2021-02-04 15:56:11,814 INFO spawned: 'netbox' with pid 17768

I checked the apache error logs and saw this:

HTTPD Error logs
================
[Thu Feb 04 16:26:14.012716 2021] [proxy_http:error] [pid 5413] (104)Connection reset by peer: [client 10.70.255.214:58039] AH01102: error reading status line from remote server 127.0.0.1:8001
[Thu Feb 04 16:26:14.012855 2021] [proxy:error] [pid 5413] [client 10.70.255.214:58039] AH00898: Error reading from remote server returned by /login/
[Thu Feb 04 16:26:14.265869 2021] [proxy:error] [pid 5413] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8001 (127.0.0.1) failed
[Thu Feb 04 16:26:14.265906 2021] [proxy:error] [pid 5413] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 60s
[Thu Feb 04 16:26:14.265913 2021] [proxy_http:error] [pid 5413] [client 10.70.255.214:58039] AH01114: HTTP: failed to make connection to backend: 127.0.0.1, referer: https://netbox-lab.company.com/login/?next=/
[Thu Feb 04 16:26:17.275865 2021] [proxy:error] [pid 14424] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8001 (127.0.0.1) failed
[Thu Feb 04 16:26:17.275935 2021] [proxy:error] [pid 14424] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 60s
[Thu Feb 04 16:26:17.275952 2021] [proxy_http:error] [pid 14424] [client 10.70.255.214:58044] AH01114: HTTP: failed to make connection to backend: 127.0.0.1

Apache may be having issues communicating with gunicorn? (I'm not too familiar with the software stack here.

I reviewed some previous posts and reviewed the docs and I gave this a try:

Changed to the apache user
ran cd /opt/netbox
ran . venv/bin/activate
python3 netbox/manage.py runserver 0.0.0.0:8000 --insecure

============================================
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/netbox/venv/lib64/python3.6/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/opt/netbox/venv/lib64/python3.6/site-packages/django/core/management/commands/runserver.py", line 109, in inner_run
    autoreload.raise_last_exception()
  File "/opt/netbox/venv/lib64/python3.6/site-packages/django/utils/autoreload.py", line 76, in raise_last_exception
    raise _exception[1]
  File "/opt/netbox/venv/lib64/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    autoreload.check_errors(django.setup)()
  File "/opt/netbox/venv/lib64/python3.6/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/opt/netbox/venv/lib64/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/opt/netbox/venv/lib64/python3.6/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/opt/netbox/venv/lib64/python3.6/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/netbox/netbox/circuits/models.py", line 8, in <module>
    from dcim.models import CableTermination
  File "/opt/netbox/netbox/dcim/models/__init__.py", line 1, in <module>
    from .device_component_templates import *
  File "/opt/netbox/netbox/dcim/models/device_component_templates.py", line 7, in <module>
    from extras.models import ObjectChange
  File "/opt/netbox/netbox/extras/models/__init__.py", line 1, in <module>
    from .change_logging import ChangeLoggedModel, ObjectChange
  File "/opt/netbox/netbox/extras/models/change_logging.py", line 48, in <module>
    class ObjectChange(models.Model):
  File "/opt/netbox/netbox/extras/models/change_logging.py", line 106, in ObjectChange
    object_data = models.JSONField(
AttributeError: module 'django.db.models' has no attribute 'JSONField'

Any ideas as to what may be the cause of this issue?

Thank you!

[Brian Candler]

On Thursday, 4 February 2021 at 23:22:51 UTC TravJ wrote:

Reason: Error reading from remote server"

That generally means "the target server (Netbox) isn't running".

 

2021-02-04 15:56:10,560 INFO success: netbox entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

2021-02-04 15:56:10,811 INFO exited: netbox (exit status 1; not expected)
2021-02-04 15:56:11,814 INFO spawned: 'netbox' with pid 17768

That confirms it.

 

Apache may be having issues communicating with gunicorn?

gunicorn simply isn't running (i.e. it started and immediately crashed).

 

 File "/opt/netbox/netbox/extras/models/change_logging.py", line 106, in ObjectChange

    object_data = models.JSONField(
AttributeError: module 'django.db.models' has no attribute 'JSONField'

And there's the crash.

 

Any ideas as to what may be the cause of this issue?

It looks like the upgrade has not been completed properly - in particular, the virtual environment seems to have the wrong version of django.  You can compare "pip3 list" with the contents of requirements.txt to confirm this.

However, this should be resolved automatically simply by running ./upgrade.sh, which is the main upgrade step.  If you didn't omit this, then did it show an error when it ran?