Upgrade from 2.7.12 to Latest

[Deniss S]

Please, help me with updating NetBox 2.7.12 to Latest Netbox. 

When I perform " git checkout" I receive this error:

error: Your local changes to the following files would be overwritten by checkout:

        netbox/netbox/settings.py

Please commit your changes or stash them before you switch branches.

Aborting

If I use force than NetBox webpage is not opening - 512 Proxy error. 

And I could not find how to solve this. 

[Brian Candler]

What *exact* command did you type?  Was it "git checkout v<version>" ?

When I perform " git checkout" I receive this error:

error: Your local changes to the following files would be overwritten by checkout:

        netbox/netbox/settings.py

Please commit your changes or stash them before you switch branches.

You should not have changed that file: the file you should edit is called "configuration.py".

First you need to find out how that file was altered.  The following command will tell you

git diff

Then probably the best thing to do is to take a copy just in case, then reset it back to its original unaltered version:

cp netbox/netbox/settings.py netbox/netbox/settings.py.broken

git checkout netbox/netbox/settings.py

After that the "git checkout vX.X.X" to switch tags will work.  This is clearly the right fix if the change to settings.py is only minor (e.g. someone had opened it in an editor, added a few spaces accidentally, and then saved it)

However if there are significant differences to settings.py then you'll have to work out what the intention of those changes was, and then work out how to apply those changes correctly to Netbox.

You *could* attempt to merge the changes back into the new version:

git stash

git checkout vX.X.X

git stash apply

But I don't think this is a good idea.  Running with a modified version of settings.py is not a supported Netbox configuration; and there's certainly no guarantee that any changes you made for v2.7.12 would do the right thing with any later version.  Plus, the changes might not even merge cleanly in which case you'd be left with a merge conflict to fix.

As long as you've taken a backup of your postgres database before starting the upgrade, as well as any images/scripts/reports, you should at least be able to revert to how it was before.

[Deniss S]

Thanks for your response. 

I was previously updated NetBox to 2.7.12 using this commands and everything was OK:

  cd /opt/netbox

  git checkout master

  git pull origin master

  git status

  ./upgrade.sh

  systemctl restart supervisord

  supervisorctl restart netbox

  systemctl restart apache2

  

So. using git diff i receive this output:

  root@netbox:/opt/netbox# git diff

  diff --git a/netbox/netbox/settings.py b/netbox/netbox/settings.py

  index ba2060ce..9b1226c3 100644

  --- a/netbox/netbox/settings.py

  +++ b/netbox/netbox/settings.py

  @@ -97,7 +97,7 @@ NAPALM_TIMEOUT = getattr(configuration, 'NAPALM_TIMEOUT', 30)

  NAPALM_USERNAME = getattr(configuration, 'NAPALM_USERNAME', '')

  PAGINATE_COUNT = getattr(configuration, 'PAGINATE_COUNT', 50)

  PREFER_IPV4 = getattr(configuration, 'PREFER_IPV4', False)

 -RELEASE_CHECK_URL = getattr(configuration, 'RELEASE_CHECK_URL', None)

 +RELEASE_CHECK_URL = getattr(configuration, 'RELEASE_CHECK_URL', 'https://api.github.com/repos/netbox-community/netbox/releases')

  RELEASE_CHECK_TIMEOUT = getattr(configuration, 'RELEASE_CHECK_TIMEOUT', 24 * 3600)

  REPORTS_ROOT = getattr(configuration, 'REPORTS_ROOT', os.path.join(BASE_DIR, 'reports')).rstrip('/')

  SCRIPTS_ROOT = getattr(configuration, 'SCRIPTS_ROOT', os.path.join(BASE_DIR, 'scripts')).rstrip('/')

I perform : 

  cp netbox/netbox/settings.py netbox/netbox/settings.py.broken

  git checkout netbox/netbox/settings.py 

After it, I perform mentioned update commands.

Everything was went fine, except Web page is not starting. 

  Proxy Error

  The proxy server received an invalid response from an upstream server.

  The proxy server could not handle the request

  Reason: Error reading from remote server

  systemctl status  supervisor

  Sep 13 17:39:50 netbox supervisord[31760]: 2020-09-13 17:39:50,059 INFO spawned: 'netbox' with pid 33760

  Sep 13 17:39:51 netbox supervisord[31760]: 2020-09-13 17:39:51,420 INFO success: netbox entered RUNNING state, process has stayed up for > than 1

  Sep 13 17:39:54 netbox supervisord[31760]: 2020-09-13 17:39:54,401 INFO exited: netbox (exit status 3; not expected)

  Sep 13 17:39:55 netbox supervisord[31760]: 2020-09-13 17:39:55,404 INFO spawned: 'netbox' with pid 33778

  Sep 13 17:39:56 netbox supervisord[31760]: 2020-09-13 17:39:56,804 INFO success: netbox entered RUNNING state, process has stayed up for > than 1

  Sep 13 17:39:59 netbox supervisord[31760]: 2020-09-13 17:39:59,659 INFO exited: netbox (exit status 3; not expected)

  Sep 13 17:40:00 netbox supervisord[31760]: 2020-09-13 17:40:00,662 INFO spawned: 'netbox' with pid 33796

  Sep 13 17:40:02 netbox supervisord[31760]: 2020-09-13 17:40:02,216 INFO success: netbox entered RUNNING state, process has stayed up for > than 1

  Sep 13 17:40:04 netbox supervisord[31760]: 2020-09-13 17:40:04,664 INFO exited: netbox (exit status 1; not expected)

  Sep 13 17:40:05 netbox supervisord[31760]: 2020-09-13 17:40:05,666 INFO spawned: 'netbox' with pid 33814

When I try to perform this action : sudo python3 manage.py runserver 0.0.0.0:8000 --insecure I get:

  Exception in thread django-main-thread:

  Traceback (most recent call last):

    File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner

      self.run()

    File "/usr/lib/python3.6/threading.py", line 864, in run

      self._target(*self._args, **self._kwargs)

    File "/usr/local/lib/python3.6/dist-packages/django/utils/autoreload.py", line 54, in wrapper

      fn(*args, **kwargs)

    File "/usr/local/lib/python3.6/dist-packages/django/core/management/commands/runserver.py", line 109, in inner_run

      autoreload.raise_last_exception()

    File "/usr/local/lib/python3.6/dist-packages/django/utils/autoreload.py", line 77, in raise_last_exception

      raise _exception[1]

    File "/usr/local/lib/python3.6/dist-packages/django/core/management/__init__.py", line 337, in execute

      autoreload.check_errors(django.setup)()

    File "/usr/local/lib/python3.6/dist-packages/django/utils/autoreload.py", line 54, in wrapper

      fn(*args, **kwargs)

    File "/usr/local/lib/python3.6/dist-packages/django/__init__.py", line 24, in setup

      apps.populate(settings.INSTALLED_APPS)

    File "/usr/local/lib/python3.6/dist-packages/django/apps/registry.py", line 114, in populate

      app_config.import_models()

    File "/usr/local/lib/python3.6/dist-packages/django/apps/config.py", line 211, in import_models

      self.models_module = import_module(models_module_name)

    File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module

      return _bootstrap._gcd_import(name[level:], package, level)

    File "<frozen importlib._bootstrap>", line 994, in _gcd_import

    File "<frozen importlib._bootstrap>", line 971, in _find_and_load

    File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

    File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

    File "<frozen importlib._bootstrap_external>", line 678, in exec_module

    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

    File "/opt/netbox/netbox/circuits/models.py", line 8, in <module>

      from dcim.models import CableTermination

    File "/opt/netbox/netbox/dcim/models/__init__.py", line 1, in <module>

      from .device_component_templates import *

    File "/opt/netbox/netbox/dcim/models/device_component_templates.py", line 7, in <module>

      from extras.models import ObjectChange

    File "/opt/netbox/netbox/extras/models/__init__.py", line 1, in <module>

      from .change_logging import ChangeLoggedModel, ObjectChange

    File "/opt/netbox/netbox/extras/models/change_logging.py", line 48, in <module>

      class ObjectChange(models.Model):

    File "/opt/netbox/netbox/extras/models/change_logging.py", line 106, in ObjectChange

      object_data = models.JSONField(

  AttributeError: module 'django.db.models' has no attribute 'JSONField'

воскресенье, 13 сентября 2020 г. в 01:25:43 UTC+3, Brian Candler:

[Brian Candler]

On Sunday, 13 September 2020 18:44:05 UTC+1, Deniss S wrote:

Everything was went fine, except Web page is not starting. 

  Proxy Error

  The proxy server received an invalid response from an upstream server.

  The proxy server could not handle the request

  Reason: Error reading from remote server

That is just code for "netbox backend server isn't running"

 

When I try to perform this action : sudo python3 manage.py runserver 0.0.0.0:8000 --insecure I get:

  Exception in thread django-main-thread:

  Traceback (most recent call last):

    File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner

      self.run()

    File "/usr/lib/python3.6/threading.py", line 864, in run

      self._target(*self._args, **self._kwargs)

    File "/usr/local/lib/python3.6/dist-packages/django/utils/autoreload.py", line 54, in wrapper

Recent versions of Netbox must run inside a python virtual environment.  If you run it the way you just did, you're using the version of django which is installed globally in your system (/usr/local/lib), so it won't work.

You'll need to do something like this:

sudo -s -u www-data   # or whatever username the netbox process needs to run as

cd /opt/netbox

. venv/bin/activate

python3 netbox/manage.py runserver 0.0.0.0:8000 --insecure

If that works, I think your most likely problem is with your supervisord configuration, can you show it please?  it needs to be running /opt/netbox/venv/bin/gunicorn, not /usr/local/bin/gunicorn

[Deniss S]

So Running this:

  root@netbox:/opt/netbox# . venv/bin/activate                                    

  (venv) root@netbox:/opt/netbox# python3 netbox/manage.py runserver 0.0.0.0:8000 --insecure

  Performing system checks...

  System check identified no issues (0 silenced).

  September 14, 2020 - 20:22:17

  Django version 3.1, using settings 'netbox.settings'

  Starting development server at http://0.0.0.0:8000/

  Quit the server with CONTROL-C.

  [14/Sep/2020 20:25:04] "GET / HTTP/1.1" 200 31379

  [14/Sep/2020 20:25:04] "GET /static/bootstrap-3.4.1-dist/css/bootstrap.min.css HTTP/1.1" 200 121457

  [14/Sep/2020 20:25:04] "GET /static/select2-4.0.13/dist/css/select2.min.css HTTP/1.1" 200 14966

  [14/Sep/2020 20:25:04] "GET /static/font-awesome-4.7.0/css/font-awesome.min.css HTTP/1.1" 200 31000

  [14/Sep/2020 20:25:04] "GET /static/jquery-ui-1.12.1/jquery-ui.css HTTP/1.1" 200 37326

  [14/Sep/2020 20:25:04] "GET /static/css/base.css?v2.9.3 HTTP/1.1" 200 6568

  [14/Sep/2020 20:25:04] "GET /static/jquery/jquery-3.5.1.min.js HTTP/1.1" 200 89476

  [14/Sep/2020 20:25:04] "GET /static/flatpickr-4.6.3/themes/light.css HTTP/1.1" 200 18996

  [14/Sep/2020 20:25:04] "GET /static/select2-bootstrap-0.1.0-beta.10/select2-bootstrap.min.css HTTP/1.1" 200 16792

  [14/Sep/2020 20:25:04] "GET /static/select2-4.0.13/dist/js/select2.min.js HTTP/1.1" 200 70891

  [14/Sep/2020 20:25:04] "GET /static/clipboard.js/clipboard-2.0.6.min.js HTTP/1.1" 200 10454

  [14/Sep/2020 20:25:04] "GET /static/bootstrap-3.4.1-dist/js/bootstrap.min.js HTTP/1.1" 200 39680

  [14/Sep/2020 20:25:04] "GET /static/flatpickr-4.6.3/flatpickr.min.js HTTP/1.1" 200 48518

  [14/Sep/2020 20:25:04] "GET /static/jquery-ui-1.12.1/jquery-ui.min.js HTTP/1.1" 200 253669

  [14/Sep/2020 20:25:04] "GET /static/js/forms.js?v2.9.3 HTTP/1.1" 200 18595

  [14/Sep/2020 20:25:04] "GET /static/img/netbox_logo.svg HTTP/1.1" 200 4719

  [14/Sep/2020 20:25:04] "GET /static/font-awesome-4.7.0/fonts/fontawesome-webfont.woff2?v=4.7.0 HTTP/1.1" 200 77160

  [14/Sep/2020 20:25:04] "GET /static/img/netbox.ico HTTP/1.1" 200 1174

Supervisor: 

  root@netbox:/home/monitoring# cat /etc/supervisor/conf.d/netbox.conf

  [program:netbox]

  command = gunicorn -c /opt/netbox/gunicorn.py netbox.wsgi

  directory = /opt/netbox/netbox/

  

понедельник, 14 сентября 2020 г. в 11:28:56 UTC+3, Brian Candler:

[Brian Candler]

  command = gunicorn -c /opt/netbox/gunicorn.py netbox.wsgi

is wrong.  It will be looking in $PATH for gunicorn, so it will find /usr/local/bin/gunicorn, which doesn't use the virtualenv (it will use global packages instead).

Change it to:

  command = /opt/netbox/venv/bin/gunicorn -c /opt/netbox/gunicorn.py netbox.wsgi

[Deniss S]

Thank you very much. Seems it is solved the problem. 

But as I understand, there was a lot of changes in Netbox. LDAP changes, HTTPS, systemctl. 

Will it be better to reinstall to the same version using new approach, import backup data and update to latest version? 

вторник, 15 сентября 2020 г. в 10:10:32 UTC+3, Brian Candler:

[Brian Candler]

On Friday, 18 September 2020 17:36:14 UTC+1, Deniss S wrote:

Thank you very much. Seems it is solved the problem. 

But as I understand, there was a lot of changes in Netbox. LDAP changes, HTTPS, systemctl. 

The instructions now recommend using systemd instead of supervisord, but you are not required to change.  I think it's a good idea though, as it's one less piece of software to maintain.  Plus, the packaged versions of supervisord tend to use obsolete Python2, and it's nice to be able to remove it.

I don't know about LDAP changes, as I don't use LDAP.  There are no HTTPS changes, because Netbox doesn't do HTTPS; this is a function of the proxy you put in front (e.g. Apache or Nginx).

 

Will it be better to reinstall to the same version using new approach, import backup data and update to latest version? 

Certainly it's a good, safe way to approach the upgrade.  It also gives you an opportunity to review any scripts, reports, and API integrations you have, because all these are likely to change.  However it sounds like you've completed the upgrade successfully now anyway.

I have been doing updates in-situ, but always sure to backup beforehand.

The changes from 2.7.12 to 2.9.3 are not too huge.  You already had the virtualenv change in 2.7.9.  There are some settings in configuration.py that need to change.  The main problems are incompatible changes in the REST API and the data model, which will affect any integrations you have done.

[Deniss S]

OK, Thanks for reply. 

I will try to use received acknowledge if further updates. 

;)

 

пятница, 18 сентября 2020 г. в 20:27:30 UTC+3, Brian Candler:

[Deniss S]

But I have maybe one more question.

If I export from 2.9.3 version (test) and import to 2.9.3 version (production) using WEB I receive an error:

Enter the list of column headers followed by one line per record to be imported, using commas to separate values. Multi-line data and values containing commas may be wrapped in double quotes.

• Unexpected column header "rack_name" found.
  

I can see it has only rack column. 

Can there be any error in versions after update? 

пятница, 18 сентября 2020 г. в 21:31:57 UTC+3, Deniss S:

[Anders Hagman]

Hi

> 18 sep. 2020 kl. 19:27 skrev Brian Candler <b.ca...@pobox.com>:
>
> The instructions now recommend using systemd instead of supervisord, but you are not required to change. I think it's a good idea though, as it's one less piece of software to maintain. Plus, the packaged versions of supervisord tend to use obsolete Python2, and it's nice to be able to remove it.

We non linux users still exist and we use supervisord. It has been py2.x clean for a while and working good.

/Anders

[Brian Candler]

On Friday, 18 September 2020 19:51:01 UTC+1, Deniss S wrote:

If I export from 2.9.3 version (test) and import to 2.9.3 version (production) using WEB I receive an error:

Enter the list of column headers followed by one line per record to be imported, using commas to separate values. Multi-line data and values containing commas may be wrapped in double quotes.

• Unexpected column header "rack_name" found.
  

I am not sure what you mean by "export .. import .. using WEB".

Do you mean using CSV export and import?  Unfortunately, these are not suitable for transferring data from one Netbox instance to another.  The export and import CSV formats are not compatible with each other.

The only useful ways to transfer data from one Netbox to another are:

* replicate the entire postgres database

* use the REST API

Even using the REST API, you need to remember that the only guaranteed way to reference an object uniquely is via its database ID (for example: not all devices have names).  The database ID in the source and destination databases will be different, since you can't control this on object creation.  So it's up to you to maintain a third, mapping database which links the ID in the old database to the ID in the new database.

[Deniss S]

Yes, I mentioned CSV. 

Needed to copy some Devices. 

суббота, 19 сентября 2020 г. в 11:14:36 UTC+3, Brian Candler: