Upgrade python 2.7 and netbox

[Nguyen Quoc Van]

Hi,

I'm using Netbox 2.7.12 and python 2.7.12 on Ubuntu 16.04

How can i upgrade Python to 3.6 to upgrade Netbox? I tried some method and upgraded python to 3.6 but when i upgrade Netbox, it showed error

Installing collected packages: pip

  Found existing installation: pip 8.1.1

    Uninstalling pip-8.1.1:

      Successfully uninstalled pip-8.1.1

Successfully installed pip-21.0.1

Traceback (most recent call last):

  File "/opt/netbox/netbox/venv/bin/pip", line 7, in <module>

    from pip._internal.cli.main import main

Regards

[Nguyễn Trọng Vĩnh (Tovin Seven)]

Hi,

Can you provide a full stack trace and list of commands that you typed?

Inside your venv, can you show the output of:

• which python
• which pip
• pip --version

Tovin!

--------

Nguyễn Trọng Vĩnh (Tovin Seven)

Email: tov...@gmail.com

--
You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/8cc0e4ee-2565-499e-a355-e02ba6d0f8a0n%40googlegroups.com.

[Nguyen Quoc Van]

Hi,

nqvan@dcim:~$ which python

/usr/bin/python

nqvan@dcim:~$ which pip

/usr/local/bin/pip

nqvan@dcim:~$ pip --version

pip 18.1 from /usr/local/lib/python3.5/dist-packages/pip-18.1-py3.5.egg/pip (python 3.5)

nqvan@dcim:~$ 

Vào lúc 17:40:10 UTC+7 ngày Thứ Ba, 2 tháng 3, 2021, tov...@gmail.com đã viết:

[Nguyễn Trọng Vĩnh (Tovin Seven)]

Hi,

Looks like you're not using python from venv and the versions of both py and pip is not the same with your last email.

Please try again with py in venv.

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/ff04e64e-4b03-4b2b-8917-e4cd77c92208n%40googlegroups.com.

[Nguyen Quoc Van]

Hi

I have 2 python version (2.7 and 3.5) but 2.7 is default

nqvan@dcim:~$ ll -ls /usr/bin/python

0 lrwxrwxrwx 1 root root 9 Nov 24  2017 /usr/bin/python -> python2.7*

nqvan@dcim:~$ 

pip also mapped with python 3.5

How can i do? Remove python 2.7 and old pip?

Vào lúc 20:09:52 UTC+7 ngày Thứ Ba, 2 tháng 3, 2021, tov...@gmail.com đã viết:

[Nguyễn Trọng Vĩnh (Tovin Seven)]

Just using the correct venv (with py36 as your original email).

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/10897f5e-f013-4b67-ba9d-82258d3814d3n%40googlegroups.com.

[Nguyen Quoc Van]

I tried install python 3.6 default but when i upgrade Netbox it show error with old pip.

Vào lúc 20:30:48 UTC+7 ngày Thứ Ba, 2 tháng 3, 2021, tov...@gmail.com đã viết:

[Brian Candler]

When Netbox abandoned python 3.5 I found it too hard to keep working on Ubuntu 16.04; I ended up rebuilding systems using Ubuntu 18.04.

However, I think you should take the opportunity to go straight to Ubuntu 20.04.  This is because the Netbox team has already decided to drop python 3.6 soon: #5532.

[Jeremy Stretch]

> When Netbox abandoned python 3.5

If by "abandoned" you mean we moved to Django 3.0, which dropped support for Python 3.5, yes.

There's no need to move to a new Ubuntu release for v2.11, though. Installing Python 3.7 is quite simple:

sudo apt install software-properties-common

sudo add-apt-repository ppa:deadsnakes/ppa

sudo apt install python3.7

The virtual environment in which NetBox runs can use this Python without interfering with the system default version.

Jeremy Stretch

Founder

NetVerity, LLC

--

You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/c855b0bf-3723-4cf8-95bd-d92811863d40n%40googlegroups.com.

[Nguyen Quoc Van]

Hi,

I had installed Ubuntu 20 and  lastest Netbox version.

How can i migrate data from Netbox 2.7.12 to lastest version?

Regards

Vào lúc 01:22:37 UTC+7 ngày Thứ Năm, 4 tháng 3, 2021, jer...@netverity.dev đã viết:

[Nguyễn Trọng Vĩnh (Tovin Seven)]

For upgrading netbox, you can follow these steps in the docs: https://netbox.readthedocs.io/en/stable/installation/upgrading/

Notes: Please read the content of the upgrade.sh script before running it. https://github.com/netbox-community/netbox/blob/develop/upgrade.sh

Tovin!

--------

Nguyễn Trọng Vĩnh (Tovin Seven)

Email: tov...@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/e4f8a709-e773-4c07-b1e9-7d63aa7c9e74n%40googlegroups.com.

[Brian Candler]

Just follow the instructions for "replicating Netbox" - i.e. make a postgres dump from the old server, import it onto the new server, and then run `./upgrade.sh` which will apply the necessary migrations to the data to make it compatible with the latest version.

If you have any uploaded media (images), custom scripts or reports or plugins, they'll all need to be copied over by hand.

[Nguyen Quoc Van]

Hi, 

I have tried dump old db and restore to lastest version db but it show error on foreign key...

I think about export module  data to csv (site, rack, type...) and import on web page but this solution is not good. 

Regards

Vào lúc 17:27:04 UTC+7 ngày Thứ Năm, 4 tháng 3, 2021, Brian Candler đã viết:

[Nguyễn Trọng Vĩnh (Tovin Seven)]

As I said before, can you provide the output of your commands? Full stack trace, error log, or something useful?

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/768d035f-ec4a-42ac-a834-465e7a18cacen%40googlegroups.com.

[Nguyen Quoc Van]

Hi

I installed clean Netbox 2.10.5 on Ubuntu 20

nqvan@dcim-new:/opt/netbox$ python3 -V

Python 3.8.5

nqvan@dcim-new:/opt/netbox$ pip -V

Command 'pip' not found, but there are 18 similar ones.

nqvan@dcim-new:/opt/netbox$ pip3 -V

pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

nqvan@dcim-new:/opt/netbox$ 

I had dump db on 2.7.12 version, restored in version 2.10.5, change owner, previ...

When i run ./upgrade.sh, it show error

Installing collected packages: MarkupSafe, urllib3, sqlparse, pytz, Jinja2, idna, chardet, certifi, asgiref, uritemplate, six, ruamel.yaml.clib, requests, pyrsistent, pyparsing, itypes, Django, coreschema, attrs, ruamel.yaml, redis, PyYAML, packaging, jsonschema, inflection, djangorestframework, coreapi, click, swagger-spec-validator, rq, prometheus-client, funcy, drf-yasg, django-js-asset, svgwrite, pycryptodome, psycopg2-binary, Pillow, netaddr, Markdown, gunicorn, django-timezone-field, django-taggit, django-tables2, django-rq, django-prometheus, django-pglocks, django-mptt, django-filter, django-debug-toolbar, django-cors-headers, django-cacheops

Successfully installed Django-3.1.3 Jinja2-2.11.2 Markdown-3.3.3 MarkupSafe-1.1.1 Pillow-8.0.1 PyYAML-5.3.1 asgiref-3.3.1 attrs-20.3.0 certifi-2020.12.5 chardet-4.0.0 click-7.1.2 coreapi-2.3.3 coreschema-0.0.4 django-cacheops-5.1 django-cors-headers-3.5.0 django-debug-toolbar-3.1.1 django-filter-2.4.0 django-js-asset-1.2.2 django-mptt-0.11.0 django-pglocks-1.0.4 django-prometheus-2.1.0 django-rq-2.4.0 django-tables2-2.3.3 django-taggit-1.3.0 django-timezone-field-4.0 djangorestframework-3.12.2 drf-yasg-1.20.0 funcy-1.15 gunicorn-20.0.4 idna-2.10 inflection-0.5.1 itypes-1.2.0 jsonschema-3.2.0 netaddr-0.8.0 packaging-20.9 prometheus-client-0.9.0 psycopg2-binary-2.8.6 pycryptodome-3.9.9 pyparsing-2.4.7 pyrsistent-0.17.3 pytz-2021.1 redis-3.5.3 requests-2.25.1 rq-1.7.0 ruamel.yaml-0.16.12 ruamel.yaml.clib-0.2.2 six-1.15.0 sqlparse-0.4.1 svgwrite-1.4 swagger-spec-validator-2.7.3 uritemplate-3.0.1 urllib3-1.26.3

Skipping local dependencies (local_requirements.txt not found)

Applying database migrations (python3 netbox/manage.py migrate)...

Operations to perform:

  Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, secrets, sessions, taggit, tenancy, users, virtualization

Running migrations:

  Applying extras.0051_migrate_customfields...Old-Position-U (text): u6 (3993)

Traceback (most recent call last):

  File "netbox/manage.py", line 10, in <module>

    execute_from_command_line(sys.argv)

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line

    utility.execute()

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv

    self.execute(*args, **cmd_options)

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute

    output = self.handle(*args, **options)

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/base.py", line 85, in wrapped

    res = handle_func(*args, **kwargs)

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 243, in handle

    post_migrate_state = executor.migrate(

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate

    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards

    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration

    state = migration.apply(state, schema_editor)

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply

    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)

  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards

    self.code(from_state.apps, schema_editor)

  File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 72, in migrate_customfieldvalues

    raise e

  File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 69, in migrate_customfieldvalues

    cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)

TypeError: 'NoneType' object is not subscriptable

Vào lúc 22:03:43 UTC+7 ngày Thứ Năm, 4 tháng 3, 2021, tov...@gmail.com đã viết:

[Brian Candler]

It's hard to see in that traceback, but it does actually report the custom field value which is causing the problem:

  Applying extras.0051_migrate_customfields...Old-Position-U (text): u6 (3993)

This message was added after I suffered the same problem - see https://github.com/netbox-community/netbox/issues/5573 - and it's coming from netbox/extras/migrations/0051_migrate_customfields.py

        # Read and update custom field value for each instance

        # TODO: This can be done more efficiently once .update() is supported for JSON fields

        cf_data = model.objects.filter(pk=cfv.obj_id).values('custom_field_data').first()

        try:

            cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)

        except Exception as e:

            print(f'{cfv.field.name} ({cfv.field.type}): {cfv.serialized_value} ({cfv.pk})')

            raise e

What it's saying is: the error occurred while processing the customfieldvalue with id 3993.  You can find it using:

select * from extras_customfieldvalue where id=3993;

 Now, cf_data is None.  Almost certainly, this means that the object referred to by this custom field value (using the combination of obj_type_id and obj_id) does not exist: it's an orphan piece data.  obj_type_id is a reference to the django_content_type table, and tells you the model; obj_id is the id column in that model.  Somehow, you managed to delete an object without removing its linked custom field data.

What you should do is restore your data from your pg_dump again, and then either:

1. Go through your extras_customfieldvalue table, find and remove or fix this and any further dangling values; OR

2. If you're sure these orphan custom field values are all obsolete and you don't care about losing them, you can temporarily modify 0051_migrate_customfields.py to discard them

        cf_data = model.objects.filter(pk=cfv.obj_id).values('custom_field_data').first()

        if cf_data is None:

            print(f'{cfv.field.name} ({cfv.field.type}): {cfv.serialized_value} ({cfv.pk}): references non-existent {model.__name__} {cfv.obj_id}')

            continue
        try:

Then run upgrade.sh again.

[Nguyen Quoc Van]

Hi  Brian Candler

It's worked for me. Thank you very much.

Regards

 

Vào lúc 17:44:10 UTC+7 ngày Thứ Sáu, 5 tháng 3, 2021, Brian Candler đã viết: