Viable upgrade path for Netbox v2.9.9, supervisor, no venv?

[Matt Rogers]

I have had a working and heavily-used and relied-upon instance of Netbox v2.9.9 running for almost a year now on an Ubuntu 18.04.05 virtual machine with Python 3.6.9. This installation was carried out utilizing supervisor (not systemd) and does not include a Python virtual environment.

I'd like to upgrade this to the newest Netbox release (v3.0.2) and I understand that the following need to happen in some kind of order more or less, but I need some help sorting it all out:

• upgrade to latest Python, Redis and PostgreSQL releases
• instantiate a Python virtual environment 
• migrate from supervisor to systemd
• upgrade to Ubuntu 20.04 and run updates
• upgrade Netbox to v2.11.x
• run  sudo ./upgrade.sh to update the database schema?
• upgrade Netbox to v3.0.2

I have tried building a separate Ubuntu 20.04 VM, running through a fresh install of Netbox v2.11.12, importing the database from the Netbox v2.9.9 instance and running the upgrade script, as this approach seems to have the least potential for destruction of my working instance. But, this has only yielded error messages and failure.

At this point I'm trying to come up with a decent MOP for either upgrading my current setup or replicating to a new VM and upgrading (starting at v2.11.x) from there.

Will a database exported from Netbox v2.9.9 (sans systemd & venv) even jive with v2.11.12 built to spec? 

Thanks.

[João Lucas B. Macedo]

Hi. I recommend you to move to a fresh Ubuntu 20.04 install, and then migrate your database. There is an official guide for Netbox migration at ReadTheDocs.

You can make a new Netbox installation using 2.99 on a new machine, and hence using systemd, python 3.8 and virtual environment.

Then you proceed with the database migration from your current instance. Once you have your Netbox 2.9.9 on a new environment I recommend you upgrade each version. Remember that you can upgrade to Netbox 3.0 only from Netbox 2.11.

• upgrade to Ubuntu 20.04 and run updates

• upgrade to the latest Python, Redis and PostgreSQL releases
• Install Netbox 2.9.9.
  
• Migrate your database and media files. Checkout the official guide.
  
• upgrade Netbox to v2.10.x
• upgrade Netbox to v2.11.x
• upgrade Netbox to v3.0.2

--
You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/f004467a-6661-4b3a-a5a6-ae183d5f71f9n%40googlegroups.com.

--

João Lucas Macedo

Telecom Engineer.

[Matt Rogers]

Thanks João! I believe I will try that today. The official documentation is rather sparse on the replication issue, but for good reason. They can't (nor should they be responsible to) account for every situation a user may find themselves in.

[Brian Candler]

I agree that the best approach is making a fresh new VM.

The proposed scheme will work, but it's unnecessarily complicated because there's no need to install either netbox v2.9 or v2.10 on the target VM.  Just install netbox v2.11.12, copy across the database, and run "./upgrade.sh" (which will upgrade your database from v2.9 to v.11), before then upgrading to v3.0.2

A bit more detail: if you're using the git method of fetching netbox, then

git pull

git checkout v2.11.12

will give you a temporary checkout of v2.11.12 in a detached branch. Then drop the database, import the one you dumped from your source machine, and continue with:

./upgrade.sh

git checkout master

./upgrade.sh

If you have done any integrations with Netbox v2.9 - e.g. any reports, custom scripts, or API integrations - then you will need to test them, and possibly modify them, to work with v3.  Check the release notes for v2.10.0, v2.11.0 and v3.0.0 for details of incompatible changes.

[Matt Rogers]

After spinning up a new VM, installing Ubuntu 20.04.3 LTS, upgrading all packages, following install guide (Netbox v2.11.12 installed using Option A because I couldn't figure out the "git pull" and "git checkout" commands), importing my database (a .sql file) and running ./upgrade.sh I got the following error:

Applying database migrations (python3 netbox/manage.py migrate)...

Traceback (most recent call last):

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute

    return self.cursor.execute(sql, params)

psycopg2.errors.InsufficientPrivilege: permission denied for table django_migrations

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "netbox/manage.py", line 10, in <module>

    execute_from_command_line(sys.argv)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line

    utility.execute()

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv

    self.execute(*args, **cmd_options)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute

    output = self.handle(*args, **options)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped

    res = handle_func(*args, **kwargs)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 92, in handle

    executor = MigrationExecutor(connection, self.migration_progress_callback)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 18, in __init__

    self.loader = MigrationLoader(self.connection)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 53, in __init__

    self.build_graph()

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 220, in build_graph

    self.applied_migrations = recorder.applied_migrations()

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 78, in applied_migrations

    return {(migration.app, migration.name): migration for migration in self.migration_qs}

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/models/query.py", line 280, in __iter__

    self._fetch_all()

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/cacheops/query.py", line 271, in _fetch_all

    return self._no_monkey._fetch_all(self)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all

    self._result_cache = list(self._iterable_class(self))

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__

    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql

    cursor.execute(sql, params)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 97, in execute

    result = self._no_monkey.execute(self, sql, params)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute

    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers

    return executor(sql, params, many, context)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute

    return self.cursor.execute(sql, params)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__

    raise dj_exc_value.with_traceback(traceback) from exc_value

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute

    return self.cursor.execute(sql, params)

django.db.utils.ProgrammingError: permission denied for table django_migrations

When I went to the http page, I got this Server Error message:

    <class 'django.db.utils.ProgrammingError'>

    permission denied for table django_session

    Python version: 3.8.10

    NetBox version: 2.11.12

After that, I did some searching, found a post on Reddit where a user had a similar issue and tried the following commands:

• sudo -u postgres psql netbox -c "GRANT ALL ON ALL TABLES IN SCHEMA public to netbox;"
  
• sudo -u postgres psql netbox -c "GRANT ALL ON ALL SEQUENCES IN SCHEMA public to netbox;"
  
• sudo -u postgres psql netbox -c "GRANT ALL ON ALL FUNCTIONS IN SCHEMA public to netbox;"
  
• sudo ./upgrade.sh
  

I then got the following error:

  Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, secrets, sessions, taggit, tenancy, users, virtualization

Running migrations:

  Applying virtualization.0018_custom_field_data...Traceback (most recent call last):

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute

    return self.cursor.execute(sql, params)

psycopg2.errors.InsufficientPrivilege: must be owner of table virtualization_cluster

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "netbox/manage.py", line 10, in <module>

    execute_from_command_line(sys.argv)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line

    utility.execute()

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv

    self.execute(*args, **cmd_options)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute

    output = self.handle(*args, **options)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped

    res = handle_func(*args, **kwargs)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle

    post_migrate_state = executor.migrate(

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate

    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards

    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration

    state = migration.apply(state, schema_editor)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 126, in apply

    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 104, in database_forwards

    schema_editor.add_field(

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 522, in add_field

    self.execute(sql, params)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 145, in execute

    cursor.execute(sql, params)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 97, in execute

    result = self._no_monkey.execute(self, sql, params)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute

    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers

    return executor(sql, params, many, context)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute

    return self.cursor.execute(sql, params)

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__

    raise dj_exc_value.with_traceback(traceback) from exc_value

  File "/opt/netbox-2.11.12/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute

    return self.cursor.execute(sql, params)

django.db.utils.ProgrammingError: must be owner of table virtualization_cluster

After that, when I went to the http page, I got this Server Error message:

A database programming error was detected while processing this request. Common causes include the following:

- Database migrations missing -When upgrading to a new NetBox release, the upgrade script must be run to apply any new database migrations.
  You can run migrations manually by executing python3 manage.py migrate from the command line.

- Unsupported PostgreSQL version - Ensure that PostgreSQL version 9.6 or higher is in use.
  You can check this by connecting to the database using NetBox's credentials and issuing a query for SELECT VERSION().

    <class 'django.db.utils.ProgrammingError'>

    relation "dcim_location" does not exist

    LINE 1: SELECT COUNT(*) AS "__count" FROM "dcim_location"

    Python version: 3.8.10

    NetBox version: 2.11.12

I am running postgres (PostgreSQL) 12.8 (Ubuntu 12.8-0ubuntu0.20.04.1), so I don't think that is the issue. It seems like the db is off somehow. But I don't see how because there is only the one user 'netbox' that was ever created or should have privileges. 

And running sudo python3 manage.py migrate only gets me this:

Traceback (most recent call last):

  File "manage.py", line 8, in <module>

    from django.core.management import execute_from_command_line

ModuleNotFoundError: No module named 'django'

I am just not at all sure how to proceed here. I may try a v2.9.9 install just to see if the export and import process I'm attempting is correct at all because I need to start ruling out variables here.

Thanks João and Brian for the input today. I appreciate it very much.

[Brian Candler]

Look in the sql dump (it's just a text file) for the "ALTER TABLE ... OWNER TO ..." lines

If these are inconsistent, e.g. some say "netbox" and some say "postgres", then change them all to "netbox", before importing the dump.

Also when creating the new database, try adding this line:

psql -c 'drop database netbox'

psql -c 'create database netbox'

psql -c 'GRANT ALL PRIVILEGES ON DATABASE netbox TO netbox'

psql netbox < netbox.sql

If that "GRANT ALL" line gives an error, then maybe you didn't create the netbox user on the new system: see https://netbox.readthedocs.io/en/stable/installation/1-postgresql/#database-creation

> And running sudo python3 manage.py migrate only gets me this:

If you really want to do that, you have to activate the virtualenv first. But there's no need to do this, `./upgrade.sh` will do it for you (although it's slow because each attempt will rebuild the virtualenv).

[Matt Rogers]

Brian, you got me pointed in the right direction. Thank you! Turns out the ownership was getting scrubbed by the way I was exporting.

Long story, but I had no idea what I was doing with pg_dump a year ago and I could not get the syntax on the official docs to do anything.

So I ended up with a terrible method that stuck around until just now. And I feel like a fool, but I think others can benefit from my mistake.

Anyways, I just exported from the "old" server again using this (which is straight from TheDocs):

sudo pg_dump --username netbox --password --host localhost netbox> /tmp/netbox_db_backup.sql

Then, uploaded that to the target server. I chose the /tmp/ directory arbitrarily.

I then did the following on the target server, per your instructions:

sudo systemctl stop netbox netbox-rq

sudo -u postgres psql -c 'drop database netbox'

sudo -u postgres psql -c 'create database netbox'

sudo -u postgres psql -c 'grant all privileges on database netbox to netbox'

sudo -u postgres psql netbox < /tmp/netbox_db_backup.sql

cd /opt/netbox

sudo ./upgrade.sh

sudo systemctl restart netbox netbox-rq

Everything works great now on v2.11.12. Couldn't be happier. Ready to upgrade to v3.0.2 now!

[smolz]

How would one go about checking which version Python/Redis/Postgres?  Also, is there a recommended way of doing the upgrade?

[Matt Rogers]

Hello smolz,

For Python: python3 --version

For Redis: redis-server --version

Postgres can be a little tricky because "postgres --version" will probably not give you what you need. So install mlocate by doing: sudo apt install mlocate

Then you can do something like: locate postgres (or: locate bin/postres)

Based on the output, you can then incorporate that into the version command, like this example: /usr/lib/postrgesql/9.6/bin/postgres -V

As far as the upgrade goes, that depends on what version you're starting from and going to (gotta be at v2.11.x before going to v3.x.x and up). But it's all like 99% similar. I'd recommend checking out the official docs.

If anything, the process is:

• Know which Netbox Install method you used originally (Download a Release vs Clone the Git Repo)
• Use the same method to Upgrade
• Follow the official docs
• Run the update script (sudo ./upgrade.sh) from /opt/netbox
• Restart the netbox services

If you're upgrading from a really old version, make sure it's using systemd vs supervisor and that you're using a python virtual environment. The Installation instructions are pretty tight these days so they are a good place to start. As long as you are able to export a copy of your database, you never really run the risk of losing all your work. It's kind of scary, but so far I haven't managed to mess it up--and I'd be the one to do it.

Good luck, and let us know how you do :)

You received this message because you are subscribed to a topic in the Google Groups "NetBox" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netbox-discuss/7IN6rAXJPHA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netbox-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/691be1c7-f163-483a-bd4f-83309a48404cn%40googlegroups.com.

--

Matt Rogers               
(512) 529-1273

[smolz]

Starting netbox version - 2.9.9

Ubuntu - 18.04

Python - 3.6.9

Redis-server - 4.0.9

postgres -

/usr/bin/postgresql/12/bin/postgres

/usr/bin/postgresql/13/bin/postgres

Just upgrading to 2.11.1 fails. I tried doing from above to get to 2.11.12:

```

git pull

git checkout v2.11.12

will give you a temporary checkout of v2.11.12 in a detached branch. Then drop the database, import the one you dumped from your source machine, and continue with:

./upgrade.sh

systemctl restart netbox netbox-rq

```

After restarting the services I am at a "Internal Server Error" page.

The Netbox service is running but the netbox-rq is not.

Running `journalctl -eu netbox`, it looks like there is a missing django library (tablib)

[Brian Candler]

Have you checked that your systemd unit file (/etc/systemd/system/netbox.service) matches the one in the contrib directory? Especially the ExecStart line.

You must run the gunicorn which is within your venv, not the system one.

[smolz]

Verified the file and it was indeed different, not sure how it was working for 2.99?  updated the file with what was in the doc and it started working again.

upgraded os to 20.04 and then to 3.0.3, which failed and I ended up having to remove a topology plugin.  Restarted upgrade and it completed successfully.