Custom Fields migration error

Bogdan Cordos

unread,

Aug 31, 2021, 9:29:54 AM8/31/21

to NetBox

Hi Team,

I have built a new instance of NetBox 3.0.0 and load the sql dump.

Most of the tables were loaded OK, however, I'm missing the IPAM Prefixes.

On top of this, I get the fallowing error while trying to edit or delete any custom fields:

The complete exception is provided below:

<class 'django.core.exceptions.FieldError'>

Cannot resolve keyword 'custom_field_data' into field. Choices are: _name, created, description, device_type, device_type_id, id, label, last_updated, name, object_changes, type

Python version: 3.8.10
NetBox version: 3.0.0

Can anyone point me into the right direction on how to solve this?

Thank you

Brian Candler

unread,

Aug 31, 2021, 11:01:16 AM8/31/21

to NetBox

What do you mean "load the sql dump"? A brand new instance won't have this. Do you mean you're copying the database from an older version of Netbox? If so, what version?

After restoring the database from the old version, you will need to run an upgrade to apply the modifications ("migrations") to the database to bring it up to v3.0.0:

cd /opt/netbox

./upgrade.sh

If you get any errors in that stage, don't ignore them - report them to the list. It won't work if something goes wrong there.

Once ./upgrade.sh has completed successfully, the database should be in the correct state, and you'll then have to restart netbox (systemctl restart netbox netbox-rq)

Bogdan Cordos

unread,

Sep 1, 2021, 4:30:31 AM9/1/21

to NetBox

Thank you for feedback.

I do apologize for creating confusion.

I have dumped the sql database from a 2.11.12 build and imported to a 3.0.0 build. I had to build a new server as the upgrade from 2.11.12 to 3.0.0 failed, and I wanted to start fresh with the new version.

The fields are imported, and I did perform a migration (./upgrade.sh), however, when I want to edit or delete any custom fields I get the mentioned error.

" <class 'django.core.exceptions.FieldError'>
Cannot resolve keyword 'custom_field_data' into field. Choices are: _name, created, description, device_type, device_type_id, id, label, last_updated, name, object_changes, type

Python version: 3.8.10

NetBox version: 3.0.0"

Thanks

Brian Candler

unread,

Sep 1, 2021, 5:08:24 AM9/1/21

to NetBox

Building a new 3.0.0 from scratch and migrating the data is a very good way to approach this.

Can you provide the full exception backtrace? Set EMAIL and ADMINS in configuration.py, and it should be mailed to you. Alternatively, set DEBUG = True and then copy-paste it from the error page.

Does creating a new custom field work, i.e. is the error only changing or deleting existing custom fields which were migrated? If so, which model(s) are your existing custom fields bound to?

Brian Candler

unread,

Sep 1, 2021, 5:09:06 AM9/1/21

to NetBox

Also, do you have any customisations to Netbox - particularly plugins, custom scripts, custom links?

Bogdan Cordos

unread,

Sep 1, 2021, 5:35:53 AM9/1/21

to NetBox

Creating a new custom field works, however, it doesn't save the "Content Types" no matter what I choose. This concerns me more than the imported fields, as I can't recreate them either.

The imported custom fields are all assigned to " DCIM > console server port template" which was not the original value. On the 2.11.12, the custom field were assigned to "DCIM > device".

I have no customization implemented yet, apart from a custom link pointing to my NMS. This works fine.

Debug Output:
Environment:

Request Method: POST
Request URL: https://FQDN/extras/custom-fields/delete/

Django Version: 3.2.6
Python Version: 3.8.10
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.humanize',
'corsheaders',
'debug_toolbar',
'graphiql_debug_toolbar',
'django_filters',
'django_tables2',
'django_prometheus',
'graphene_django',
'mptt',
'rest_framework',
'taggit',
'timezone_field',
'circuits',
'dcim',
'ipam',
'extras',
'tenancy',
'users',
'utilities',
'virtualization',
'django_rq',
'drf_yasg']
Installed Middleware:
['graphiql_debug_toolbar.middleware.DebugToolbarMiddleware',
'django_prometheus.middleware.PrometheusBeforeMiddleware',
'corsheaders.middleware.CorsMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware',
'netbox.middleware.ExceptionHandlingMiddleware',
'netbox.middleware.RemoteUserMiddleware',
'netbox.middleware.LoginRequiredMiddleware',
'netbox.middleware.APIVersionMiddleware',
'netbox.middleware.ObjectChangeMiddleware',
'django_prometheus.middleware.PrometheusAfterMiddleware']

Traceback (most recent call last):
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
File "/opt/netbox/netbox/utilities/views.py", line 93, in dispatch
    return super().dispatch(request, *args, **kwargs)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
File "/opt/netbox/netbox/netbox/views/generic.py", line 1037, in post
    obj.delete()
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/base.py", line 954, in delete
    return collector.delete()
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/deletion.py", line 404, in delete
    signals.pre_delete.send(
File "/opt/netbox/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
File "/opt/netbox/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
File "/opt/netbox/netbox/extras/signals.py", line 145, in handle_cf_deleted
    instance.remove_stale_data(instance.content_types.all())
File "/opt/netbox/netbox/extras/models/customfields.py", line 146, in remove_stale_data
    instances = model.objects.filter(**{f'custom_field_data__{self.name}__isnull': False})
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/query.py", line 941, in filter
    return self._filter_or_exclude(False, args, kwargs)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/query.py", line 961, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/query.py", line 968, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1393, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1412, in _add_q
    child_clause, needed_inner = self.build_filter(
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1286, in build_filter
    lookups, parts, reffed_expression = self.solve_lookup_type(arg)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1112, in solve_lookup_type
    _, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1539, in names_to_path
    raise FieldError("Cannot resolve keyword '%s' into field. "

Exception Type: FieldError at /extras/custom-fields/delete/
Exception Value: Cannot resolve keyword 'custom_field_data' into field. Choices are: _name, created, description, device_type, device_type_id, id, label, last_updated, name, object_changes, type

Brian Candler

unread,

Sep 1, 2021, 8:20:39 AM9/1/21

to NetBox

That code is trying to remove a particular member from custom_field_data when that custom_field is removed.

> The imported custom fields are all assigned to " DCIM > console server port template" which was not the original value. On the 2.11.12, the custom field were assigned to "DCIM > device"

That's worrying, but I think it explains the problem: console server port template doesn't have custom_field_data. This also might correlate with other people's problems I've seen reported, where certain content types are missing. I'm wondering if they're not being migrated properly.

Can you get a postgres shell (e.g. "sudo -u postgres psql netbox") and show the output of the following queries:

select * from extras_customfield_content_types;

select * from django_content_type;

Brian Candler

unread,

Sep 1, 2021, 8:24:53 AM9/1/21

to NetBox

FWIW, my netbox 2.11.12 has 87 rows in django_content_type, and netbox 3.0.0 has 88 rows. The extra one is:

95 | ipam | iprange

Bogdan Cordos

unread,

Sep 1, 2021, 9:06:05 AM9/1/21

to NetBox

At this point, I just want to get rid of any custom field, as I can't add or edit any device.

I think I will rebuild the 3.0.0 and import the data via CSV to prevent ingesting any broken fields or scheme.

*****************************

netbox=# select * from extras_customfield_content_types;
id | customfield_id | contenttype_id
----+----------------+----------------
1 |              1 |             19
2 |              2 |             19
3 |              3 |             19
4 |              4 |             19
5 |              5 |             19
6 |              6 |             19
(6 rows)

****************************

netbox=# select * from django_content_type;

**********************************

Brian Candler

unread,

Sep 1, 2021, 10:14:56 AM9/1/21

to NetBox

The sticking plaster would be:

update extras_customfield_content_types set contenttype_id=20 where contenttype_id=19;

but something seriously bad is going on here - and without understanding how it got into this state, you may be storing up problems for the future (and other users may be breaking too).

If you still have access to the older Netbox instance, can you do the same two queries against that as well please? I want to see if custom fields are linked to the correct content model there or not. Can you also check if the sql dump you made of the older netbox includes the "django_content_type" table.

As for your 83 rows versus my 88 rows: it turns out I have 1 for "netbox_animal_sounds" (from playing with plugins) and 4 for "secrets" which is now gone anyway, so that's fine.

Brian Candler

unread,

Sep 1, 2021, 10:16:41 AM9/1/21

to NetBox

Another thing: just to confirm, you did *completely* replace the netbox database with the sql dump? i.e. drop the database, create empty database, restore into it? As per https://netbox.readthedocs.io/en/stable/administration/replicating-netbox/#load-an-exported-database

Brian Candler

unread,

Sep 1, 2021, 4:03:29 PM9/1/21

to NetBox

So: I just did a fresh migration from Netbox v2.11.12 to v3.0.1. In fact, I have the two databases side by side in the same postgres server called "netbox2" and "netbox3", with netbox2 and netbox3 in separate containers.

I dumped the netbox2 database, restored it with the following script:

#!/bin/bash -eux

# sudo -u postgres ./netbox2to3

psql -c 'drop database netbox3'

psql -c 'create database netbox3'

gzip -dc /var/backups/netbox/netbox2.202109011927.gz |

sed -e 's/OWNER TO netbox2/OWNER TO netbox3/g' |

psql netbox3

# Or if it was a non-text pg_dump: pg_restore --no-owner --role=owner2

# https://stackoverflow.com/questions/31469008/postgresql-backup-database-and-restore-on-different-owner

Now if I do "select * from django_content_type order by id;" on the two databases, I find they are both exactly the same content types and content IDs, except netbox3 has the ipam_iprange added at the end.

Furthermore, the one custom field I had (which was linked to DCIM > device, Virtualization > virtual machine) migrated correctly.

Therefore, my guess is that something went wrong during your restore of the pg_dump, which means that the database was in a partial state, and a migration filled in the gaps in django_content_type (but with different sequence numbers). I suggest you do your restore again and watch carefully for any errors as it runs. You could try setting ON_ERROR_STOP, or the --echo-errors flag.

Another way to check is after you've done the restore and are running ./upgrade.sh, only a few migrations should be applied:

Operations to perform:

Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, sessions, taggit, tenancy, users, virtualization

Running migrations:

Applying dcim.0132_cable_length... OK

Applying dcim.0133_port_colors... OK

Applying extras.0060_customlink_button_class... OK

Applying extras.0061_extras_change_logging... OK

Applying ipam.0049_prefix_mark_utilized... OK

Applying ipam.0050_iprange... OK

Checking for missing cable paths (python3 netbox/manage.py trace_paths --no-input)...

Found no missing console port paths; skipping

Found no missing console server port paths; skipping

Found no missing interface paths; skipping

Found no missing power feed paths; skipping

Found no missing power outlet paths; skipping

Found no missing power port paths; skipping

Finished.

If you see more migrations than those, then some of the database tables are incomplete and/or unpopulated, and so the migrations are creating them from scratch. Look carefully at what's going on.

Just one other random idea: try this query

select relnamespace::regnamespace as "schema", relname, relkind, relowner::regrole

from pg_class where relname not like 'pg_%';

and check for any rows which are in the "public" schema which are not owned by your netbox database user.

Reply all

Reply to author

Forward