[Django] #28972: `RelatedObjectDoesNotExist` during `loaddata` of models with multi-table inheritance
Django
inheritance
------------------------------------------------+------------------------
Reporter: Kal Sze | Owner: nobody
Type: Bug | Status: new
Component: Core (Serialization) | Version: 2.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
Software versions
* Python 3.6.3
* Django 2.0
* django-model-utils 3.1.1 (from git)
* PostgreSQL 9.4
* psycopg2 2.7.3.2
I have a `licenses` app, with these models:
* `BaseLicense`, a concrete model, with a `ForeignKey` to
`settings.AUTH_USER_MODEL`.
* `CorelDrawLicense`, which inherits from `BaseLicense` through multi-
table inheritance
* `AdobePhotoshopLicense`, which inherits from `BaseLicense` through
multi-table inheritance
I created an instance of `AdobePhotoshopLicense`, then tried to do
`manage.py dumpdata` and `manage.py loaddata`.
`dumpdata` "worked" - there was no error.
However, `loaddata` failed with this traceback:
{{{
Traceback (most recent call last):
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/serializers/json.py", line 69, in Deserializer
yield from PythonDeserializer(objects, **options)
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/serializers/python.py", line 170, in Deserializer
obj = base.build_instance(Model, data, using)
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/serializers/base.py", line 225, in build_instance
natural_key = obj.natural_key()
File "/home/kal/git/nk_test/licenses/models.py", line 33, in natural_key
return (self.user.natural_key(), self.license_key)
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/db/models/fields/related_descriptors.py", line 175, in
__get__
"%s has no %s." % (self.field.model.__name__, self.field.name)
django.db.models.fields.related_descriptors.RelatedObjectDoesNotExist:
BaseLicense has no user.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/management/__init__.py", line 371, in
execute_from_command_line
utility.execute()
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/management/commands/loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/management/commands/loaddata.py", line 113, in
loaddata
self.load_label(fixture_label)
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/management/commands/loaddata.py", line 168, in
load_label
for obj in objects:
File "/home/kal/.virtualenvs/nk_test-5iWsqJVv/lib/python3.6/site-
packages/django/core/serializers/json.py", line 73, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing
fixture '/home/kal/git/nk_test/./fixtures/dumpdata.json':
}}}
Here are the exact steps:
1. Create model instances in management shell
{{{
>>> from django.contrib.auth import get_user_model
>>> User = get_user_model()
>>> kal = User.objects.create_user('kal', 'k...@foobar.com', 'qwer1234')
>>> from licenses.models import AdobePhotoshopLicense
>>> l = AdobePhotoshopLicense.objects.create(user=kal,
license_key='12345')
}}}
2. Dump the data into a fixtures json
{{{
$ python manage.py dumpdata -o fixtures.json --format json --natural-
foreign --natural-primary auth licenses
}}}
3. Edit settings.py to switch to an empty database
{{{
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
#'NAME': 'nk_test',
'NAME': 'nk_test_2',
'USER': 'kal',
'HOST': '',
'PORT': 5432,
}
}
}}}
4. Run migrations on the new empty database
{{{
$ python manage.py migrate
}}}
5. Try to load the data from fixtures.json
{{{
$ python manage.py loaddata --format json fixtures.json
}}}
I have attached the models.py and fixtures.json file to the ticket.
--
Ticket URL: <https://code.djangoproject.com/ticket/28972>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Kal Sze):
* Attachment "models.py" added.
models.py
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Kal Sze):
* Attachment "fixtures.json" added.
fixtures.json, reformatted for readability
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Kal Sze:
Old description:
New description:
Also note that this bug doesn't have to do with my use of the custom
`InheritanceManager`. I could let the `BaseLicense` model use the default
manager and still get the same bug.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28972#comment:1>
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Kal Sze:
Old description:
> Software versions
> Also note that this bug doesn't have to do with my use of the custom
> `InheritanceManager`. I could let the `BaseLicense` model use the default
> manager and still get the same bug.
New description:
I have attached the `models.py` and `fixtures.json` files to the ticket.
Also note that this bug doesn't have to do with my use of the custom
`InheritanceManager`. I could let the `BaseLicense` model use the default
manager and still get the same bug.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28972#comment:2>
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Kal Sze:
Old description:
> Software versions
> I have attached the `models.py` and `fixtures.json` files to the ticket.
>
> Also note that this bug doesn't have to do with my use of the custom
> `InheritanceManager`. I could let the `BaseLicense` model use the default
> manager and still get the same bug.
New description:
I have attached the `models.py` and `fixtures.json` files to the ticket.
I know that I don't (and I can't) enforce the natural key uniqueness at
the database level. However, due to my application logic, I know that the
user+license_key combination is effectively unique for any one of the
child license models. And this is specifically allowed,
[https://docs.djangoproject.com/en/2.0/topics/serialization
/#deserialization-of-natural-keys according to the documentation]:
[U]niqueness doesn’t need to be enforced at the database level. If you
are certain that a set of fields will be effectively unique, you can still
use those fields as a natural key.
Also note that this bug doesn't have to do with my use of the custom
`InheritanceManager`. I could let the `BaseLicense` model use the default
manager and still get the same bug.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28972#comment:3>
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
It may be a duplicate of #24607. Could you test the patch there?
--
Ticket URL: <https://code.djangoproject.com/ticket/28972#comment:4>
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Component: Core | Version: 2.0
(Serialization) |
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Kal Sze):
* status: new => closed
* resolution: => duplicate
Comment:
Replying to [comment:4 Tim Graham]:
> It may be a duplicate of #24607. Could you test the patch there?
[https://github.com/django/django/pull/8370/ PR 8370] fixes it.
I did see #24607, but I didn't think it was the same problem because the
symptoms were a bit different: I got a `RelatedObjectDoesNotExist` whereas
as the original ticket got a `ValueError`.
--
Ticket URL: <https://code.djangoproject.com/ticket/28972#comment:5>
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Kal Sze):
* status: closed => new
* resolution: duplicate =>
Comment:
Actually there is still a bit of a problem.
Even though serialization and deserialization "work", I think the result
still misses the *point* of the natural key.
If I look at the fixtures being dumped out, the parent model is still
referred to by pk:
```
[...
{
"model": "licenses.baselicense",
"pk": 1,
"fields": {
"user": ["kal"]
}
},
{
"model": "licenses.adobephotoshoplicense",
"fields": {
"baselicense_ptr": 1,
"license_key": "12345"
}
}]
```
Whereas I expect it to be just something like:
```
{
"model": "licenses.adobephotoshoplicense",
"fields": {
"user": ["kal"],
"license_key": "12345"
}
}
```
--
Ticket URL: <https://code.djangoproject.com/ticket/28972#comment:6>
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Is that a separate issue or a defect in the patch for #24607? If the
latter, I'd point out the issue there rather than opening a different
ticket.
--
Ticket URL: <https://code.djangoproject.com/ticket/28972#comment:7>
Django
inheritance
Reporter: Kal Sze | Owner: nobody
Component: Core | Version: 2.0
(Serialization) |
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* status: new => closed
* resolution: => duplicate
Comment:
I don't think the output you proposed could be deserialized properly. If
`baselicense_ptr` isn't included in the `adobephotoshoplicense` object,
there's no way to link `adobephotoshoplicense` to its `baselicense` object
since `user` doens't uniquely identify a `baselicense`.
--
Ticket URL: <https://code.djangoproject.com/ticket/28972#comment:8>