[Django] #34847: Serializer infinite recursion on M2M field if reference vars in init
Django
-----------------------------------------+------------------------
Reporter: Arthur Hanson | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 4.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
This is a bit of a strange one. serializers.serialize to json will get an
infinite recursion error on an M2M field if that M2M field has a custom
init method that references two class variables. A quick example below
that reproduces the error - create a new django project and create an app
called testit and put the following into models.py :
{{{
from django.db import models
# Create your models here.
class Tag(models.Model):
name = models.CharField(max_length=200)
position = models.IntegerField(default=0)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._name = self.name
self._original_position = self.position
class Item(models.Model):
name = models.CharField(max_length=200)
position = models.IntegerField(default=0)
tags = models.ManyToManyField(
to=Tag,
related_name='+',
blank=True
)
}}}
Then a management command that does the serialization:
{{{
from django.core.management.base import BaseCommand, CommandError
from testit.models import Item, Tag
from django.core import serializers
class Command(BaseCommand):
help = ""
def handle(self, *args, **options):
tag, created = Tag.objects.get_or_create(name="tag1")
item, created = Item.objects.get_or_create(name="item1")
item.tags.add(tag)
json_str = serializers.serialize('json', [item])
print(json_str)
}}}
When you run the management command the following error is produced
produced:
{{{
Traceback (most recent call last):
File "/Users/ahanson/dev/test/m2mjson/m2mjson/manage.py", line 22, in
<module>
main()
File "/Users/ahanson/dev/test/m2mjson/m2mjson/manage.py", line 18, in
main
execute_from_command_line(sys.argv)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/core/management/__init__.py", line 442, in
execute_from_command_line
utility.execute()
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
File
"/Users/ahanson/dev/test/m2mjson/m2mjson/testit/management/commands/testcmd.py",
line 14, in handle
json_str = serializers.serialize('json', [item])
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/core/serializers/__init__.py", line 134, in serialize
s.serialize(queryset, **options)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/core/serializers/base.py", line 167, in serialize
self.handle_m2m_field(obj, field)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/core/serializers/python.py", line 93, in handle_m2m_field
self._current[field.name] = [m2m_value(related) for related in
m2m_iter]
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/core/serializers/python.py", line 93, in <listcomp>
self._current[field.name] = [m2m_value(related) for related in
m2m_iter]
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 516, in _iterator
yield from iterable
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 122, in __iter__
obj = model_cls.from_db(
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/base.py", line 582, in from_db
new = cls(*values)
File "/Users/ahanson/dev/test/m2mjson/m2mjson/testit/models.py", line
11, in __init__
self._name = self.name
...
File "/Users/ahanson/dev/test/m2mjson/m2mjson/testit/models.py", line 12,
in __init__
self._original_position = self.position
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query_utils.py", line 178, in __get__
instance.refresh_from_db(fields=[field_name])
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/base.py", line 724, in refresh_from_db
db_instance = db_instance_qs.get()
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 633, in get
num = len(clone)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 380, in __len__
self._fetch_all()
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 1881, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 122, in __iter__
obj = model_cls.from_db(
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/base.py", line 582, in from_db
new = cls(*values)
File "/Users/ahanson/dev/test/m2mjson/m2mjson/testit/models.py", line
11, in __init__
self._name = self.name
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query_utils.py", line 178, in __get__
instance.refresh_from_db(fields=[field_name])
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/base.py", line 724, in refresh_from_db
db_instance = db_instance_qs.get()
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 633, in get
num = len(clone)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 380, in __len__
self._fetch_all()
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 1881, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 122, in __iter__
obj = model_cls.from_db(
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/base.py", line 582, in from_db
new = cls(*values)
File "/Users/ahanson/dev/test/m2mjson/m2mjson/testit/models.py", line
12, in __init__
self._original_position = self.position
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query_utils.py", line 178, in __get__
instance.refresh_from_db(fields=[field_name])
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/base.py", line 707, in refresh_from_db
db_instance_qs = self.__class__._base_manager.db_manager(
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 1436, in filter
return self._filter_or_exclude(False, args, kwargs)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 1454, in _filter_or_exclude
clone._filter_or_exclude_inplace(negate, args, kwargs)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/query.py", line 1461, in
_filter_or_exclude_inplace
self._query.add_q(Q(*args, **kwargs))
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/sql/query.py", line 1545, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/sql/query.py", line 1576, in _add_q
child_clause, needed_inner = self.build_filter(
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/sql/query.py", line 1462, in build_filter
if isinstance(value, Iterator):
File "/Users/ahanson/.pyenv/versions/3.10.6/lib/python3.10/abc.py", line
119, in __instancecheck__
return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
Exception ignored in: <generator object cursor_iter at 0x102ec03c0>
Traceback (most recent call last):
File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-
packages/django/db/models/sql/compiler.py", line 2096, in cursor_iter
cursor.close()
sqlite3.ProgrammingError: Cannot operate on a closed database.
}}}
What is strange is that two variables have to be referenced in the init,
if you only reference one it will work fine. The assignment in the init
is just for clarity, just the reference to the var will cause the issue:
{{{
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.name
# second one needed to make it fail
self.position
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34847>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Arthur Hanson | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 4.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Arthur Hanson):
* Attachment "m2mjson.zip" added.
Sample project of model and management command that reproduces the issue
Django
Reporter: Arthur Hanson | Owner: nobody
Type: Uncategorized | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by David Sanders):
* component: Uncategorized => Core (Serialization)
* stage: Unreviewed => Accepted
Comment:
Thanks for the report!
Initial investigation monitoring the SQL it goes into a death spiral as it
alternates between selecting `name` & `position` from the database.
--
Ticket URL: <https://code.djangoproject.com/ticket/34847#comment:1>
Django
--------------------------------------+------------------------------------
Reporter: Arthur Hanson | Owner: nobody
Component: Core (Serialization) | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------
Changes (by David Sanders):
* type: Uncategorized => Bug
--
Ticket URL: <https://code.djangoproject.com/ticket/34847#comment:2>
Django
--------------------------------------+------------------------------------
Reporter: Arthur Hanson | Owner: nobody
Type: Bug | Status: new
Component: Core (Serialization) | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------
Comment (by David Sanders):
Hi Arthur,
Just FYI this appears to make it work if you're looking for an immediate
fix:
{{{
serializers.serialize("json", Item.objects.prefetch_related('tags'))
}}}
The docs state that `serialize()` expects a queryset rather than a
material list, however, without prefetching we get the same infinite
recursion… so I'd wager this will remain accepted.
--
Ticket URL: <https://code.djangoproject.com/ticket/34847#comment:3>
Django
Reporter: Arthur Hanson | Owner: nobody
Component: Core | Version: 4.2
(Serialization) |
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Simon Charette):
* cc: model, init, recursionerror (added)
* status: new => closed
* resolution: => invalid
* stage: Accepted => Unreviewed
Comment:
I agree with your assessment David,
[https://code.djangoproject.com/ticket/31435#comment:2 this came up] when
adjusting cascade deletion to limit the number of fields that get selected
(#30191).
In order for your model definition `__init__` override to adequately
support field deferral, which is a feature the serialization framework
make use of, you must use `self.__dict__.get(field_name)` to retrieve
possibly deferred values.
--
Ticket URL: <https://code.djangoproject.com/ticket/34847#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: Arthur Hanson | Owner: nobody
Type: Bug | Status: closed
Component: Core | Version: 4.2
(Serialization) |
Severity: Normal | Resolution: invalid
recursionerror | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: model, init, recursionerror (removed)
* keywords: => model, init, recursionerror
Comment:
Moved keywords from cc to keywords ;)
--
Ticket URL: <https://code.djangoproject.com/ticket/34847#comment:5>