[Django] #35996: Missing chunk_size throws exception when serializing many-to-many Model field

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: | Owner: ericapisani
ericapisani |
Type: Bug | Status: assigned
Component: Core | Version: 5.0
(Serialization) |
Severity: Release | Keywords:
blocker |
Triage Stage: | Has patch: 1
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Related issue: https://code.djangoproject.com/ticket/35238

I'm currently upgrading a project from v4.x to v5.0.1 and, when attempting
to save a model with a prefetched many-to-many field, the "ValueError:
chunk_size must be provided when using QuerySet.iterator() after
prefetch_related()" exception is thrown within the core Django
[https://github.com/django/django/blob/9c436a09b3a641874881706495ae07293aa97c2f/django/core/serializers/python.py#L86
serializer]:


Here's the trace within my project that leads to the exception:
{{{
File ".../site-packages/django/db/models/base.py", line 822, in save
self.save_base(
File ".../site-packages/django/db/models/base.py", line 924, in
save_base
post_save.send(
File ".../site-packages/django/dispatch/dispatcher.py", line 189, in
send
response = receiver(signal=self, sender=sender, **named)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

# The revisions files below are referring to the django-reversion library

File ".../site-packages/reversion/revisions.py", line 340, in
_post_save_receiver
add_to_revision(instance, model_db=using)
File ".../site-packages/reversion/revisions.py", line 209, in
add_to_revision
_add_to_revision(obj, db, model_db, True)
File ".../site-packages/reversion/revisions.py", line 184, in
_add_to_revision
serialized_data=serializers.serialize(
^^^^^^^^^^^^^^^^^^^^^^

File ".../site-packages/django/core/serializers/__init__.py", line 134, in
serialize
s.serialize(queryset, **options)
File ".../site-packages/django/core/serializers/base.py", line 143, in
serialize
self.handle_m2m_field(obj, field)
File ".../site-packages/django/core/serializers/python.py", line 91, in
handle_m2m_field
queryset_iterator(obj, field),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/django/core/serializers/python.py", line 86, in
queryset_iterator
.iterator()
^^^^^^^^^^
File ".../site-packages/django/db/models/query.py", line 532, in
iterator
raise ValueError(
ValueError: chunk_size must be provided when using QuerySet.iterator()
after prefetch_related().
}}}


I confirmed that applying the same fix as the issue linked above addresses
the issue.

**Proposed solution:**
I'm happy to put up a pull request with the patch, and can emulate what
the issue linked above did which is conditionally supply a `chunk_size`
similar to the solution implemented in the issue linked above'
--
Ticket URL: <https://code.djangoproject.com/ticket/35996>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------

Reporter: Erica Pisani | Owner: (none)
Type: Bug | Status: new

Component: Core | Version: 5.0
(Serialization) |

Severity: Release blocker | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Erica Pisani):

* owner: Erica Pisani => (none)
* status: assigned => new

--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:1>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: (none)

Type: Bug | Status: closed

Component: Core | Version: 5.0
(Serialization) |

Severity: Release blocker | Resolution: duplicate

Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham):

* resolution: => duplicate
* status: new => closed

Comment:

The patch was backported to the stable/5.0.x branch and released in 5.0.3.
See ticket:35238#comment:7.
--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:2>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: (none)
Type: Bug | Status: closed
Component: Core | Version: 5.0
(Serialization) |
Severity: Release blocker | Resolution: duplicate
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Erica Pisani):

Hi Tim,

The patch that you're referring to addresses a similar issue, but in a
different area of the code.

I opened a [PR https://github.com/django/django/pull/18917] with a patch
that mirrors the one that you linked if you'd like to see what I'm
referring to.

It's still missing tests but I'm looking to add some and would appreciate
any guidance on where the best place to do so would be.
--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:3>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: (none)

Type: Bug | Status: new

Component: Core | Version: 5.0
(Serialization) |
Severity: Release blocker | Resolution:

Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0

Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Erica Pisani):

* needs_tests: 0 => 1
* resolution: duplicate =>
* status: closed => new

--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:4>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------

Reporter: Erica Pisani | Owner: Erica
| Pisani
Type: Bug | Status: assigned

Component: Core | Version: 5.0
(Serialization) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Erica Pisani):

* owner: (none) => Erica Pisani
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:5>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani

Type: Bug | Status: closed

Component: Core | Version: 5.0
(Serialization) |

Severity: Release blocker | Resolution: needsinfo

Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* resolution: => needsinfo
* status: assigned => closed

Comment:

Have you upgraded to Django 5.1.4 and can you please share the model/model
managers so that we can reproduce?
--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:6>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani
Type: Bug | Status: closed
Component: Core | Version: 5.0
(Serialization) |
Severity: Release blocker | Resolution: needsinfo
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Erica Pisani):

Hi Sarah,

I've tried upgrading to Django 5.1.4 and the issue still exists (and it
still does on the main branch of the Django repository as well).

For steps to reproduce:

We can use these models within the Django test suite in
`tests/serializers/models/base.py`:

{{{
class CategoryMetaData(models.Model):
kind = models.CharField(max_length=10)
name = models.CharField(max_length=10)
value = models.CharField(max_length=10)
objects = CategoryMetaDataManager()

class Category(models.Model):
name = models.CharField(max_length=20)
meta_data = models.ForeignKey(
CategoryMetaData, models.SET_NULL, null=True, default=None
)

class Article(models.Model):
author = models.ForeignKey(Author, models.CASCADE)
headline = models.CharField(max_length=50)
pub_date = models.DateTimeField()
categories = models.ManyToManyField(Category)
meta_data = models.ManyToManyField(CategoryMetaData)
topics = models.ManyToManyField(Topic)

}}}

if `test_serialize_prefetch_related_m2m` in `tests/serializers/tests.py`
were updated from:


{{{
with self.assertNumQueries(4):
serializers.serialize(
self.serializer_name,
Article.objects.prefetch_related("categories",
"meta_data", "topics"),
)
with self.assertNumQueries(7):
serializers.serialize(self.serializer_name,
Article.objects.all())
}}}

to the following:


{{{
with self.assertNumQueries(4):
serializers.serialize(
self.serializer_name,
Article.objects.prefetch_related(
"meta_data",
"topics",
Prefetch(
"categories",
queryset=Category.objects.prefetch_related("meta_data"),
)
with self.assertNumQueries(7):
serializers.serialize(self.serializer_name,
Article.objects.all())
}}}

the ValueError related to missing chunk_size will be raised in the JSON,
YAML, and XML serializers.
--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:7>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani

Type: Bug | Status: new

Component: Core | Version: 5.0
(Serialization) |
Severity: Release blocker | Resolution:

Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Erica Pisani):

* needs_tests: 1 => 0
* resolution: needsinfo =>

* status: closed => new

Old description:

New description:

I've put up a pull request with the patch which mirrors what was done in
the linked issue above (conditionally adding a `chunk_size` to the
`.iterator` call).

--
--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:8>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani
Type: Bug | Status: new
Component: Core | Version: 5.0
(Serialization) |

Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* severity: Release blocker => Normal
* stage: Unreviewed => Accepted

Comment:

Thank you Erica!
--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:9>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani
Type: Bug | Status: new
Component: Core | Version: 5.0
(Serialization) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:10>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani
Type: Bug | Status: new
Component: Core | Version: 5.0
(Serialization) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* needs_better_patch: 1 => 0
* needs_tests: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:11>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani
Type: Bug | Status: new
Component: Core | Version: 5.0
(Serialization) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Erica Pisani):

* needs_tests: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:12>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani
Type: Bug | Status: new
Component: Core | Version: 5.0
(Serialization) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* needs_tests: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:13>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani
Type: Bug | Status: new
Component: Core | Version: 5.0
(Serialization) |
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* needs_tests: 1 => 0
* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:14>

[Django]

#35996: Missing chunk_size throws exception when serializing many-to-many Model
field
-------------------------------------+-------------------------------------
Reporter: Erica Pisani | Owner: Erica
| Pisani

Type: Bug | Status: closed

Component: Core | Version: 5.0
(Serialization) |

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce <42296566+sarahboyce@…>):

* resolution: => fixed

* status: new => closed

Comment:

In [changeset:"20f9f6180531279785ceabed6ab2f29ab634d8dc" 20f9f61]:
{{{#!CommitTicketReference repository=""
revision="20f9f6180531279785ceabed6ab2f29ab634d8dc"
Fixed #35996 -- Fixed database serialization crash when serializing a
many-to-many field that had a prefetch.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35996#comment:15>