Re: [Django] #36732: Sitemaps with i18n=True fetch the entire queryset of items despite self.limit (was: Sitemaps with i18n=True load the whole table in memory)

[Django]

#36732: Sitemaps with i18n=True fetch the entire queryset of items despite
self.limit
----------------------------------+------------------------------------
Reporter: Julien Palard | Owner: (none)
Type: Bug | Status: new
Component: contrib.sitemaps | Version: 5.2
Severity: Normal | Resolution:
Keywords: sitemap, memory | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Changes (by Jacob Walls):

* cc: Roxane (added)
* stage: Unreviewed => Accepted
* summary: Sitemaps with i18n=True load the whole table in memory =>
Sitemaps with i18n=True fetch the entire queryset of items despite
self.limit
* type: Cleanup/optimization => Bug

Comment:

Thanks for the report. #33662 introduced eager evaluation of
`self.items()` (if i18n is in use) instead of letting `self.items()` (a
queryset) remain unevaluated until paged by the paginator.

Tentatively accepting as a bug that we should at least respect self.limit.

Could be as simple as something like this (could limit even further by
doing some language arithmetic):

{{{#!diff
diff --git a/django/contrib/sitemaps/__init__.py
b/django/contrib/sitemaps/__init__.py
index c0c0ac77a7..faf60f346b 100644
--- a/django/contrib/sitemaps/__init__.py
+++ b/django/contrib/sitemaps/__init__.py
@@ -185,7 +185,7 @@ class GenericSitemap(Sitemap):

def items(self):
# Make sure to return a clone; we don't want premature
evaluation.
- return self.queryset.filter()
+ return self.queryset[:self.limit]

def lastmod(self, item):
if self.date_field is not None:
}}}


----

I'm not seeing what you initially suggested about a cartesian product of
items and languages -- if the order of the `for` loops in the
comprehension were reversed, then I'd see the problem, but it's not, so
it's the same item. Am I missing something?

{{{#!py
(Pdb) items
[(<I18nTestModel: I18nTestModel object (1)>, 'en'), (<I18nTestModel:
I18nTestModel object (1)>, 'pt')]
(Pdb) items[0][0] is items[1][0]
True
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36732#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.