Re: [Django] #36732: Sitemaps with i18n=True fetch the entire queryset of items despite self.limit

[Django]

#36732: Sitemaps with i18n=True fetch the entire queryset of items despite
self.limit
-------------------------------------+-------------------------------------
Reporter: Julien Palard | Owner: Varun
| Kasyap Pentamaraju
Type: Bug | Status: assigned
Component: contrib.sitemaps | Version: 5.2
Severity: Normal | Resolution:
Keywords: sitemap, memory | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Varun Kasyap Pentamaraju):

* owner: (none) => Varun Kasyap Pentamaraju
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36732#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36732: Sitemaps with i18n=True fetch the entire queryset of items despite
self.limit
-------------------------------------+-------------------------------------
Reporter: Julien Palard | Owner: Varun
| Kasyap Pentamaraju
Type: Bug | Status: assigned
Component: contrib.sitemaps | Version: 5.2
Severity: Normal | Resolution:
Keywords: sitemap, memory | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Julien Palard):

> I'm not seeing what you initially suggested about a cartesian

You're right, it's not a cartesian product database-side, it's only a
carthesian product Python-side, so 8 bytes per object, so ~15 MB per
language for my 2 million objects, that's nothing compared to the 16GB+
that it takes currently.

This can't be as simple as `return self.queryset[:self.limit]` because the
limit is not a limit overall but a limit per page, so putting the limit
here imples a single page (the pagination occurs over the result of
`items`: `return paginator.Paginator(self._items(), self.limit)`).

The path of doing arithmetics to be able to query the right slice of items
depending on the requested page *and* the number of languages breaks if
`get_languages_for_item` uses its `item` argument.
--
Ticket URL: <https://code.djangoproject.com/ticket/36732#comment:5>

[Django]

#36732: Sitemaps with i18n=True fetch the entire queryset of items despite
self.limit
-------------------------------------+-------------------------------------
Reporter: Julien Palard | Owner: Varun
| Kasyap Pentamaraju
Type: Bug | Status: assigned
Component: contrib.sitemaps | Version: 5.2
Severity: Normal | Resolution:
Keywords: sitemap, memory | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls):

Right, okay. Maybe we can pursue checking if `get_languages_for_item` has
been overridden and if not, avoid the materialization of the items before
paginating.
--
Ticket URL: <https://code.djangoproject.com/ticket/36732#comment:6>