[Django] #36732: Sitemaps with i18n=True load the whole table in memory

4 views

Skip to first unread message

Django

unread,

Nov 14, 2025, 8:47:25 AM11/14/25

to django-...@googlegroups.com

#36732: Sitemaps with i18n=True load the whole table in memory
-------------------------------------+-------------------------------------
Reporter: Julien Palard | Type:
| Cleanup/optimization
Status: new | Component:
| contrib.sitemaps
Version: 5.2 | Severity: Normal
Keywords: sitemap, memory | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Hi!

It's a bit like: https://code.djangoproject.com/ticket/11572

when enabling i18n on a sitemap, _items uses the following code:

```python
# Create (item, lang_code) tuples for all items and languages.
# This is necessary to paginate with all languages already
considered.
items = [
(item, lang_code)
for item in self.items()
for lang_code in self.get_languages_for_item(item)
]
```

The list comprehension loads the whole table times the number of languages
in memory.

We tried with a table containing two millions elements and two languages:
it does not fit in 16GB of memory and the process gets killed.
--
Ticket URL: <https://code.djangoproject.com/ticket/36732>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Nov 14, 2025, 8:55:20 AM11/14/25

to django-...@googlegroups.com

#36732: Sitemaps with i18n=True load the whole table in memory
-------------------------------------+-------------------------------------

Comment (by Julien Palard):

While:

{{{
def get_languages_for_item(self, item):
"""Languages for which this item is displayed."""
return self._languages()
}}}

does not use item, something (non trivial) could be implemented to slice
the queryset at the right spot (depending on the number of languages), and
the total number of elements could be computed by a simple multiplication.

But as long as get_languages_for_item uses item all of this break.
--
Ticket URL: <https://code.djangoproject.com/ticket/36732#comment:1>

Django

unread,

Nov 14, 2025, 8:55:44 AM11/14/25

to django-...@googlegroups.com

Description changed by Julien Palard:

Old description:

> Hi!
>
> It's a bit like: https://code.djangoproject.com/ticket/11572
>
> when enabling i18n on a sitemap, _items uses the following code:
>
> ```python
> # Create (item, lang_code) tuples for all items and
> languages.
> # This is necessary to paginate with all languages already
> considered.
> items = [
> (item, lang_code)
> for item in self.items()
> for lang_code in self.get_languages_for_item(item)
> ]
> ```
>
> The list comprehension loads the whole table times the number of
> languages in memory.
>
> We tried with a table containing two millions elements and two languages:
> it does not fit in 16GB of memory and the process gets killed.

New description:

Hi!

It's a bit like: https://code.djangoproject.com/ticket/11572

when enabling i18n on a sitemap, _items uses the following code:

{{{

# Create (item, lang_code) tuples for all items and languages.
# This is necessary to paginate with all languages already
considered.
items = [
(item, lang_code)
for item in self.items()
for lang_code in self.get_languages_for_item(item)
]
}}}

The list comprehension loads the whole table times the number of languages
in memory.

We tried with a table containing two millions elements and two languages:
it does not fit in 16GB of memory and the process gets killed.

--