[Django] #36980: TranslationCatalog.__getitem__ gives wrong priority to non-plural translations when plural forms mismatch

[Django]

#36980: TranslationCatalog.__getitem__ gives wrong priority to non-plural
translations when plural forms mismatch
-------------------------------------+-------------------------------------
Reporter: UHHHHHHHHHHHHHH | Type: Bug
Status: new | Component:
| Internationalization
Version: 5.1 | Severity: Normal
Keywords: i18n translation | Triage Stage:
plural TranslationCatalog | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
= Description =

When a third-party package (e.g. django-allauth) has a slightly different
`Plural-Forms` header than the project's own translations (e.g. `plural=n
!= 1` vs `plural=(n != 1)`), `TranslationCatalog.update()` correctly
prepends a separate catalog to preserve plural lookup correctness.

However, `TranslationCatalog.__getitem__()` iterates catalogs in order and
returns the '''first match'''. This means the prepended catalog wins for
'''all''' lookups, including non-plural `gettext()` calls where plural
form separation is irrelevant. This silently overrides higher-priority
translations (from `LOCALE_PATHS` or later `INSTALLED_APPS`) with lower-
priority ones.

= Steps to reproduce =

1. Create a Django project with a translation in `LOCALE_PATHS`:
{{{
# project/locale/nl/LC_MESSAGES/django.po
# Plural-Forms: nplurals=2; plural=(n != 1);
msgid "Activate"
msgstr "Activeren"
}}}

2. Install a third-party app (e.g. django-allauth) that defines the same
msgid with a different `Plural-Forms` header:
{{{
# allauth/locale/nl/LC_MESSAGES/django.po
# Plural-Forms: nplurals=2; plural=n != 1;
msgid "Activate"
msgstr "Activeer"
}}}

Note: `n != 1` and `(n != 1)` are functionally identical and compile to
identical bytecode, but Python's `gettext.c2py()` produces functions with
different `__code__` objects. Django's `TranslationCatalog.update()`
compares `__code__` to decide whether to merge or prepend.

3. `gettext("Activate")` returns `"Activeer"` (allauth's version) instead
of `"Activeren"` (the project's version), even though `LOCALE_PATHS`
should have highest priority per Django's documentation.

= Root cause =

In `django/utils/translation/trans_real.py`:

{{{#!python
class TranslationCatalog:
def update(self, trans):
# Mismatched plural -> prepend to position 0
for cat, plural in zip(self._catalogs, self._plurals):
if trans.plural.__code__ == plural.__code__:
cat.update(trans._catalog)
break
else:
self._catalogs.insert(0, trans._catalog.copy())
self._plurals.insert(0, trans.plural)

def __getitem__(self, key):
# First catalog wins for ALL lookups
for cat in self._catalogs:
try:
return cat[key]
except KeyError:
pass
raise KeyError(key)
}}}

The `update()` prepend is correct for plural lookups (each catalog needs
its own plural function). But `__getitem__` shouldn't give the prepended
catalog priority for non-plural string lookups — these should follow the
normal priority order (LOCALE_PATHS > INSTALLED_APPS > Django built-in).

= Expected behavior =

Non-plural `gettext()` lookups should respect the documented translation
priority order regardless of plural form differences between catalogs.

= Actual behavior =

A third-party package with a cosmetically different `Plural-Forms` header
silently overrides all matching non-plural translations from higher-
priority sources.

= Impact =

In our project, this causes 28 translations to be silently wrong. The
project has no way to fix this without either:
* Monkey-patching `TranslationCatalog`
* Patching the third-party package's `.po` files in the Docker build
* Adding `pgettext` context to every conflicting string

= Environment =

* Django 5.1 (also affects 5.0, likely all versions since #34221 was
fixed)
* Python 3.12
* The issue was introduced/amplified by the fix for #34221 which changed
`update()` to only merge with the top catalog
--
Ticket URL: <https://code.djangoproject.com/ticket/36980>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36980: TranslationCatalog.__getitem__ gives wrong priority to non-plural
translations when plural forms mismatch
-------------------------------------+-------------------------------------

Reporter: UHHHHHHHHHHHHHH | Owner: (none)
Type: Bug | Status: closed
Component: | Version: 5.1
Internationalization | Resolution:
Severity: Normal | worksforme

Keywords: i18n translation | Triage Stage:
plural TranslationCatalog | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* resolution: => worksforme
* status: new => closed

Comment:

I couldn't reproduce:

{{{#!py
In [4]: gettext('Activate')
Out[4]: 'Activeren'
}}}

After removing my LOCALE_PATHS translations, I get allauth's translation:

{{{#!py
In [3]: gettext('Activate')
Out[3]: 'Activeer'
}}}

Did you recompile messages?
--
Ticket URL: <https://code.djangoproject.com/ticket/36980#comment:1>

[Django]

#36980: TranslationCatalog.__getitem__ gives wrong priority to non-plural
translations when plural forms mismatch
-------------------------------------+-------------------------------------
Reporter: UHHHHHHHHHHHHHH | Owner: (none)
Type: Bug | Status: closed
Component: | Version: 5.1
Internationalization | Resolution:
Severity: Normal | worksforme
Keywords: i18n translation | Triage Stage:
plural TranslationCatalog | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by UHHHHHHHHHHHHHH):

The reproduction depends on whether the `Plural-Forms` headers actually
mismatch. The bug only triggers when they do.

'''What allauth ships (nl):'''
{{{
"Plural-Forms: nplurals=2; plural=n != 1;\n"
}}}

'''What Django's own nl translations use:'''
{{{
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
}}}

These are functionally identical, but `gettext.c2py()` produces functions
with different `__code__` objects:

{{{#!python
>>> from gettext import c2py
>>> c2py('(n != 1)').__code__ == c2py('n != 1').__code__
False
}}}

If your allauth version's `Plural-Forms` happens to match Django's (with
parentheses), everything merges into a single catalog and `LOCALE_PATHS`
correctly wins. That's likely why you couldn't reproduce.

'''Here's the exact loading trace on Django 5.1 with allauth 65.15.0:'''

1. Django's own nl loaded → `catalogs = [{django}(P1)]` where P1 = `(n !=
1)`
2. allauth loaded → `P2 = n != 1`, iterates catalogs, `P2.__code__ !=
P1.__code__` → '''prepended''' → `catalogs = [{allauth}(P2),
{django}(P1)]`
3. LOCALE_PATHS loaded → `P1 = (n != 1)`, iterates catalogs: doesn't match
P2 at position 0, '''matches P1 at position 1''' → merged into Django's
catalog → `catalogs = [{allauth}(P2), {django+project}(P1)]`

`__getitem__()` iterates in order → allauth at position 0 wins for all
shared msgids.

'''Minimal reproducer (4 files):'''

`requirements.txt`:
{{{
django>=5.1,<5.2
django-allauth>=65.0
}}}

`settings.py`:
{{{#!python
import os

BASE_DIR = os.path.dirname(os.path.abspath(__file__))

SECRET_KEY = "not-secret"
USE_I18N = True
LANGUAGE_CODE = "nl"

LOCALE_PATHS = [os.path.join(BASE_DIR, "locale")]

INSTALLED_APPS = [
"allauth",
]
}}}

`locale/nl/LC_MESSAGES/django.po` (compile with `msgfmt -o django.mo
django.po`):
{{{
# Project translations — should have highest priority via LOCALE_PATHS
msgid ""
msgstr ""
"Language: nl\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"

msgid "Incorrect password."
msgstr "Wachtwoord onjuist (PROJECT OVERRIDE)."
}}}

`reproduce.py`:
{{{#!python
import os
import django.conf

django.conf.settings.configure(
USE_I18N=True,
LANGUAGE_CODE="nl",
LOCALE_PATHS=[os.path.join(os.path.dirname(__file__), "locale")],
INSTALLED_APPS=["allauth"],
)

import django
django.setup()

from django.utils.translation import activate, gettext
from django.utils.translation.trans_real import _translations

activate("nl")

trans = _translations.get("nl")
if trans:
print(f"Number of catalogs: {len(trans._catalog._catalogs)}")
for i, (cat, plural) in enumerate(zip(trans._catalog._catalogs,
trans._catalog._plurals)):
print(f"\n Catalog {i}: {len(cat)} entries")
if "Incorrect password." in cat:
print(f" 'Incorrect password.' -> {cat['Incorrect
password.']!r}")

result = gettext("Incorrect password.")
expected = "Wachtwoord onjuist (PROJECT OVERRIDE)."

print(f"\ngettext('Incorrect password.') = {result!r}")

if result == expected:
print("\nOK: LOCALE_PATHS translation wins (correct behavior)")
else:
print(f"\nBUG: Expected LOCALE_PATHS: {expected!r}")
print(f" Got allauth's version: {result!r}")
}}}

'''Output:'''
{{{
Number of catalogs: 2

Catalog 0: 377 entries
'Incorrect password.' -> 'Ongeldig wachtwoord.'

Catalog 1: 365 entries
'Incorrect password.' -> 'Wachtwoord onjuist (PROJECT OVERRIDE).'

gettext('Incorrect password.') = 'Ongeldig wachtwoord.'

BUG: Expected LOCALE_PATHS: 'Wachtwoord onjuist (PROJECT OVERRIDE).'
Got allauth's version: 'Ongeldig wachtwoord.'
}}}

'''Side note:''' In Django 5.2, `update()` was changed to only compare
with `_plurals[0]` (the top catalog) instead of iterating all catalogs.
This accidentally masks this specific scenario — LOCALE_PATHS also gets
prepended and lands at position 0, so the project wins. But the underlying
design issue (that `__getitem__` gives prepended catalogs priority for
non-plural lookups) remains.
--
Ticket URL: <https://code.djangoproject.com/ticket/36980#comment:2>

[Django]

#36980: TranslationCatalog.__getitem__ gives wrong priority to non-plural
translations when plural forms mismatch
-------------------------------------+-------------------------------------
Reporter: UHHHHHHHHHHHHHH | Owner: (none)
Type: Bug | Status: closed
Component: | Version: 5.1
Internationalization | Resolution:
Severity: Normal | worksforme
Keywords: i18n translation | Triage Stage:
plural TranslationCatalog | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls):

Oh, so it affects Django 5.1 only. That's why it worked for me. 5.1
doesn't receive bug fixes any more. Thanks for clarifying.
--
Ticket URL: <https://code.djangoproject.com/ticket/36980#comment:3>