Missing support for gettext fallback translations

[Matt Westcott]

Hi all,

I've noticed that the django.utils.translation module apparently doesn't recognise gettext's handling of fallback translations when both a territory-specific translation catalog (e.g. Spanish (Spain), 'es_ES') and a generic one (e.g. Spanish, 'es') exist.

According to Python's locale logic (specifically gettext._expand_lang and locale.normalize), the territory-specific catalog always takes precedence - even when only the generic 'es' locale was requested - and so when the gettext module constructs a translation catalog for Spanish, it uses the 'es_ES' dictionary with 'es' as a fallback. When django.utils.translation assembles the final merged catalog to apply Django's own precedence rules, it doesn't check for the existence of fallbacks on the individual gettext objects, so we end up not using the 'es' translations at all.

We just got bitten by this on the Wagtail project, because the last version added a small and incomplete 'es_ES' translation, inadvertently causing our existing (comprehensive) 'es' translations to disappear from view: https://github.com/wagtail/wagtail/issues/3600

I haven't been able to find any previous mention of this gotcha, which surprises me a bit since it seems like quite a fundamental part of gettext's functionality. I've come up with a quick-and-dirty fix at https://github.com/wagtail/wagtail/issues/3600#issuecomment-302354552, but a proper solution that respects both gettext's and Django's precedence rules is probably going to be a lot more involved. So, before I go ahead and work on that fix, I was wondering if A) this is legitimately a missing feature rather than just our translation management not being up to scratch, and B) whether there's been any previous work on addressing this?

Cheers,

- Matt

[Ramiro Morales]

If this isn't a bug IMHO then it would be a nice to have feature because of predictability, being a good citizen to the python/gettext ecosystem and to do a good work for our users by providing for reuse of hopefully more helpful translations.

In the "it's a bug" case, chances are it's a regression. And I really hope it wasn't me who introduced it. It would be interesting to test some really old Django version (circa 1.0) and if needed, perform a bisection.

I'm also surprised by your findings. I guess it's something we simply took for granted. It's mentioned in the [1]docs and has been so for [2]years.

I found a couple of threads somewhat related to the topic:

https://groups.google.com/d/topic/django-users/EaPVUi2KMhE/discussion

https://groups.google.com/d/topic/django-developers/gDU8HlZg4Ug/discussion (this is actually about JS i18n but drove unification of   the translation catalog handling code for JS and backend.)

Thanks for pointing out this issue.

Regards,

1. https://github.com/django/django/blob/master/docs/topics/i18n/translation.txt#L1996-L1999
2. https://github.com/django/django/blob/attic/i18n/docs/translation.txt#L267-L270

--

Ramiro Morales
@ramiromorales

[Patryk Zawadzki]

W dniu niedziela, 21 maja 2017 01:35:28 UTC+2 użytkownik Ramiro Morales napisał:

I'm also surprised by your findings. I guess it's something we simply took for granted. It's mentioned in the [1]docs and has been so for [2]years.

It's not the same case though. Docs say that if `de-at` is not available, Django will use `de` and that's the case. What does not work is the case where `de-at` exists but is a sparse catalog of only translation differences specific to `de-at`. In that situation translations missing from `de-at` should be filled in by falling back to the generic `de` catalog but Django doesn't support it.