**Suggestion**: It would be nice to have an easy way for finding out a
languange preference from request, even if it's not listed in your
supported languages. You could think of an optional flag, so that the
preference from `get_language_from_request` is checked against `LANG_INFO`
in the translation module rather than only against `settings.LANGUAGES`.
This would give you a locale from all languages that are available in the
translation module. You could use that e.g. for comparing with
`settings.LANGUAGE_CODE` and add a fallback for specific cases.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Comment (by Denis Anuschewski):
In my specific case I wrote a sub class from `LocaleMiddleware` in order
to use `add_fallback` whenever a language gets detected that is not equal
to `settings.LANGUAGE_CODE`. But that's not possible currently because you
always get the value of `settings.LANGUAGE_CODE` when the discovered
language is not in `settings.LANGUAGES`.
Overwriting the function `get_languages` didn't work for me because
translation was then activated for every language. I only want
translations for specific languages when I explicitly put them into
`settings.LANGUAGES`, but I want a fallback for every language except of
`settings.LANGUAGE_CODE`.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:1>
* keywords: translation, locale, request => translation,
internationalization, request
* component: Utilities => Internationalization
Old description:
> **Problem**: In the current state, the user's discovered language
> preference is only returned when found in `settings.LANGUAGES`. That's
> more than sufficient for the normal translation routine, but there seems
> to be no way (at least no DRY way) to find out a user's locale preference
> REGARDLESS of `settings.LANGUAGES`.
>
> **Suggestion**: It would be nice to have an easy way for finding out a
> languange preference from request, even if it's not listed in your
> supported languages. You could think of an optional flag, so that the
> preference from `get_language_from_request` is checked against
> `LANG_INFO` in the translation module rather than only against
> `settings.LANGUAGES`. This would give you a locale from all languages
> that are available in the translation module. You could use that e.g. for
> comparing with `settings.LANGUAGE_CODE` and add a fallback for specific
> cases.
New description:
**Problem**: In the current state, the user's discovered language
preference is only returned when found in `settings.LANGUAGES`. That's
more than sufficient for the normal translation routine, but there seems
to be no way (at least no DRY way) to find out a user's preference
REGARDLESS of `settings.LANGUAGES`.
**Suggestion**: It would be nice to have an easy way for finding out a
languange preference from request, even if it's not listed in your
supported languages. You could think of an optional flag, so that the
preference from `get_language_from_request` is checked against `LANG_INFO`
in the translation module rather than against `settings.LANGUAGES`. This
would give you one of all available languages in the translation module.
You could use that e.g. for comparing with `settings.LANGUAGE_CODE` and
add a fallback for specific cases.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:2>
* needs_docs: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:3>
Old description:
> **Problem**: In the current state, the user's discovered language
> preference is only returned when found in `settings.LANGUAGES`. That's
> more than sufficient for the normal translation routine, but there seems
> to be no way (at least no DRY way) to find out a user's preference
> REGARDLESS of `settings.LANGUAGES`.
>
> **Suggestion**: It would be nice to have an easy way for finding out a
> languange preference from request, even if it's not listed in your
> supported languages. You could think of an optional flag, so that the
> preference from `get_language_from_request` is checked against
> `LANG_INFO` in the translation module rather than against
> `settings.LANGUAGES`. This would give you one of all available languages
> in the translation module. You could use that e.g. for comparing with
> `settings.LANGUAGE_CODE` and add a fallback for specific cases.
New description:
**Problem**: The user's discovered language preference is only returned
when found in `settings.LANGUAGES`. That's more than sufficient for the
normal translation routine, but there seems to be no way (at least no DRY
way) to find out a user's preference REGARDLESS of `settings.LANGUAGES`.
**Suggestion**: It would be nice to have an easy way for finding out a
languange preference from request, even if it's not listed in your
supported languages. You could think of an optional flag, so that the
preference from `get_language_from_request` is checked against `LANG_INFO`
in the translation module rather than against `settings.LANGUAGES`. This
would give you one of all available languages in the translation module.
You could use that e.g. for comparing with `settings.LANGUAGE_CODE` and
add a fallback for specific cases.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:4>
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:5>
Comment (by Claude Paroz):
> You could use that e.g. for comparing with settings.LANGUAGE_CODE and
add a fallback for specific cases.
Could you please elaborate a bit more about the use case for that request?
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:6>
Comment (by Denis Anuschewski):
Replying to [comment:6 Claude Paroz]:
> > You could use that e.g. for comparing with settings.LANGUAGE_CODE and
add a fallback for specific cases.
>
> Could you please elaborate a bit more about the use case for that
request?
Yes of course.
Let's say we have following settings:
{{{
LANGUAGE_CODE = 'de-DE'
LANGUAGES = [
('en', 'English'),
('de', 'German'),
('es', 'Spanish'),
}}}
So we have German as standard language and the 3 languages in
`settings.LANGUAGES` as those which we want to have translated. And now
imagine we want to have a custom middleware that looks something like
this:
{{{
class CustomLocaleMiddleware(LocaleMiddleware):
def process_request(self, request):
super(CustomLocaleMiddleware, self).process_request(request)
lan = request.LANGUAGE_CODE
preference = translation.get_language_from_request(request,
support_all_languages=True)
if not settings.LANGUAGE_CODE.startswith(preference): #
Preference differs from LANGUAGE_CODE
translation.trans_real.translation(lan).add_fallback(
translation.trans_real.DjangoTranslation('en'))
}}}
So every language preference that gets identified by
`get_language_from_request` and differs from `settings.LANGUAGE_CODE` gets
English as fallback language, even if it's not in `settings.LANGUAGES`.
When you get e.g. ''pl'' as preference you get ''de'' as language, but
with ''en'' as fallback (that's what we want). With current Django you
would also get ''de'' as language, but no fallback because preference
would not be ''pl'' but ''de''. It would only work with "en", "de", and
"es".
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:7>
Comment (by Claude Paroz):
In which language are the original strings in your scenario?
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:8>
Comment (by Denis Anuschewski):
You have de-DE (German) in `LANGUAGE_CODE` and pl (Polish) in
`HTTP_ACCEPT_LANGUAGE` header of the user's request.
Currently, Django's output of `get_language_from_request` is pl if it is
in `settings.LANGUAGES` and de-DE if it is not.
With my modification you get pl when you explicitly set
`support_all_languages` to True, regardless of `settings.LANGUAGES`.
Check my PR for more clarification in the code base.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:9>
Comment (by Claude Paroz):
Yes, I understand the difference of output produced by your code. However,
I would like to completely understand your use case to be sure we are
fixing it at the right level.
Returning to your example, why would you like to get a language code which
is not in your site `LANGUAGES`? Why would the fallback language be
different depending on the requested language?
Maybe you could present a table whose columns are: language of original
strings, client language, LANGUAGE_CODE, LANGUAGES, resulting
language/fallback (one line with current code, one line with your wanted
situation).
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:10>
Comment (by Denis Anuschewski):
At first thanks a lot for the time and effort you put into my problem, it
is very much appreciated. I did not get what you meant by language of
original strings, sorry. You mean the de facto language of the strings
that are marked as translations I guess. It is German, just like the value
of `LANGUAG_CODE` (our project's working language).
So here is the table as requested:
|| **language of original strings** || **client language** ||
**LANGUAGE_CODE** || **LANGUAGES** || **resulting language** ||
**fallback**
|| de || pl || de || en, de, es || de || None
|| de || pl || de || en, de, es || pl || en
First row is current Django, second is my wanted behaviour.
> Returning to your example, why would you like to get a language code
which is not in your site `LANGUAGES`?
Because I want to detect a user with Polish language (or any other foreign
language) regardless of `LANGUAGES`. I dont't want to place ''pl'' in
`LANGUAGES` because I don't have and I don't want a Polish translation,
but I want to KNOW that I am dealing with a user with Polish language.
With current Django (1st row) I just don't have the chance because I get
the value of `LANGUAGE_CODE` as resulting language, so a Polish user gets
treated just like a German one.
> Why would the fallback language be different depending on the requested
language?
Because I don't want a fallback for German users. They shall only get
German texts no matter what because that's the de facto standard language
in the project. For any other case English shall be the desired fallback
so that we get the following order for strings marked as translated:
1) Try to get a translation for the client language (if it is in
`LANGUAGES`, so either for ''en'', ''de'' or ''es'')
2) If the client language differs from the one in `LANGUAGE_CODE`
(''de''=German), fall back to English
3) If there is no English fallback translation, just use the language of
original strings (German)
So to be clear: I dont want to alter the existing translation routine. I
just want to be able to get the right client language and afterwards add a
fallback language depending on the client languages' value.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:11>
Comment (by Claude Paroz):
I also appreciate your patience while explaining your use case to me :-)
Just one more question, wouldn't setting `LANGUAGE_CODE = 'en'` solve your
issue?
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:12>
Comment (by Denis Anuschewski):
No problem :-)
> Just one more question, wouldn't setting `LANGUAGE_CODE = 'en'` solve
your issue?
Actually that was the first thing I did. But no, it has one giant
drawback: because my language of original strings is German, setting
`LANGUAGE_CODE` to 'en' leads to the necessity of having to maintain
redundant German translations (or I get English translations for German
users). Additionally when changing the central `LANGUAGE_CODE` setting you
have to also change a lot of other stuff, e.g. in order for unit tests to
go through.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:13>
Comment (by Claude Paroz):
OK, I would suggest a slightly different approach. Could you have a look
to this commit and tell me your opinion?
https://github.com/django/django/compare/master...claudep:28636?expand=1
The idea is to allow for custom fallback by subclassing the
`LocaleMiddleware` and setting the `fallback_language` class variable
(which would be `'en'` in your case).
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:14>
Comment (by Denis Anuschewski):
Thx again. I tested your code and it really does exactly what I want!
You were right that I don't really need to know the request language if
it's not in `LANGUAGES`, because the resulting language is always 'en' for
these cases. So my custom middleware's `process_request` would look like
this:
{{{
def process_request(self, request):
self.fallback_language = 'en'
super(CustomMiddleWare, self).process_request(request)
lan = request.LANGUAGE_CODE
if not settings.LANGUAGE_CODE.startswith(lan):
translation.trans_real.translation(lan).add_fallback(DjangoTranslation(self.fallback_language))
}}}
Therefore I get either `LANGUAGE_CODE`, English or a language from
`LANGUAGES` with English translation fallback. Bingo!
Your approach seems cleaner and more readable for me, plus it affects less
code. But there is one little thing I am missing: I would like to see an
additional paragraph in the docstring of `get_language_from_request`
describing what fallback does. I would suggest something like this:
{{{
With a given fallback, this value will be returned instead of
settings.LANGUAGE_CODE if no user language could be found.
}}}
Otherwise I am not missing anything. If you make a pull request with the
little addition I mentioned, I am more than happy to declare it as a fix
for this bug and vote for it.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:15>
* version: 1.11 => master
* stage: Unreviewed => Accepted
* needs_tests: 0 => 1
Comment:
Thanks for testing, I'll made a pull request soon.
However, are you sure you need that special `process_request` method? What
about simply:
{{{
class CustomLocaleMiddleWare(LocaleMiddleware):
fallback_language = 'en'
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:16>
Comment (by Denis Anuschewski):
Yes, I'm sure. Because what you implemented as a fallback differs from
what `add_fallback` is providing:
Your fallback determines what translation language to use when none is
found. `add_fallback` on the other hand gives a fallback on string basis
when no translation for the string ist found in a translation file.
So with your code a Spanish user e.g. get's Spanish language, but missing
Spanish translations get displayed in German. The Goal ist to have an
English fallback in this case which `add_fallback` provides.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:17>
Comment (by Claude Paroz):
I'm still not completely satisfied by my proposal. It only partially
solves the issue, as shown by the hack you are forced to do in the
`process_request` middleware method.
I think we have a problem in Django i18n in that we generally assume
untranslated strings are in English (some problems you mentioned in
comment:13). We can see that for example in the `en` special-casing in
`DjangoTranslation._add_fallback`. We may miss a `LANGUAGE_SOURCE` setting
(even if we know that non-English original strings are problematic with
gettext when the language has more than 2 plurals). I don't know if that
would completely solve the current issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:18>
Comment (by Denis Anuschewski):
There are 3 things about this that come to my mind:
1)
> I think we have a problem in Django i18n in that we generally assume
untranslated strings are in English (some problems you mentioned in
comment:13). We can see that for example in the `en` special-casing in
`DjangoTranslation._add_fallback`.
I agree that having an exception for English in `add_fallback` just
because of how Django's English translation files are or might be
maintained seems awkward. But I would say that it doesn't really address
the core of my problem – that is adding a specific fallback language – so
it would be out of the scope of this ticket (maybe add a new ticket?).
2)
> We may miss a `LANGUAGE_SOURCE` setting (even if we know that non-
English original strings are problematic with gettext when the language
has more than 2 plurals).
I don't think this would work out because a `LANGUAGE_SOURCE` (assuming
that the language of original strings is meant) can be totally different
in your project and for Django's core translations. So in #24413 you would
have Polish as `LANGUAGE_SOURCE` I assume but it still wouldn't work for
this particular issue which was solved with the English exception patch in
`add_fallback`, because the language of original strings in Django's core
is always English. In my case it is German for my project so it would also
be in conflict with Django's language of original strings (so it either
works for your custom translations or for Django's built-in translations).
I would say if you explicitly use a specific language or add a specific
fallback you just have to make sure that translations for your language
are provided! After all you have the possibility to add missing
translations to your own .po files. Or for Django's English core
translation the solution could really be just to make sure all
translations in the .po files are provided, even if it's hellishly
redundant (because msgid and msgstr are always the same).
> I'm still not completely satisfied by my proposal. It only partially
solves the issue, as shown by the hack you are forced to do in the
`process_request` middleware method.
I wouldn't necessarily call it a 'hack' because `process_request` is not
plainly overwritten, it calls `process_request` of the super class and
adds additional stuff so it's nearly as maintainable as setting a flag. A
custom middleware has to be used either way.
But it might be that I have found an even better way to accomplish what we
want. I will test it tomorrow and then make a commit.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:19>
Comment (by Denis Anuschewski):
OK, here is what I ended up with: could you maybe take a look and tell me
what you think?
https://github.com/django/django/compare/master...claudep:28636?expand=1
With this approach you can set your custom fallback exactly like with your
changes by setting the middleware's class variable, but with an additional
fallback for missing translations.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:20>
Comment (by Denis Anuschewski):
I think I have an even better approach:
https://github.com/denisiko/django/commit/faeb8a4db34121a9cfe61b281849b084dfbe6625
Of course there is missing documentation and unit tests at this point. But
first I would like to get some feedback. What do you think Claude Paroz?
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:21>
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:22>
Comment (by Claude Paroz):
I think that this makes sense, please make it a pull request after
tests/docs have been added.
Adding new settings is not something we are doing lightly, though, but I
don't really see how we could avoid that for your use case. Maybe other
reviewers will suggest ideas.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:23>
Comment (by Denis Anuschewski):
Thanks for your review! I also think a new setting makes sense and fits
nicely with the other language settings. And not having to implement a
custom middleware is a big advantage and therefore justifies a new setting
in my opinion.
I am going to write the tests and the doc now and let you know when the PR
is ready.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:24>
Comment (by Denis Anuschewski):
I have a pull request now: https://github.com/django/django/pull/9248
Unfortunately, I could not find out why some tests are failing (has
something to do with failing requests returning 404's where something is
expected). Maybe you have an idea Claude Paroz?
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:25>
Comment (by Denis Anuschewski):
Nevermind, I fixed it myself :-)
The problem was my default of `LANGUAGE_FALLBACK` (I used `LANGUAGE_CODE`
for that). In normal cases it works fine, but in unit tests it gives
unintended results when you mock `LANGUAGE_CODE`. The language code then
changes, but the fallback stays the same, giving you English values as
result.
I solved this by setting `LANGUAGE_FALLBACK` to `None` by default, because
when it gets used in `get_fallback()` it will be overwritten with
`LANGUAGE_CODE` anyway if not set properly.
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:26>
--
Ticket URL: <https://code.djangoproject.com/ticket/28636#comment:27>