[Django] #18419: Language code is not correct for Chinese
Django
----------------------------------------+-------------------------------
Reporter: Olli Wang <olliwang@…> | Owner: nobody
Type: Bug | Status: new
Component: Internationalization | Version: 1.4
Severity: Normal | Keywords: i18n, chinese, zh
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------+-------------------------------
Currently Django uses zh_TW for Traditional Chinese and zh_CN for
Simplified Chinese. This should work fine, but may not in the correct way.
This is because Traditional Chinese is not used only by Taiwan (tw) but
also Honk Kong (hk), and Simplified Chinese is not only used in China (cn)
but also Singapore (sg) and Malaysia. A new standard is to use "zh_Hant"
for Traditional Chinese and "zh_Hans" for Simplified Chinese to eliminate
the confusion of the content is not just for people lived in China or
Taiwan.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Olli Wang | Owner: nobody
<olliwang@…> | Status: new
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage:
Severity: Normal | Unreviewed
Keywords: i18n, chinese, zh | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Changes (by Olli Wang <olliwang@…>):
* needs_better_patch: => 0
* needs_tests: => 0
* easy: 0 => 1
* needs_docs: => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:1>
Django
Reporter: Olli Wang | Owner: nobody
<olliwang@…> | Status: new
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage:
Severity: Normal | Unreviewed
Keywords: i18n, chinese, zh | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Comment (by ramiro):
Can you point us to an URL to [a document describing] such new standard?
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:2>
Django
Reporter: Olli Wang | Owner: nobody
<olliwang@…> | Status: new
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage:
Severity: Normal | Unreviewed
Keywords: i18n, chinese, zh | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Comment (by Olli Wang <olliwang@…>):
Replying to [comment:2 ramiro]:
> Can you point us to an URL to [a document describing] such new standard?
Apple has updated their developer guide to reflect the change. You may
want to visit the guide at
https://developer.apple.com/library/mac/#documentation/MacOSX/Conceptual/BPInternational/Articles/LanguageDesignations.html
and search "hant" or "hans".
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:3>
Django
Reporter: Olli Wang | Owner: nobody
<olliwang@…> | Status: new
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage:
Severity: Normal | Unreviewed
Keywords: i18n, chinese, zh | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Comment (by Olli Wang <olliwang@…>):
After I dig into the source code of Django. I found it may have some
issues to adapt the new standard. The first is the conversion between
local and language code. Locale is in the format of ll_CC, so for example,
the "zh_TW" locale can be converted to the "zh-tw" language code. But as
the new standard, language code for Traditional Chinese is "zh-Hant",
which is not possible to converted to the locale in ll_CC format because
"Hant" is not a country.
The second issue is country-specific codes should fallback to new standard
if not available. For example, currently most browsers only use "zh-tw",
"zh-cn", "zh-sg", "zh-hk" in the "HTTP_ACCEPT_LANGUAGE" field. In such
situation, Django should convert "zh-tw" and "zh-hk" to "zh-Hant" and
convert "zh-cn" and "zh-sg" to "zh-Hans". This should be done only if the
request one ("zh-tw", "zh-hk", "zh-cn", "zh-sg") is not set in the
django.conf.LANGUAGES setting.
The last part is Django should merge these translations. For example, if a
browser request a "zh-tw" language. Django should merge both "zh-tw" and
"zh-Hant" translation files.
Chinese is my mother language. Please free feel to ask me questions if you
are still confusing.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:4>
Django
Reporter: Olli Wang | Owner: nobody
<olliwang@…> | Status: new
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage:
Severity: Normal | Unreviewed
Keywords: i18n, chinese, zh | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Comment (by Olli Wang <olliwang@…>):
There is another good reason why Django should adopt "zh-Hant" for
Traditional Chinese and "zh-Hans" for Simplified Chinese. Imaging there is
a site visitor come from Hong Kong. The "HTTP_ACCEPT_LANGUAGE" field sent
by his browser may probably only include zh-hk but no zh-tw. In such
situation, even though both zh-hk and zh-tw return Traditional Chinese,
the Hong Kong user won't see the Traditional Chinese content because
Django see zh-hk doesn't match to zh-tw, and fallback to the default
language.
Of course some site administrator would still want to target Hong Kong or
Taiwan specific users. In such situation, he can still provide both zh-tw
and zh-hk translations. But most of the time, a Chinese site admin don't
care where the user come from at all. Just like en-us and en-uk, the site
admin may only want to provide one English version. So if Django support
zh-hant, no matter user come from Hong Kong or Taiwan, they can see
Traditional Chinese as wish. The same applies to "zh-hans", no matter the
user come from China or Singapore, they both see Simplified Chinese as
wish.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:5>
Django
Reporter: Olli Wang | Owner: nobody
<olliwang@…> | Status: new
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage:
Severity: Normal | Unreviewed
Keywords: i18n, chinese, zh | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Comment (by jaylett):
zh-Hant and zh-Hans are registered with IANA:
http://www.iana.org/assignments/lang-tags/zh-Hant and
http://www.iana.org/assignments/lang-tags/zh-Hans.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:6>
Django
Reporter: Olli Wang | Owner: nobody
<olliwang@…> | Status: new
Type: Bug | Version: 1.4
Component: | Resolution:
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 0 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by aaugustin):
* easy: 1 => 0
* stage: Unreviewed => Accepted
Comment:
I'm going to accept this ticket, but we need a good plan to address
backwards compatibility before this can move forward.
You could start a discussion on the django-i18n mailing list to get more
input.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:7>
Django
Reporter: Olli Wang | Owner: nobody
<olliwang@…> | Status: new
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 0 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: kitsunde@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:8>
Django
Reporter: Olli Wang | Owner: nobody
<olliwang@…> | Status: new
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 0 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by jezdez):
We can do it the same way we did with the two Norwegian translations, have
both translations (zh-Hant/-Hans and zh-TW/-CN) in core and deprecate the
latter after two releases.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:9>
Django
Reporter: Olli Wang | Owner: bouke
<olliwang@…> | Status: assigned
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* owner: nobody => bouke
* status: new => assigned
* has_patch: 0 => 1
* cc: bouke (added)
Comment:
I've created a pull request for this ticket:
https://github.com/django/django/pull/1868, based on the strategy proposed
by jezdez. So for Django 1.7 and 1.8 there are duplicate chinese
translations present; the old ones being deprecated. However when
researching I've also found that some browsers send the old (deprecated)
language codes, including the latest Firefox. So I've included a check
such that visitors with those browsers also get the correct display
language, despite incorrect `Accept-Language` headers.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:10>
Django
Reporter: Olli Wang | Owner: bouke
<olliwang@…> | Status: assigned
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by aaugustin):
The pull request looks good, but although you sent it a few hours ago it
doesn't apply properly. Did you work from an up-to-date copy of master?
If you bring it up to date, you can mark the ticket as RFC.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:11>
Django
Reporter: Olli Wang | Owner: bouke
<olliwang@…> | Status: assigned
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by aaugustin):
Fixed in c0a2388a1c4ead1afaec98e4ebc953a772ca3849. (Commit referenced
#18149 by mistake.)
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:12>
Django
Reporter: Olli Wang | Owner: bouke
<olliwang@…> | Status: assigned
Type: Bug | Version: 1.4
Component: | Resolution:
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by bouke):
Woops, my bad!
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:13>
Django
Reporter: Olli Wang | Owner: bouke
Type: Bug | Version: 1.4
Component: | Resolution: fixed
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:14>
Django
Reporter: Olli Wang | Owner: bouke
Type: Bug | Version: 1.4
Component: | Resolution:
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* status: closed => new
* resolution: fixed =>
Comment:
After discussing on IRC we found that (1) some unit tests failed after the
patch and (2) the upgrade path to be somewhat bumpy. The problem occurs
when `LANGUAGES` isn't overridden and would contain both old and new
language codes. Now if a browser comes along and requests the old language
code (which might include the most recent versions of IE and Firefox),
Django will upgrade this old language code to the new language code. Now,
if there is some customization done for the old language codes, Django
would thus use the new language code and ignored the translations. To
mitigate `get_supported_language_variant` also needs to check if the old
language code is still in the list of supported languages.
PR: https://github.com/django/django/pull/1872
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:15>
Django
Reporter: Olli Wang | Owner: bouke
<olliwang@…> | Status: new
Component: | Resolution:
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by bmispelon):
* cc: bmispelon (added)
* version: 1.4 => master
* needs_tests: 0 => 1
Comment:
The PR does fix the broken tests.
I think we should also add a test for the situation that broke the two
tests.
From what I understand, that means when `zh-cn` is in
`settings.LANGUAGES`.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:16>
Django
Reporter: Olli Wang | Owner: bouke
<olliwang@…> | Status: new
Type: Bug | Version: master
Component: | Resolution:
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by bouke):
* needs_tests: 1 => 0
Comment:
PR updated with added tests.
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:17>
Django
Reporter: Olli Wang | Owner: bouke
Type: Bug | Version: master
Component: | Resolution: fixed
Internationalization | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: i18n, chinese, zh | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"e5e044da87800feb6ef63fef1765d8c05022d926"]:
{{{
#!CommitTicketReference repository=""
revision="e5e044da87800feb6ef63fef1765d8c05022d926"
Fixed #18419 -- Full backwards compatibility for old language codes
Improved documentation about zh-* deprecation and upgrade path.
Thanks to Baptiste Mispelon for the code reviews.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/18419#comment:18>