[Django] #33565: makemessages command skips the nl-nl-x-informal folder

Django

unread,

Mar 7, 2022, 12:06:40 PM3/7/22

to django-...@googlegroups.com

#33565: makemessages command skips the nl-nl-x-informal folder
-------------------------------------+-------------------------------------
Reporter: Ronnie | Owner: nobody
van den Crommenacker |
Type: Bug | Status: new
Component: | Version: 3.2
Internationalization |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
There is a mismatch on the DjangoTranslation folder and the makemessages
command for the value `nl-nl-x-informal`

DjangoTranslation uses the `to_locale` method to determine the language
folder to read the `django.po`. `to_locale` translates `nl-nl-x-informal`
correctly to the `nl_NL-x-informal` folder.

However makemessages skips the `nl_NL-x-informal` folder and displays the
following message
**invalid locale nl_NL-x-informal, did you mean nl_NL_x_informal?**

# This makemessages behaviour is introduced in commit
https://github.com/django/django/commit/f63f3cdf0969c23fd0c05de0f4a2a1df0cd5112e

The check for `-` in the locale should only be for the first section
a.k.a. `nl_NL`

--
Ticket URL: <https://code.djangoproject.com/ticket/33565>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Mar 8, 2022, 12:45:17 AM3/8/22

to django-...@googlegroups.com

#33565: makemessages command skips the nl-nl-x-informal folder
-------------------------------------+-------------------------------------

Reporter: Ronnie van den | Owner: nobody

Crommenacker |
Type: Bug | Status: new
Component: | Version: 3.2
Internationalization |

Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* cc: Claude Paroz (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:1>

Django

unread,

Mar 8, 2022, 3:43:59 AM3/8/22

to django-...@googlegroups.com

Comment (by Claude Paroz):

Could you tell us more about the need of this complicated `-x-informal`
stuff?

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:2>

Django

unread,

Mar 8, 2022, 8:49:04 AM3/8/22

to django-...@googlegroups.com

Comment (by Ronnie van den Crommenacker):

It's called a ''private-use subtag'' extension of the ''language_tag''.
Source: https://en.wikipedia.org/wiki/IETF_language_tag
>An optional private-use subtag, composed of the letter x and a hyphen
followed by subtags of one to eight characters each, separated by hyphens.

For our use it's because of our language has a ''more fomal'' and
''informal'' way of communicating. e.g. "Dear <name>" vs "Hi <name>". With
the use of the ''private-use subtag'', we can have 2 variants of the same
language.

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:3>

Django

unread,

Mar 8, 2022, 9:51:08 AM3/8/22

to django-...@googlegroups.com

Comment (by Mariusz Felisiak):

Replying to [comment:3 Ronnie van den Crommenacker]:

> It's called a ''private-use subtag'' extension of the ''language_tag''.
> Source: https://en.wikipedia.org/wiki/IETF_language_tag

> Source: https://www.w3.org/International/articles/language-
tags/#extension

> >An optional private-use subtag, composed of the letter x and a hyphen
followed by subtags of one to eight characters each, separated by hyphens.

`makemessages --locale` accepts locales in ISO/IEC 15897 format, not IETF
language tags. I'm not sure what private subtags are formatted in ISO/IEC
15897 🤔.

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:4>

Django

unread,

Mar 8, 2022, 10:45:04 AM3/8/22

to django-...@googlegroups.com

Comment (by Ronnie van den Crommenacker):

I cannot find examples of valid ISO/IEC 15897 private subtags either.

But i found some, non standard locales:
https://www.localeplanet.com/icu/sr-Latn-BA/index.html `sr_Latn_BA` (this
case is mentioned in the django's `to_locale` comment)
https://www.localeplanet.com/icu/en-US-POSIX/index.html `en_US_POSIX`

So looking at those examples i would suspect the `to_locale` should return
`nl_NL_X_INFORMAL` or `nl_NL_XINFORMAL` or `nl_Informal_NL`

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:5>

Django

unread,

Mar 10, 2022, 4:40:02 AM3/10/22

to django-...@googlegroups.com

Comment (by Carlton Gibson):

Hi Ronnie.

> The check for - in the locale should only be for the first section
a.k.a. nl_NL

Do you have a particular patch in mind? Do you want to make a PR?

I'm not sure reading whether this is really supported or not, but if a fix
is small and doesn't break anything, then it's easier to think positively
about. (If it's disruptive, the opposite might apply...)

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:6>

Django

unread,

Mar 16, 2022, 4:30:44 AM3/16/22

to django-...@googlegroups.com

#33565: makemessages command skips the nl-nl-x-informal folder
-------------------------------------+-------------------------------------
Reporter: Ronnie van den | Owner: nobody
Crommenacker |

Type: Bug | Status: closed
Component: | Version: 3.2
Internationalization |
Severity: Normal | Resolution: needsinfo

Changes (by Carlton Gibson):

* status: new => closed
* resolution: => needsinfo

Comment:

I'm going to mark this as needs info. If someone can point to clearer
specs, or provide a PoC patch, happy to review… (but pending that 🤷)

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:7>

Django

unread,

Mar 17, 2022, 8:19:35 AM3/17/22

to django-...@googlegroups.com

#33565: makemessages command skips the nl-nl-x-informal folder
-------------------------------------+-------------------------------------
Reporter: Ronnie van den | Owner: nobody
Crommenacker |

Type: Bug | Status: new

Component: | Version: 3.2
Internationalization |
Severity: Normal | Resolution:

Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Ronnie van den Crommenacker):

* status: closed => new
* has_patch: 0 => 1
* resolution: needsinfo =>

Comment:

Created a pull request: https://github.com/django/django/pull/15521

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:8>

Django

unread,

Mar 24, 2022, 7:17:48 AM3/24/22

to django-...@googlegroups.com

Comment (by Carlton Gibson):

Hi Ronnie.

Based on the [https://www.w3.org/International/articles/language-
tags/#extension W3.org article], I'd say this isn't something we should
really support:

> Because these subtags are only meaningful within private agreements and
cannot be used interoperably across the Web, they should be used with
great care, and avoided whenever possible.

So I would say `wontfix`.

Then I look at #28546, and see that we did add support in `to_locale`… —
That was your ticket too. I'm guessing you may be the only person using
this. 🙂

Can I ask, how come it's taken 5 years for you to hit the issue with
`makemessages`?
Would a project level custom `makemessages` serve?
This would mean we don't need to add the complication for few (just you?)
users 🤔

I have to say I'm still not sure. The patch isn't that big, so we could
add it for symmetry with #28546.

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:9>

Django

unread,

Mar 24, 2022, 7:53:50 AM3/24/22

to django-...@googlegroups.com

Comment (by Ronnie van den Crommenacker):

> Based on the [https://www.w3.org/International/articles/language-
tags/#extension W3.org article], I'd say this isn't something we should
really support:
>
> > Because these subtags are only meaningful within private agreements
and cannot be used interoperably across the Web, they should be used with
great care, and avoided whenever possible.
>
> So I would say `wontfix`.

The way i read the comment is "it should be ~used~ with great care", which
doesn't not say anything about the ~support~ for the private extension.
It exists in the `language-tags` for a reason, but for `locales` it seems
not that clear in `ISO/IEC 15897` what the restrictions to the format
actually are. I have seen some uncommon formats in the wild (see comment
https://code.djangoproject.com/ticket/33565?replyto=9#comment:5).

I understand that if something should not be used there is little reason
to support it. However, Django has added support for private subtags in
the `to_locale` function already via #28546 and it used to work before
Django 3.2.

> Then I look at #28546, and see that we did add support in `to_locale`… —
That was your ticket too. I'm guessing you may be the only person using
this. 🙂

Yeah, i think i'm the only one. It was (and is) still the perfect solution
for the problem we (as a company) are facing. Other solutions would be far
more complex.

> Can I ask, how come it's taken 5 years for you to hit the issue with
`makemessages`?
> Would a project level custom `makemessages` serve?
> This would mean we don't need to add the complication for few (just
you?) users 🤔

The `makemessages` command worked (maybe by accident) perfectly for many
years, until we updated to Django 3.2 recently.
This commit included in Django 3.2 added extra checks which, by accident,
stopped the support for private subtags
https://github.com/django/django/commit/f63f3cdf0969c23fd0c05de0f4a2a1df0cd5112e

As a temporary solution we rename `nl_NL_x_informal` to `nl_NL-x-informal`
after we run makemessages

> I have to say I'm still not sure. The patch isn't that big, so we could
add it for symmetry with #28546.

I hope you will reconsider to include the patch. I'll understand Django
wouldn't support it officially, document it and test for it in upcoming
releases. If it occasionally breaks, ill understand and i am happy to
contribute patches when it breaks again.

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:10>

Django

unread,

Mar 24, 2022, 9:32:36 AM3/24/22

to django-...@googlegroups.com

Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* stage: Unreviewed => Accepted

Comment:

> ... i am happy to contribute patches when it breaks again.

OK, let's take it for review then :) If no-one objects to the PR, then you
can be on the hook to help maintain it. 😄

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:11>

Django

unread,

Apr 28, 2022, 2:51:39 AM4/28/22

to django-...@googlegroups.com

#33565: makemessages command skips the nl-nl-x-informal folder
-------------------------------------+-------------------------------------

Reporter: Ronnie van den | Owner: Ronnie
Crommenacker | van den Crommenacker
Type: Bug | Status: assigned

Changes (by Mariusz Felisiak):

* owner: nobody => Ronnie van den Crommenacker
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:12>

Django

unread,

May 24, 2022, 6:03:52 AM5/24/22

to django-...@googlegroups.com

#33565: makemessages command skips the nl-nl-x-informal folder
-------------------------------------+-------------------------------------
Reporter: Ronnie van den | Owner: Ronnie
Crommenacker | van den Crommenacker
Type: Bug | Status: assigned
Component: | Version: 3.2
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* needs_tests: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:13>

Django

unread,

Jun 7, 2022, 5:12:03 AM6/7/22

to django-...@googlegroups.com

#33565: makemessages command skips the nl-nl-x-informal folder
-------------------------------------+-------------------------------------
Reporter: Ronnie van den | Owner: Ronnie
Crommenacker | van den Crommenacker
Type: Bug | Status: assigned
Component: | Version: 3.2
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Carlton Gibson):

* needs_tests: 1 => 0

Comment:

Patch is updated. Thanks Ronnie. Unchecking for another review.

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:14>

Django

unread,

Jun 8, 2022, 10:21:58 AM6/8/22

to django-...@googlegroups.com

#33565: makemessages command skips the nl-nl-x-informal folder
-------------------------------------+-------------------------------------
Reporter: Ronnie van den | Owner: Ronnie
Crommenacker | van den Crommenacker
Type: Bug | Status: assigned
Component: | Version: 3.2
Internationalization |
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Carlton Gibson):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:15>

Django

unread,

Jun 9, 2022, 3:38:02 AM6/9/22

to django-...@googlegroups.com

#33565: makemessages command skips the nl-nl-x-informal folder
-------------------------------------+-------------------------------------
Reporter: Ronnie van den | Owner: Ronnie
Crommenacker | van den Crommenacker

Type: Bug | Status: closed
Component: | Version: 3.2
Internationalization |
Severity: Normal | Resolution: fixed

Changes (by Carlton Gibson <carlton.gibson@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"c32858a8ce961d276215a040ae0ab1e4409b70f8" c32858a8]:
{{{
#!CommitTicketReference repository=""
revision="c32858a8ce961d276215a040ae0ab1e4409b70f8"
Fixed #33565 -- Improved locale format validation for the makemessages
command.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/33565#comment:16>

Reply all

Reply to author

Forward