ngettext_lazy and ngettext

689 views
Skip to first unread message

אורי

unread,
Nov 1, 2019, 4:18:39 AM11/1/19
to Django developers (Contributions to Django itself)
Django developers,

I opened ticket https://code.djangoproject.com/ticket/30939 and I was referred to https://code.djangoproject.com/ticket/30439 as a duplicate. I don't understand why you need such a long formula to decide if a number is singular or plural. If it's 1 it's singular, otherwise it's plural. What's wrong about that?

If you want to define a function with 4 strings for 4 different numbers, you should define a new function and not edit ngettext_lazy and ngettext. My code which works with all Django versions up to 2.1, breaks with Django 2.2. This issue prevents me from upgrading Django to 2.2. Is this change documented in the release notes? I didn't find it there. I would like to keep using ngettext_lazy with 2 strings (when the number is either 1 or not 1) and not with 4 strings. Actually I always use plural but because similar validators used ngettext_lazy (MinLengthValidator and MaxLengthValidator in django.core.validators) I also used it in my validators. But the number I'm using is never equal to 1.

You can see my code here (search for "ngettext_lazy"):

And the relevant Django code:

אורי

unread,
Nov 1, 2019, 4:47:40 AM11/1/19
to Django developers (Contributions to Django itself)
Hi,

I want to add that it was very simple to upgrade Django from 1.11 to 2.0 and then to 2.1, I only had to change minor changes in my code - changing to class-based views and updating my tests to reflect the 403 pages (instead of a 302 redirect), so thanks for maintaining Django and for not breaking too many things when upgrading Django from one version to another.

I hope you will agree to revert the changes done to ngettext_lazy and ngettext in version 2.2 and then I will be able to upgrade Django to the next LTS version.

Claude Paroz

unread,
Nov 1, 2019, 3:29:41 PM11/1/19
to Django developers (Contributions to Django itself)
Hi Uri,

I think you did not understand well the problem. Let me try to explain it in more details.

Nothing was changed with "ngettext" or "ngettext_lazy" in Django. The plural equation for Hebrew was changed on the Transifex platform and those changes reach Django each time we synchronize translations with Transifex (a script-driven process). Note that it was the case for some other languages too.

The plural equation defines how many different plural forms can exist for any language. English has two, Arabic has six, etc.
Apparently, someone discovered/decided that Hebrew can have 4 different plurals in certain cases, that's probably why Transifex changed it, but I cannot tell you more about the rationales. By reading the new Hebrew equation, it looks like the plural can be different if the number is 1, 2, (20, 30, 40, etc.), or anything else.

The problem with Django, which is the object of #30439, is that Django is using only the plural equation of the core django.po file, so if you have several .po files with different plural equations, the plural may be wrong or can fallback to English. Typically in your case, the core Django po file has an equation with 4 plurals, while your Speedy.net po file has only 2, so when the translation infrastruction is looking for the 3rd or 4th form, it finds nothing and return the English string. You should be able to fix that by re-doing a makemessages in your project in a Django 2.2 environment, and you should find that your updated po file will have 4 plurals for each plural string (but then running your project with older Django versions can have translation problems again :-( ).

Hopefully someone will have the time to prepare a patch to solve #30429 in order that plural string changes can be easily managed.

Regards,

Claude

אורי

unread,
Nov 1, 2019, 5:08:52 PM11/1/19
to Django developers (Contributions to Django itself)
Hi Claude,

Thanks for the explanation, I understand the problem better now. I was not aware that the number of singular and plural forms depends on the language. I would like to clarify a few things:

1. Some of the strings used by "ngettext" or "ngettext_lazy" are unique to Speedy Net, some are not. Anyway I added translations for all of them in the Speedy Net django.po files, for the case that if Django will change the strings, they will still be translated.

2. When running ./make_all_messages.sh in Django 2.2, which runs make_messages for the locales en and he, nothing changes in the django.po files (except the date) - a third and fourth line are not added.

3. I think Django should handle django.po files when there are only 2 forms - singular and plural - and use the same plural forms for all forms of plurals, whatever that means. And in many cases the plural forms are identical, as I found out for example in https://github.com/django/django/blob/master/django/contrib/auth/locale/he/LC_MESSAGES/django.po#L223-L233. Actually from my knowledge of Hebrew, in most cases the plural form is the same for all numbers. In my case for example, Speedy Net, I only use plural although the singular is defined. I just copied the definition from Django validators with "ngettext_lazy" included. So I think there should be a fallback, so if only 2 forms are defined - singular and plural - then Django should use the same plural form for all plurals. This will also provide a backward compatibility to previous versions of Django.

4. Everything worked for me with Django versions up to 2.1. I think if such a big change is introduced to Django, which is not backward compatible, then new functions should be introduced by leaving "ngettext" or "ngettext_lazy" as they are. Or at least there should be some setting to allow backward compatibility. Especially since updating the django.po files might cause problems when downgrading to previous versions of Django, which don't support it.

5. If running make_messages will introduce 2 more plural forms in the django.po files, I expect the new forms to default to the plural form currently defined, to allow backward compatibility. And also, these functions should work when downgrading to a previous version of Django.

6. I never used Transifex directly. I only use Django. I never even heard about Transifex until today. Django developers can do whatever they want with third party dependencies. What matters to me is how Django works.

Thanks,

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9d8ade17-3d6e-46dc-8ad7-a6b27de90593%40googlegroups.com.

אורי

unread,
Nov 2, 2019, 12:43:05 AM11/2/19
to Django developers (Contributions to Django itself)
Hi,

I checked and I didn't find anything related to this issue documented on the Django 2.2 release notes. Is it documented?

אורי

unread,
Nov 23, 2019, 6:32:31 AM11/23/19
to Django developers (Contributions to Django itself)
Django developers,


I think Django 2.2 (which is LTS) should be updated in a way that using these functions to translate will be backward-compatible, so that everything that worked with Django up to 2.1 will keep working in Django 2.2 and above. Currently this is not the case in Hebrew with Speedy Net, which starts displaying error messages in English in the Hebrew websites (Speedy Net & Speedy Match) when I upgrade Django to 2.2, which is not what I expect. In Django up to 2.1 these error messages are translated to Hebrew, which is what I expect.

After encountering this problem I decided to keep Speedy Net in Django 2.1 (which I upgraded from 1.11 about 3 weeks ago) until there is a solution, or at least until I can make Speedy Net work in Hebrew in Django 2.2 and above. However, I understand the end-of-life of Django 2.1 is coming soon, which means we will be using an obsolete version of Django in a production website.

As long as this issue is not resolved, I would appreciate if you will extend the end-of-life date of Django 2.1 to at least one or two months after this issue will be resolved. Yes I know I can downgrade Django to 1.11 and use it until April 2020, but since I already upgraded to 2.1 I'm not sure it's a good idea to downgrade Django now. And also, since Django 2.2 was released almost 8 months ago, I'm not sure if this problem will be resolved before April 2020. #30439 was opened about 7 months ago and it's still not resolved.

Adam Johnson

unread,
Nov 23, 2019, 7:29:25 AM11/23/19
to django-d...@googlegroups.com
Uri,

I understand this is frustrating for you as a user. Thank you for documenting your issues extensively so they're easy to follow. However it seems like a complicated problem and that Claude is the only person spending significant time working on translations these days. If you have some resources to try help fix the bug, or document the workaround (running makemessages after upgrade?) that would be helpful to the community.

Thanks,

Adam

‪On Sat, 23 Nov 2019 at 11:32, ‫אורי‬‎ <u...@speedy.net> wrote:‬
Django developers,


I think Django 2.2 (which is LTS) should be updated in a way that using these functions to translate will be backward-compatible, so that everything that worked with Django up to 2.1 will keep working in Django 2.2 and above. Currently this is not the case in Hebrew with Speedy Net, which starts displaying error messages in English in the Hebrew websites (Speedy Net & Speedy Match) when I upgrade Django to 2.2, which is not what I expect. In Django up to 2.1 these error messages are translated to Hebrew, which is what I expect.

After encountering this problem I decided to keep Speedy Net in Django 2.1 (which I upgraded from 1.11 about 3 weeks ago) until there is a solution, or at least until I can make Speedy Net work in Hebrew in Django 2.2 and above. However, I understand the end-of-life of Django 2.1 is coming soon, which means we will be using an obsolete version of Django in a production website.

As long as this issue is not resolved, I would appreciate if you will extend the end-of-life date of Django 2.1 to at least one or two months after this issue will be resolved. Yes I know I can downgrade Django to 1.11 and use it until April 2020, but since I already upgraded to 2.1 I'm not sure it's a good idea to downgrade Django now. And also, since Django 2.2 was released almost 8 months ago, I'm not sure if this problem will be resolved before April 2020. #30439 was opened about 7 months ago and it's still not resolved.

‪On Fri, Nov 1, 2019 at 10:47 AM ‫אורי‬‎ <u...@speedy.net> wrote:‬
Hi,

I want to add that it was very simple to upgrade Django from 1.11 to 2.0 and then to 2.1, I only had to change minor changes in my code - changing to class-based views and updating my tests to reflect the 403 pages (instead of a 302 redirect), so thanks for maintaining Django and for not breaking too many things when upgrading Django from one version to another.

I hope you will agree to revert the changes done to ngettext_lazy and ngettext in version 2.2 and then I will be able to upgrade Django to the next LTS version.

‪On Fri, Nov 1, 2019 at 10:18 AM ‫אורי‬‎ <u...@speedy.net> wrote:‬
Django developers,

I opened ticket https://code.djangoproject.com/ticket/30939 and I was referred to https://code.djangoproject.com/ticket/30439 as a duplicate. I don't understand why you need such a long formula to decide if a number is singular or plural. If it's 1 it's singular, otherwise it's plural. What's wrong about that?

If you want to define a function with 4 strings for 4 different numbers, you should define a new function and not edit ngettext_lazy and ngettext. My code which works with all Django versions up to 2.1, breaks with Django 2.2. This issue prevents me from upgrading Django to 2.2. Is this change documented in the release notes? I didn't find it there. I would like to keep using ngettext_lazy with 2 strings (when the number is either 1 or not 1) and not with 4 strings. Actually I always use plural but because similar validators used ngettext_lazy (MinLengthValidator and MaxLengthValidator in django.core.validators) I also used it in my validators. But the number I'm using is never equal to 1.

You can see my code here (search for "ngettext_lazy"):

And the relevant Django code:

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.


--
Adam

אורי

unread,
Nov 23, 2019, 9:17:38 AM11/23/19
to Django developers (Contributions to Django itself)
Hi Adam / Django developers,

If you want I can try to spend some time to help solving this specific problem. I would start with documenting this issue in the release notes of Django 2.2. I already opened a new task #30945 for documenting this issue. How do I proceed from here? I think I never submitted changes directly to Django Git & documentation. Where are the source code of the release notes documentation and how do I change them? After documenting this problem, I can see if I can submit a change to the code itself. However I'm not sure if you will accept it, since what I want is mainly to maintain the functionality of these functions in Django up to 2.1, which means I'm not sure how new changes will work in Django 2.2. But, I think what we have right now is not an acceptable solution, because functionality of these two functions breaks when upgrading Django to 2.2, which means that these functions stop working in some cases (such as in Speedy Net & Hebrew) when upgrading Django to 2.2 - the output is like we didn't call the translating functions at all (the functions return the same string in English, not translated).

By the way, you said it's a complicated problem. I'm not sure about other languages, but at least in Hebrew, as far as I know, it was very simple in Django up to 2.1. Either n==1, or n!=1 - there were two strings and everything worked. I don't think it should be that complicated and actually I suggested that if you (the Django developers) want a more complicated functionality you should introduce new functions and not change the (working) functionality of existing functions, or at least change them in a backward-compatible way. And as far as I know, ngettext_lazy and ngettext worked without problems until Django 2.1 (at least in Hebrew, I'm not sure about other languages).

By the way, I read here that this bug it the fault of Transifex (not Django). I'm not sure what that means, it worked in Django 2.1. Someone made a decision to change something in Django 2.2, how can it be Transifex? It must be a decision of the Django developers. If Transifex has bugs, why not use a previous version which worked? As far as I would suggest, I would postpone using the 4-strings translation (or up to 6 strings in some languages) to Django 5.0 or 10.0. Is it really that important to break all the 2-strings translations which worked?


אורי

unread,
Nov 23, 2019, 4:26:26 PM11/23/19
to Django developers (Contributions to Django itself)
Hi Adam / Django developers,

I want to clarify if it was not clear - running makemessages after upgrading Django to 2.2 doesn't solve this problem. It doesn't. At least not in Hebrew. I didn't see any changes in the Django .po files after running makemessages in Django 2.2.

On Sat, Nov 23, 2019 at 2:29 PM Adam Johnson <m...@adamj.eu> wrote:

Tobias McNulty

unread,
Nov 23, 2019, 7:51:05 PM11/23/19
to django-developers
‪On Sat, Nov 23, 2019 at 9:17 AM ‫אורי‬‎ <u...@speedy.net> wrote:‬
By the way, I read here that this bug it the fault of Transifex (not Django). I'm not sure what that means, it worked in Django 2.1. Someone made a decision to change something in Django 2.2, how can it be Transifex? It must be a decision of the Django developers. If Transifex has bugs, why not use a previous version which worked? As far as I would suggest, I would postpone using the 4-strings translation (or up to 6 strings in some languages) to Django 5.0 or 10.0. Is it really that important to break all the 2-strings translations which worked?

As far as I understand, the issue is the translations that are pulled in from Transifex (it's not a version of Transifex that we can control). I would hazard a guess that the same issue would occur with Django 2.1 if the translations were updated; i.e., it's simply a matter of timing that it happened to break with Django 2.2.

There is some sample code in the ticket which might be good to try, to see if it fixes the issue for you: https://code.djangoproject.com/ticket/30439#comment:7

Cheers,

Tobias McNulty
Chief Executive Officer

tob...@caktusgroup.com
www.caktusgroup.com


אורי

unread,
Nov 23, 2019, 9:31:54 PM11/23/19
to Django developers (Contributions to Django itself)
Hi,

Please see my latest comment on this ticket:

I would like to suggest, since I understand the problem is per locale, and I'm only familiar with the Hebrew (he) locale: Can you revert https://github.com/django/django/blob/master/django/contrib/auth/locale/he/LC_MESSAGES/django.po in Django 2.2 to use the plurals as they were defined in Django 2.1 (https://github.com/django/django/blob/stable/2.1.x/django/contrib/auth/locale/he/LC_MESSAGES/django.po:  "Plural-Forms: nplurals=2; plural=(n != 1);\n")? I checked the file and there are only 2 cases of msgstr[2] and msgstr[3] there, which are both identical to msgstr[1]. Will it break things in Hebrew? Will it break things in other locales? If people using other locales will ask, you can do similar things there too (if things are broken there in a similar way). I understand in some languages things were not broken like this in Django 2.2 (or other versions), and they used more plural forms there even before, so you don't have to change anything there. And in Django 3.0 (or whichever version you choose) you can make the .po files independent so they will not depend on each other, and the first .po file loaded will not influence the rest of the files.

If you can revert such a change which was made, maybe even by mistake, in Django 2.2 and then fix the problem in Django 3.0 it may help not only Speedy Net but possibly anyone using Django with the Hebrew locale. You might not have made a deliberate decision to change Hebrew translation in Django 2.2, but it was changed from the change in the .po file from Transifex. But you (the Django developers) can make a deliberate decision to fix it. Or alternatively, if you fix the problem in Django 3.0 (or even 3.1 or 3.2) I might upgrade Speedy Net directly from 2.1 to 3.0 etc. without using 2.2 at all. But since 2.2 is an LTS version, I think you should make an effort to make it work, and therefore revert the plural-forms definition to the one used by Django 2.1. At least in Hebrew, but maybe in any language where it has changed.

You might not have noticed this issue before because there are not many websites using these languages with Django, however for these websites this is a serious bug. I'm waiting for any solution to this problem before I upgrade to any Django version beyond 2.1. I even thought about forking Django and reverting this definition myself, but I prefer if Django itself will do it then for me forking Django. If I work Django I might not be able to apply all the changes you make in future releases, including security issues which might arise.

Have you found any .po file of Django, in Hebrew, where the third and fourth strings are not identical to the second one? I'm currently not familiar with such a .po translation.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

אורי

unread,
Nov 23, 2019, 11:25:15 PM11/23/19
to Django developers (Contributions to Django itself)
Hi,

I was not aware that even Django have several .po files itself (for each locale). I found out https://github.com/django/django/blob/master/django/conf/locale/he/LC_MESSAGES/django.po , which has about 15 translations of 4 strings (plural forms), but even there it seems to me that strings #1,2,3 are always the same, so there are only 2 different strings (of plural forms) in each case.

I don't know whether there are other .po files in Hebrew where maybe not all the last 3 strings are the same.

I kept searching and found https://github.com/django/django/blob/master/django/contrib/humanize/locale/he/LC_MESSAGES/django.po with 40 such translations, with the same case as above, although in this file there are translations which can and should be different, such as the translation for "%d weeks" - strings #1,2,3 are identical ("%d שבועות") even though there is a specific word for "2 weeks" in Hebrew ("שבועיים"), which is better to use than the current translation. So I guess it's not too late to reverse this setting at least for Hebrew, at least until specific translations are made, maybe in Django 3.0 and above (or 3.1, 3.2) and then maybe there will be an optional difference between the definition of plural forms in each .po file, so it will not cause any damage to change it back to 4 strings or even more then, without breaking current functionality like what happens now.

I apologize for the speakers of all other languages except Hebrew for not understanding their language, and I also don't have time to look into any locale. But at least in Hebrew, it seems to me that there is no need in Django 2.2 for more than 2 plural forms (n==1, n!=1). And the Django automatically relying on what is defined by Transifex is a mistake.

Tobias McNulty

unread,
Nov 23, 2019, 11:37:48 PM11/23/19
to django-developers
Hi Uri,

As I understand it, the issue is around keeping this and all other PO files in the Django source updated and consistent in the future, since we rely on Transifex to provide them. Even if someone were to manually make the change you are suggesting, it would be overwritten the next time the files are updated from Transifex. I think this means the options are to (a) convince Transifex to revert the change or (b) help find a fix for ticket #30439.

If there is not a reason for the 3rd and 4th plural forms in Hebrew, perhaps Transifex would be willing to revert this change, I'm really not sure. But I would hazard a guess that they made the change for a reason, and that your energy would be best expended in helping to find a fix for #30439. This should also improve the behavior for *all* languages where the plural forms across PO files are not consistent.

I'm not sure if they are watching this thread, but it looks like Rodrigo and Michal (?) were/are working on a fix in that ticket, and there is a branch open on one of their forks. Perhaps they could chime in if they see this, or you could reach out to them in the ticket or on GitHub to discuss and find a solution.

Please remember that we are all volunteers, and rely on the work of fellow volunteers (like you!) to make Django the best that it can be. Thank you for following up about this issue, and good luck in tracking down a fix!

Best,
Tobias

אורי

unread,
Nov 23, 2019, 11:54:26 PM11/23/19
to Django developers (Contributions to Django itself)
Hi,

I tried to join the Django team on Transifex about 3 weeks ago (for Hebrew) but I failed. I think there may be a better translation using 4 plural forms in Hebrew or even more, however with the current situation of Django 2.2 translation to Hebrew it seems to me that the last 2 strings are never used (they are always identical to string #1). I'm not sure how Transifex works, since they didn't let me join the team, but if there is an option to reverse this setting in Django 2.2 and possibly 3.0, while keeping the option of more plural forms in future versions, after #30439 is fixed, this may be a good solution. But I'm not sure how to do it with Transifex, I don't know if they have branch names and versions like Django.

If #30439 is fixed in Django 3.0 or 3.1 (which I expect), I might have to upgrade to this version directly without using 2.2 at all. This is a shame since 2.2 should be LTS and I actually wanted to upgrade to 2.2 this month (from 1.11), and wait until the next LTS before upgrading Django again. But will a fix of  #30439 be applied to Django 2.2? If a fix is applied also to 2.2 it may solve this problem for me. Claude Paroz said it will not happen in Django 2.2 [https://code.djangoproject.com/ticket/30439#comment:26].

Matemática A3K

unread,
Nov 24, 2019, 6:40:48 PM11/24/19
to django-d...@googlegroups.com

,אורי

Follow the instructions here if you want to change the plural form for your django project "temporarily": https://code.djangoproject.com/ticket/30439#comment:17

Maintaining several catalogs unmerged can lead to inconsistencies also if you don't keep track of all of them, which can lead to burdens also, i.e. if you translate one string which is already contained in another po with a different plural form, that may be used instead of yours.

This is against - I believe, not sure, please someone correct me - the design of gettext on which Django builds upon. Although all .pos have a header with a plural formula, it is assumed that they will be the same for a language, so when merging, only the first one is retained.

Supporting multiple catalogs in order to support different plural forms is plausible, it seems to be like an extension to gettext, but it may not fix your problem entirely, because although you may retain the consistency of your pos, some translations will break (Django's) because of the broken formula.

One thing is the break by a broken formula, another thing is changing Django's one plural form policy.

(JIC, for those interested in the subject and want to catch up - besides reading the ticket -  a gettext catalog plural form contains 2 things: the number of plurals of the language and an equation/formula that decides which one to use for the case. Django uses one "general" gettext catalog per language which is the result of merging several ""subcatalogs"", ""local pos"")

The broken formulas introduced can be fixed by a fix in the repo, and/or by changing the syncing scripts with Transifex to do a msgmerge instead. This won't update the plural forms from Django, they will need to be manually updated on the repo - someone will have to check the updated formula and see if it is alright, and it has to be someone knowledgeable about it (I wouldn't be able to discern if the plural form of Slovakian is good or not, besides an evidently broken one from what I learnt), so I think this would rely on the django-i18n committee(?) (I guess this would be more work for them).

Another thing is the change of the number of plurals. This will not only imply a change in the plural formula but a change in the entire catalog, as new translations will be needed. This seems something not to be changed unless a mistake was found. If this is changed for convenience, in order to use gettext (and catalog merging), you should ensure that only your catalogs are used. You can give convenience by making your translation utility generate the remaining plurals accordingly so they can be merged with others that contain all the forms.

Here is a **shamefull, bad, ugly and dirty** script that will add the remaining entries to your .po in order to be merged with others that have the total of plurals. Just download it and run "update-po-plurals your.po from=2 to=4 > your_new.po" and it will be converted from two to four (although you will have to adjust manually the header). This way, it will be "merge-able" in every software that uses gettext with all the plurals (https://code.djangoproject.com/ticket/30439#comment:24).

The only fix that comes to my mind that will solve this is to give the ability of taking out the locales out from of the django distribution tree to django project tree. This is making a "collectlocale" command that will collect and merge all the .pos distributed in django and put them on settings.LOCALE_ROOT in the project tree. Once this exists, the Django translation module will start using this as the base for merging. You can edit them and VC them in your project. This way no update in the Django locales will affect your translations. You will have to manually run "collectlocale" on an upgrade to incorporate the changes. You will also take full benefit of catalog merging.  

For some reason, the Translation community can't come up with one definitive version of the number of plurals and a plural equation for each language (gettext gets them from http://cldr.unicode.org/) and different parts of the community use different formulas. If there were a consensus here, these problems wouldn't exist.

A provisional fix was already provided so you can restore your functionality by updating your project main po plural form, now you can update the plurals of your po from 2 to 4 that would provide the same translations results as before the increase of the plurals.

Do you have any doubts about how to apply the temporarily fixes?

If not, let's then focus on solving the problem the right way and definitively :)

אורי

unread,
Nov 25, 2019, 1:27:06 AM11/25/19
to Django developers (Contributions to Django itself)
Hi Matemática,

I prefer to keep using Django 2.1 until there is a solution that doesn't require so much effort from me when upgrading. I currently don't think we need 4 strings (plural forms) in our project's .po files and anyway I don't want to change them manually, only if they are changed automatically when I upgrade Django. My websites are currently working properly (as far as I know) with Django 2.1 and I want to keep it this way.

If a solution is made but applied to Django 3.0 or 3.1, I might decide to upgrade directly from 2.1 to 3.0 or 3.1, without using Django 2.2.

Matemática A3K

unread,
Nov 26, 2019, 1:13:32 AM11/26/19
to django-d...@googlegroups.com
‪On Mon, Nov 25, 2019 at 6:26 AM ‫אורי‬‎ <u...@speedy.net> wrote:‬
Hi Matemática,

I prefer to keep using Django 2.1 until there is a solution that doesn't require so much effort from me when upgrading. I currently don't think we need 4 strings (plural forms) in our project's .po files and anyway I don't want to change them manually, only if they are changed automatically when I upgrade Django. My websites are currently working properly (as far as I know) with Django 2.1 and I want to keep it this way.

If a solution is made but applied to Django 3.0 or 3.1, I might decide to upgrade directly from 2.1 to 3.0 or 3.1, without using Django 2.2.

אורי,

OK, have in mind that a change of the number of plurals for a language is kind (if not) of an API change for i18n - now you have to feed a different input - and should be handled as such. There is no way of modifying your code on an upgrade by the software distribution (the package).

Did the script that I posted not do the job?

 

אורי

unread,
Nov 26, 2019, 2:18:59 AM11/26/19
to Django developers (Contributions to Django itself)
On Tue, Nov 26, 2019 at 8:13 AM Matemática A3K <matemat...@gmail.com> wrote:


‪On Mon, Nov 25, 2019 at 6:26 AM ‫אורי‬‎ <u...@speedy.net> wrote:‬

אורי,

OK, have in mind that a change of the number of plurals for a language is kind (if not) of an API change for i18n - now you have to feed a different input - and should be handled as such. There is no way of modifying your code on an upgrade by the software distribution (the package).

Did the script that I posted not do the job?
 
No offense, but I didn't try. As a Django user I don't expect Django to send me to install and run third party scripts that will keep my sites working when I upgrade Django. I also don't think there is need for 4 plural forms in my .po files. I would like to keep using 2 plural forms as that makes more sense to me.

I decided to keep using Django 2.1.

Maciek Olko

unread,
Nov 26, 2019, 6:29:12 AM11/26/19
to django-d...@googlegroups.com
It looks like Transifex uses [1] Unicode Language Plural Rules [2]. If they are incorrect for Hebrew, maybe they should be fixed on Unicode side?

Regards,
Maciej


‪wt., 26 lis 2019 o 08:18 ‫אורי‬‎ <u...@speedy.net> napisał(a):‬
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

אורי

unread,
Nov 26, 2019, 7:55:57 AM11/26/19
to Django developers (Contributions to Django itself)
Hi Maciej,

I would not say the rules are incorrect. It depends on the case. In some cases they might be correct and there might be 3, 4 or even more plural forms. But in many other cases, two plural forms are enough. It depends on the strings being translated. I think in most cases, two plural forms are correct, like was in Django up to 2.1.

For example in weeks, there is a word in hebrew for "two weeks" ("שבועיים"), which is more correct than writing two words for two weeks ("2 שבועות"). But in other cases, there is not a specific word for "2 <objects>". In the case of Speedy Net for example, I used function ngettext_lazy in a validator validating the number of characters in a password or username. In these cases, only the plain plural form should be used, and anyway the number given there is probable not 1 and also not 2. It might be in some specific cases 6, 8 or 120, and there is no specific plural form for that number of characters. The translation is just using the Hebrew word for characters ("תווים"). Even if this number was 2, it would have used the same word, but there is a singular word for 1.

Maciek Olko

unread,
Nov 26, 2019, 8:29:18 AM11/26/19
to django-d...@googlegroups.com
Hi Uri,

Regarding your past questions,


If you want I can try to spend some time to help solving this specific problem. I would start with documenting this issue in the release notes of Django 2.2. I already opened a new task #30945 for documenting this issue. How do I proceed from here? I think I never submitted changes directly to Django Git & documentation. Where are the source code of the release notes documentation and how do I change them?

I think that following articles might be helpful:
* Writing documentation [1],
* Submitting patches [2].

Source file for 2.2 release notes is docs/releases/2.2.txt [3].

Regards,
Maciej


‪wt., 26 lis 2019 o 13:55 ‫אורי‬‎ <u...@speedy.net> napisał(a):‬

אורי

unread,
Nov 26, 2019, 9:19:57 AM11/26/19
to Django developers (Contributions to Django itself)
Thanks you, Maciej.

I read the links and came across https://media.djangoproject.com/foundation/icla.pdf, where I decided not so sign it yet now. Especially not for submitting documentation of Django 2.2 release notes. If you or any of the Django developers want to document this issue, please contact me in private and I can try to help. I decided for now not to submit a pull request directly myself since it's too complicated for me right now. I can live with this issue undocumented in the release notes and anyway it affects only specific languages where changes have been made in Django 2.2. Other languages might have been affected in other versions of Django, I'm not sure.

Shai Berger

unread,
Nov 26, 2019, 12:52:06 PM11/26/19
to django-d...@googlegroups.com
On Tue, 26 Nov 2019 12:28:45 +0100
Maciek Olko <macie...@gmail.com> wrote:

> It looks like Transifex uses [1] Unicode Language Plural Rules [2].
> If they are incorrect for Hebrew, maybe they should be fixed on
> Unicode side?
>

Just for the record, they are indeed wrong -- not in the sense that has
sparked this thread (there are indeed 4 forms) but in the rules (the
"many" rule should say "2 < n <= 10", not "n % 10 = 0"). I'm looking
into fixing it.

Matemática A3K

unread,
Nov 27, 2019, 1:41:05 AM11/27/19
to django-d...@googlegroups.com
Django's?

The problem raised from Django sync'ing with Transifex without any notice of plural rules changes. These changes led sometimes to break Django's translations, other times third-party translations and other times both.

For what I have understood, this is a script-driven process. This has to be changed if Django is going to maintain the plural rules "independently", because they will get overwritten in the next sync.

Maintaining the plural rules "independently" should be the best way to handle this - IMO - although the project may not have the resources for all the languages it provides translations. This would be sync'ing "only strings", not plural rules (forms and equations).

Changes in the number of plurals for a language should be included in the Release Notes, as it is changing the i18n API.

Changes in the plural equation doesn't seem necessary to me (to include in the RN) as they would be for the better - given the number of plurals is the same, the new equation should provide an improvement of the results with the same input - - well, mentioning them won't hurt anybody but it isn't necessary :)

If Django detects a catalog with a different number of plurals than its main, then it won't be merged and a warning should be shown (something like "Catalog not merged to prevent inconsistencies due to a different number of plurals than the main catalog. See https://docs.djangoproject.com/en/2.2/topics/i18n/translation/#pluralization"). The documentation should contain information on how to increase the number of plurals seamlessly of your catalog to match Django's (the most likely case).

If an inconsistency is seen due to a plural equation, it should be reported as a bug. As it is not sync'ed anymore, it can be fixed for Django. The documentation should contain information on how to fix it immediately for your project while the fix is done at the "Django level".

This is under the assumptions that:
- There is one plural equation which can describe entirely the language's plurals

Giving a convenient complete control of locales persistent to upgrades - a "collectlocales" command - can be left for later or a third-party package.

Do you think this a viable way to solve the issue?


--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

Matemática A3K

unread,
Nov 27, 2019, 2:08:38 AM11/27/19
to django-d...@googlegroups.com
Just for the records, that was not Django, that was me, helping you. No problem, as you don't want it, I will recycle it to not waste the time and energy and put it in another thing :)

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

אורי

unread,
Nov 27, 2019, 2:23:02 AM11/27/19
to Django developers (Contributions to Django itself)
On Wed, Nov 27, 2019 at 8:40 AM Matemática A3K <matemat...@gmail.com> wrote:

Changes in the number of plurals for a language should be included in the Release Notes, as it is changing the i18n API.

Thank you, I think it's very important to document such changes.

Please notice, I think changing the rules should also be documented, for example changing from "n % 10 = 0" to "2 < n <= 10". Since it would also affect the API and change the plural form used for specific numbers.

אורי

Matemática A3K

unread,
Nov 28, 2019, 12:46:26 AM11/28/19
to django-d...@googlegroups.com
On Wed, Nov 27, 2019 at 1:40 AM Matemática A3K <matemat...@gmail.com> wrote:


On Tue, Nov 26, 2019 at 12:51 PM Shai Berger <sh...@platonix.com> wrote:
On Tue, 26 Nov 2019 12:28:45 +0100
Maciek Olko <macie...@gmail.com> wrote:

> It looks like Transifex uses [1] Unicode Language Plural Rules [2].
> If they are incorrect for Hebrew, maybe they should be fixed on
> Unicode side?
>

Just for the record, they are indeed wrong -- not in the sense that has
sparked this thread (there are indeed 4 forms) but in the rules (the
"many" rule should say "2 < n <= 10", not "n % 10 = 0"). I'm looking
into fixing it.

Django's?

The problem raised from Django sync'ing with Transifex without any notice of plural rules changes. These changes led sometimes to break Django's translations, other times third-party translations and other times both.

For what I have understood, this is a script-driven process. This has to be changed if Django is going to maintain the plural rules "independently", because they will get overwritten in the next sync.

Maintaining the plural rules "independently" should be the best way to handle this - IMO - although the project may not have the resources for all the languages it provides translations. This would be sync'ing "only strings", not plural rules (forms and equations).

Changes in the number of plurals for a language should be included in the Release Notes, as it is changing the i18n API.

Changes in the plural equation doesn't seem necessary to me (to include in the RN) as they would be for the better - given the number of plurals is the same, the new equation should provide an improvement of the results with the same input - - well, mentioning them won't hurt anybody but it isn't necessary :)

If Django detects a catalog with a different number of plurals than its main, then it won't be merged and a warning should be shown (something like "Catalog not merged to prevent inconsistencies due to a different number of plurals than the main catalog. See https://docs.djangoproject.com/en/2.2/topics/i18n/translation/#pluralization"). The documentation should contain information on how to increase the number of plurals seamlessly of your catalog to match Django's (the most likely case).

If an inconsistency is seen due to a plural equation, it should be reported as a bug. As it is not sync'ed anymore, it can be fixed for Django. The documentation should contain information on how to fix it immediately for your project while the fix is done at the "Django level".

This is under the assumptions that:
- There is one plural equation which can describe entirely the language's plurals

Giving a convenient complete control of locales persistent to upgrades - a "collectlocales" command - can be left for later or a third-party package.

 
Do you think this a viable way to solve the issue?
 
Claude? Michal?

Do you still think that supporting multiple unmerged catalogs is the way to go? Can you elaborate more on this?

If not, and the plural-forms aren't maintained by Django, then the option of maintaining the whole locales in the project dir independently should be made available (collectlocales).

For maintaining plural forms, it can be done by notifying django-i18n mailing list of a new PF and if no one complains in a reasonable time frame, then it goes to the repository. This extra layer can prevent a lot of errors.

Matemática A3K

unread,
Dec 3, 2019, 9:41:51 PM12/3/19
to django-d...@googlegroups.com
OK, I made some progress on the quest of solving this issue :)

While I was implementing the "not-merge-catalog-if-plural-form-differs" policy, I realized that it would be cumbersome for the users to adapt to the new situation if no additional tools are provided. I switch to a Documentation Driven Development approach to have a better picture of all the needs, and here is what I wrote:

.. _plural-forms:

Plural Forms
~~~~~~~~~~~~

Django does not support custom plural forms in po files. As all translation
catalogs are merged, only the plural form in the main Django po file (located in
``django/conf/locale/<lang_code>/LC_MESSAGES/django.po``) is considered.

Plural forms in all other po files are ignored by the GNU gettext merging process.

Therefore, you shouldn't use different plural forms in your project or application
po files, as it may lead to unexpected results.

To prevent inconsistencies and undesired behaviors in translations, Django will
not merge any catalog that contains a different plural form than the main one
and will issue a warning about the conflicting catalog.

The warning will show both plural forms (the main and the conflicting one) so
you can decide which one is to be used.

This conflict may arise mostly in two situations:

* when the main plural form for a language is updated
in Django and your po files were generated with a previous one, or
* when including third-party translations with a different plural form.

There are two aspects of the plural form to be considered: the number of
plural forms (``nplurals``) and the plural equation (``plural=``).

For the number of plurals, Django follows the standards provided by `Unicode
<http://www.unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html>`_.

In the case of an increase in the number of plurals for a language, you will
have to provide with more translations in your po files to comply with the
updated language definition (it is considered an API change, noted in the
Release Notes and unusual).

For this, you may use
djadmin:`django-admin makemessages --comply-plurals<makemessages>` to fill
the missing plurals from your projects' catalogs with the previous ones so
you avoid having translation results in the fallback language or the original
string - this will produce the same results as before though you
may want to update those later for a better expression of the language if it
corresponds.

If you do not want to comply with the new language definition, you can create
a variant for the language with less plural forms, i.e. ``he_SP`` for "Hebrew
(Simplified)" with 2 plural forms and ``he`` with 4 variants acting as a
fallback (see :ref:`how-django-discovers-translations`). Note that your
variant would need to be explicitly set by Django, or in the url, or by your
front-end, as browsers won't know about it.

For the plural equation (the way of determining which form will be used),
given a number of plurals, *one will be an improvement over the other* - as
it is assumed that there is one plural equation that can describe correctly
all the plurals' cases of a language.

Note that the order of the forms can vary depending on the equation, i.e.
the cases for the second plural form in one equation may refer to the third
one in another.

Once you decided which one is better, there are two aspects to consider for
resolving this conflict: updating the plural forms of your project's catalogs
or update the ones in Django's base catalogs.

For aligning your project's catalogs' plural forms with the main one (most
common case after a Django upgrade), use
djadmin:`django-admin makemessages --update-plural-form<makemessages>`. This
will make your catalogs mergeable again with Django's.

After this, djadmin:`compile the messages as usual<compilemessages>` and
you'll be ready to go.

If you choose to use the version different than Django's, edit the main
Django po file and then run:

* djadmin:`django-admin makemessages --update-plural-form --base LC<makemessages>`
to propagate the change to all catalogs bundled with Django, and then
* djadmin:`django-admin compilemessages --base LC<compilemessages>` to compile
them.

This will align all the Django's catalogs with your plural equation. If so,
`consider submiting your plural equation as an improvement to Django in a ticket
<https://code.djangoproject.com/newticket>`_.

.. versionchanged:: 3.1

    Handling plural forms as described above was added.

It's not tooooo complicated, but not slick at all... not every user may like to take the time to understand the plural form of the catalogs, although I think for most cases, it will be issuing a "makemessages --update-plural-form".

This led me evaluate again the Claude proposal of having a dict with the plural form as keys and merge the catalogs there. This had the main problem of no clear resolution if overlapping, but then I realized that if an ordered dict is used, then the merging order can be somehow retained, and for most cases, it will be sufficient to reproduce the precedence order that Django has actually (project translations will come first than Django, like is described in here), so this would be a better way.

But, then I realized that there is major caveat on this approach, and that is that updates on the plural equation won't reach users' catalogs, because their catalogs will be kept separately one the plural form differs. This would be the case that Shai pointed on Django 2.2 with the incorrect plural equation for HE. People who have generated their catalogs with makemessages on 2.2 will have a wrong plural equation, that once it is fixed on a new release, it won't reach their catalogs because they will be kept apart. This would make things harder for that users (non-hardcore-translators? :), because tracking that error may be hard unless they previously understood all the process. Then, all the warnings should be shown also in this case, and so the tools to align their pfs to the new one, because updates would have to be done "manually" (by running the command).

So, the fix won't be in the merging policy (merge|not-merge|dict-merge) but in the warnings raised about different plural forms co-existing and the tools to align them easily (django -> user, user -> django) to get consistent results. The different merging policy would be a way of engaging the user to be more or less proactive with the problem ({"merge": "You may have unwanted results until you fix it", "not-merge": "it won't work until you fix it", "dict-merge": "you will have to take care of the updates for fixing it"}).

Also, it would be better to have a system check for this than doing it "on merge", because only the default language will be merged "on start", you will see the warnings later when it is activated.

Any thoughts?

PS: The "makemessages --comply-plurals" needs to assume that the new pfs were handled by the previous last one to produce the same results, if not, it will prevent only the fallback.

Claude Paroz

unread,
Dec 4, 2019, 2:25:32 AM12/4/19
to Django developers (Contributions to Django itself)
Le mercredi 4 décembre 2019 03:41:51 UTC+1, Matemática A3K a écrit :
(...)

But, then I realized that there is major caveat on this approach, and that is that updates on the plural equation won't reach users' catalogs, because their catalogs will be kept separately one the plural form differs. This would be the case that Shai pointed on Django 2.2 with the incorrect plural equation for HE. People who have generated their catalogs with makemessages on 2.2 will have a wrong plural equation, that once it is fixed on a new release, it won't reach their catalogs because they will be kept apart.

Sorry, I'm not following you here. The catalog merge process is happening in realtime when Django starts, so any po file update is instantly reflected in the translation infrastructure. I don't see any caveat here.

Claude

Matemática A3K

unread,
Dec 5, 2019, 12:10:46 AM12/5/19
to django-d...@googlegroups.com
Yes, but only for the default language, the rest are lazily loaded on demand - this is why I think it is better do the check at a system level, if you have more than a language available, the others will be merged for loading them once something triggers it (like an "activate("LANGUAGE_CODE")) and only there you would start to see the warnings.

For the example of the caveat of the dict-merging, it would be like this:

- Someone starts its translation with Django 2.2 for Hebrew via makemessages, which copies the main plural form to the new file, and fills the translations.
- The user upgrades to Django 2.2.x or up, which contains a fixed plural form for Hebrew
- Under the "dict-merge" policy, the Django translation catalog would have 2 entries, one for the main plural form (the new one) and other the catalog generated in the past (the user's one).

Strings in the user catalog would use a "worse" plural equation, while strings in the Django catalog will use the better one. Updates won't reach to users' catalogs, unless they explicitly update them, because once the user's po is created, makemessages won't update the header, it will do a msgmerge (https://github.com/django/django/blob/master/django/core/management/commands/makemessages.py#L603).

That's why something like "makemessages --update-plural-form" and a warning about different plurals forms in the catalog would be needed, even under "dict-merge".

With the current "merge" policy, because only the main form used, the users would get the update but they won't be able to have parts of the catalog under a different pf.

The way to have a different plural form than the main one with the current code base would be by having a custom variant of the language (which implements "the spirit of dict-merge": independent catalogs with an order of precedence). It's not ideal, you will loose the ability of using the browser's locale config to display the language out of the box unless you add extra code to handle it, something like "if locale == 'he': use 'he_SP'" (no other caveat comes to my mind). But it would be they way of having it RN without loosing the updates.
 

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

Matemática A3K

unread,
Dec 5, 2019, 2:00:43 AM12/5/19
to django-d...@googlegroups.com
While testing the "not-merging" policy, I got this:
Those warnings should get to the Translators teams if they are not looking here, i.e.
is still using a 2-plurals form in a 4 plural form - it doesn't matter in this case because there are no translations with plurals, but they trigger the warning because the pf hasn't been updated and they should if the Transifex front-end use the pf of a file for showing the options for translating and to be inline with the main form.

Maciek Olko

unread,
Dec 5, 2019, 4:23:57 PM12/5/19
to django-d...@googlegroups.com
Hi,
I am wondering if Django shouldn't use Unicode Plural Rules as standard and promote it for third-party apps. Even if sometimes number of forms are not applicable to certain cases, there may be cases when all of forms will be needed.

Especially if implementing having different plural rules for various apps is not trivial.

For the record, here is a section of Transifex documentation that describes their statement about plural rules and Unicode standard: https://docs.transifex.com/localization-tips-workflows/plurals-and-genders#how-pluralized-strings-are-handled-by-transifex .

Regards,
Maciej

Matemática A3K

unread,
Dec 6, 2019, 2:15:28 AM12/6/19
to django-d...@googlegroups.com
On Thu, Dec 5, 2019 at 4:23 PM Maciek Olko <macie...@gmail.com> wrote:
Hi,
I am wondering if Django shouldn't use Unicode Plural Rules as standard and promote it for third-party apps. Even if sometimes number of forms are not applicable to certain cases, there may be cases when all of forms will be needed.

I agree for Django, for third-party translations, for what I understood (CIIW) this may be a differentiator
 

Matemática A3K

unread,
Dec 7, 2019, 12:26:16 AM12/7/19
to django-d...@googlegroups.com
On Fri, Dec 6, 2019 at 2:14 AM Matemática A3K <matemat...@gmail.com> wrote:


On Thu, Dec 5, 2019 at 4:23 PM Maciek Olko <macie...@gmail.com> wrote:
Hi,
I am wondering if Django shouldn't use Unicode Plural Rules as standard and promote it for third-party apps. Even if sometimes number of forms are not applicable to certain cases, there may be cases when all of forms will be needed.

I agree for Django, for third-party translations, for what I understood (CIIW) this may be a differentiator
 

Especially if implementing having different plural rules for various apps is not trivial.

For the record, here is a section of Transifex documentation that describes their statement about plural rules and Unicode standard: https://docs.transifex.com/localization-tips-workflows/plurals-and-genders#how-pluralized-strings-are-handled-by-transifex .

Yes, there seems to be a source for errors there, because the Transifex front-end shows the entries according to the plural form of the file, maybe when it was built they were assumed to be individual catalogs not to be merged with others (there is no "project" or "group" constrain). If the plural form says 3 plurals, it will be considered as a valid submit - and it is according to the pf - but once it gets merged with a 4-plurals catalog, it will break if there is pluralization in it (as it has happened before).

A solution for this is to use the Unicode plural form for the language in each catalog on Transifex, it seems that this has to be set manually.

In the same direction, these seems broken:
and they use a main PF of 'nplurals=2; plural=(n!=1);' and 'nplurals=2; plural=(n > 1);',

There should be an agreement also of how to express the rules in the equations in order to be able to detect inconsistencies easier.

Note that as the contrib.sessions catalog does not contain any pluralized strings the error goes in-adverted.

If Django adopts the Unicode standard as official for its plural forms, then they shouldn't be changed unless the standard changes - and that should make it to the Release Notes.

That would be the first step for getting the Django catalog consistent and "PF-stable", making easier for third-parties to build upon them.

If there is an agreement, those have to been updated by the Django Translators Teams, because it seems that there is no enforcement or constrains in the Transifex front-end about PFs for a language, otherwise catalogs like Slovak, Lithuanian and Ukranian in contrib.sessions wouldn't have a non-Unicode PF (see Inconsistencies in Catalogs).

Matemática A3K

unread,
Dec 13, 2019, 1:04:56 AM12/13/19
to django-d...@googlegroups.com
On Fri, Dec 6, 2019 at 2:14 AM Matemática A3K <matemat...@gmail.com> wrote:


On Thu, Dec 5, 2019 at 4:23 PM Maciek Olko <macie...@gmail.com> wrote:
Hi,
I am wondering if Django shouldn't use Unicode Plural Rules as standard and promote it for third-party apps. Even if sometimes number of forms are not applicable to certain cases, there may be cases when all of forms will be needed.

I agree for Django, for third-party translations, for what I understood (CIIW) this may be a differentiator

Under this rationale, this would be the fix:

.. _plural-forms:

Plural Forms
~~~~~~~~~~~~

Django does not support multiple plural forms in catalogs. As all translation

catalogs are merged, only the plural form in the main Django po file (located in
``django/conf/locale/<lang_code>/LC_MESSAGES/django.po``) is considered.

Plural forms in all other po files are ignored by the GNU gettext merging process.

Therefore, you shouldn't use different plural forms in your project or application
po files, as it may lead to unexpected results.

To prevent inconsistencies and undesired behaviors in translations, Django will
not merge any catalog that contains a different plural form than the main one
and will issue a warning about the conflicting catalog.

This conflict may arise mostly in two situations:

* when the main plural form for a language is updated
in Django and your po files were generated with a previous one, or
* when including third-party translations with a different plural form.

for plural equations for each language, and encourages to align with it.

If you had generated your catalog with a previous version of Django, the
standard may have changed or a bug has been fixed.

For aligning with the new version of the standard (or addressing the bug), you
you may use
djadmin:`django-admin makemessages --comply-plural-form<makemessages>` to
reorganize your catalog so it is aligned with it.

The script will ask you to map each plural form in the new form to your
catalog's forms. After this, the messages in your catalog will be reorganized
accordingly. In the case of an increase in the number of plurals, it will fill
those with the previously chosen forms so you avoid having translation results

in the fallback language or the original string - this will produce the same
results as before though you may want to update those later for a
better expression of the language if it corresponds.

In the cases where you want to use a "bleeding-edge-yet-to-be-approved" plural
form, define the :setting:`LOCALE_ROOT` and use
djadmin:`django-admin makemessages --collect-base-catalogs<makemessages>`.
This will merge all the locale catalogs bundled with Django into your
:setting:`LOCALE_ROOT` and use them as your main Django po file.


.. versionchanged:: 3.1

    Handling plural forms as described above was added.

The problem that I see to updating plural forms in the package tree is that modifying the base package seems not be a good practice to me (independently of not being persistent across updates), you either copy, subclass, monkey patch, etc. So, I think this is a more proper solution.

Any thoughts?

Matemática A3K

unread,
Jan 6, 2020, 4:18:25 PM1/6/20
to django-d...@googlegroups.com
Well, seems that the Three Magic Kings have left us this

Matemática A3K

unread,
Jan 10, 2020, 8:38:46 PM1/10/20
to django-d...@googlegroups.com
Trying to recap all the discussion done in the mailing list, Trac and Github:

The problem that was originally reported in #30439 was about mixed plural forms in catalogs bundled with Django, which led to broken translations.

Then, it added the not announced changes in the plural forms of locales, which led to break users' translations.

These problems occur when catalogs gets updated in Django and users updates their version.

Michal proposed not merging and doing catalogs look-ups per language, Claude proposed the "dict-merge" policy (have a dict of merged catalogs according to their plural forms), which is a variant of Michal's.

The "dict-merge" policy have the problem of updates in the plural forms in Django won't reach users' catalogs.

Whether merge or not merge the catalogs (the merging policy) is about how to encourage the user to take action, not the fix itself.

The fix is on the warning of the situation and on the tools for addressing it.

The proposed fix is:
1- Warn the user at a system check level and at run-time
2- Use the "not-merge" policy to encourage action.
3- Provide the makemessages --comply-plural-forms tool so both Django and its users can have consistent plural forms in their catalogs conveniently.
4- Provide the LOCALE_ROOT setting and the makemessages --collect-base-catalogs tool so users can make changes in their plural forms conveniently and persisted across updates.

The PR is cluttered with a lot of inconsistencies fixes across all catalogs bundled with Django and the catalogs used for tests.

Once a warning is raised when merging (either merging or not), most of the tests will fail because of that warning, so all bundled catalogs (and those used for tests) have to be aligned in order to have things working.


For ensuring Django's catalogs consistency, there is a test that will fail if there is any catalog unaligned. The next time Django's catalogs will be updated, CI will check the consistency. If not, who will do the merge has to run "python -m django makemessages --comply-plural-forms" and "python -m django compilemessages" on the corresponding dir (django.conf, django.contrib.app or the test locale) to fix it. This way, the messages from the provider will be retained and the consistency be assured.

There are valid reasons for users to customize their plural forms, i.e. fixing a broken one while a fix is in the way or use another implementation of the standard. Having to modify the source distribution for customizing is not ideal, besides not being persistent across updates and may not be possible in some setups. This should be done locally, at the project tree. This is what the LOCALE_ROOT setting addresses.

Claude already objected this fix though he did not provide reasons (https://github.com/django/django/pull/12280#issuecomment-571483273).

Does anyone see any rationale, design or implementation problem in the fix? Any comment is welcomed :)

אורי

unread,
Jan 18, 2020, 7:54:17 AM1/18/20
to Django developers (Contributions to Django itself)
On Sat, Jan 11, 2020 at 3:38 AM Matemática A3K <matemat...@gmail.com> wrote:
Does anyone see any rationale, design or implementation problem in the fix? Any comment is welcomed :)


We use Django 2.1 in Speedy Net and this issue has prevented us from upgrading to Django 2.2. I'm trying to understand this PR but it seems to me quite complicated. Will we be able to keep using 2 plural forms in Hebrew for our translations if this PR is accepted? Or will we have to convert our translations to 4 plural forms like is used in Django 2.2? For me it makes sense to use 2 plural forms (n==1, n!=1) because we don't need any complicated translations. I checked Django translations and almost/maybe in 100% of the Hebrew translations, 3 plural forms are identical. Do we have to convert our translations to 4 plural forms too? And if we want to keep using 2 plural forms in Hebrew, how do we do it while still using the latest version of Django?

You can see my comments on #30439 and in the Django developers mailing list.

Uri
אורי
 

אורי

unread,
Jan 18, 2020, 10:33:17 PM1/18/20
to Django developers (Contributions to Django itself)

אורי

unread,
Jan 20, 2020, 11:57:17 PM1/20/20
to Django developers (Contributions to Django itself)
Hi,

Anyone wants to review PR #12332?

Movie Hunter

unread,
Jan 22, 2020, 10:05:45 AM1/22/20
to django-d...@googlegroups.com
Hello how can i help you

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

Matemática A3K

unread,
Jan 24, 2020, 2:17:37 AM1/24/20
to django-d...@googlegroups.com
On Fri, Jan 10, 2020 at 8:38 PM Matemática A3K <matemat...@gmail.com> wrote:
Trying to recap all the discussion done in the mailing list, Trac and Github:

The problem that was originally reported in #30439 was about mixed plural forms in catalogs bundled with Django, which led to broken translations.

Then, it added the not announced changes in the plural forms of locales, which led to break users' translations.

These problems occur when catalogs gets updated in Django and users updates their version.

Michal proposed not merging and doing catalogs look-ups per language, Claude proposed the "dict-merge" policy (have a dict of merged catalogs according to their plural forms), which is a variant of Michal's.

The "dict-merge" policy have the problem of updates in the plural forms in Django won't reach users' catalogs.

Whether merge or not merge the catalogs (the merging policy) is about how to encourage the user to take action, not the fix itself.

The fix is on the warning of the situation and on the tools for addressing it.

The proposed fix is:
1- Warn the user at a system check level and at run-time
2- Use the "not-merge" policy to encourage action.
3- Provide the makemessages --comply-plural-forms tool so both Django and its users can have consistent plural forms in their catalogs conveniently.
4- Provide the LOCALE_ROOT setting and the makemessages --collect-base-catalogs tool so users can make changes in their plural forms conveniently and persisted across updates.

The PR is cluttered with a lot of inconsistencies fixes across all catalogs bundled with Django and the catalogs used for tests.

Once a warning is raised when merging (either merging or not), most of the tests will fail because of that warning, so all bundled catalogs (and those used for tests) have to be aligned in order to have things working.


For ensuring Django's catalogs consistency, there is a test that will fail if there is any catalog unaligned. The next time Django's catalogs will be updated, CI will check the consistency. If not, who will do the merge has to run "python -m django makemessages --comply-plural-forms" and "python -m django compilemessages" on the corresponding dir (django.conf, django.contrib.app or the test locale) to fix it. This way, the messages from the provider will be retained and the consistency be assured.

There are valid reasons for users to customize their plural forms, i.e. fixing a broken one while a fix is in the way or use another implementation of the standard. Having to modify the source distribution for customizing is not ideal, besides not being persistent across updates and may not be possible in some setups. This should be done locally, at the project tree. This is what the LOCALE_ROOT setting addresses.

Claude already objected this fix though he did not provide reasons (https://github.com/django/django/pull/12280#issuecomment-571483273).

Does anyone see any rationale, design or implementation problem in the fix? Any comment is welcomed :)

To my surprise, yesterday when I started to work again in this issue - I thought we had agreed for a review after the code was separated in commits - I saw that the ticket was disowned from me.

So, I will leave my thoughts here.

I think the discussion hasn't been the best, both in constructiveness and fluidity terms, but as I had committed to get this fixed, I have continued working to get the best for it despite the differences.

I will write my concerns about the accepted fix ("dict-merge" policy).

- Users may be left in an unsatisfactory situation

Plural forms only get to the users' catalogs once it is created by makemessages. Once that a catalog is "filled", it is not safe to just update plural forms as it may require content modification.

Given that broken (buggy) plural forms have been distributed with Django, those catalogs will remain broken and unnoticed under the "dict-merge" policy.

The only rationale that I can think of for justifying this is "If they don't see anything broken and they are happy with it, it doesn't matter".

This is not acceptable for me, you should choose under the full understanding of the situation. If you are happy with it, you may choose the situation.
for it.

If there is another justification, it hasn't been answered the several times that I asked for it.

The broken plural forms are mostly broken equations that will never evaluate to that forms, making wasting effort. One thing is completing plural forms for adhering to an standard which will give a better expression in some cases - where it would be a cost of having the better expression for some situations given the design of the software - and another thing is have to do it in the future without any purpose, because a broken equation made its way to the catalog.

Under the "don't do to others what you don't like to be done to you" principle, this is not acceptable for me either.

If a decision is made, it should be stated the reasons for it. Otherwise, it may lessen the confidence on the project.

The right thing to do here to me would be adding at least a note saying something like "Buggy equations have been distributed in the past, you should check your catalog's plural forms with the current main plural forms to see if there is any improvement".

The first one was enough reason for not going on with this policy for me. Users won't get updates unless they look for them. Digging more into the policy, I also find the next issues.

- It may lead to broken translations in some projects

Django translation support is built upon the GNU gettext toolset. People who deal with translations have a workflow determined by it.

The expected gettext workflow is managing plural forms in a centralized manner under the assumption that only the main plural forms will be considered.

For example,
have no pluralization enabled while other catalogs do.

Managing plural forms in a "de-centralized" way needs to ensure every catalog is "right", "self-contained".

Changing will lead to a revision on the plural forms of all the catalogs involved, in order to check that this situation does not affects a catalog (check at least that every catalog has consistent plural forms for the catalog content).

The change on this assumption may lead to broken translations in some projects if it goes unnoticed.

Therefore, it should be stated in the docs.

- The order of precedence is not warrantied

This was my main concern with the original Michal proposal, then I thought it could be fixed by using something like an ordered dict. Digging deeper, it produces the same results in some situations but not in every.

The first key would be the Django main plural forms, then all the catalogs with different plural forms will be merged in the corresponding key before the first key and taking precedence over the main one.

If all the user's catalogs have different plural forms than the main ones, they will take precedence over it (as expected), but if there is one at the highest order of precedence but with the main plural forms, it will be merged in the first key, taking precedence over Django's (expected) but not over the rest (unexpected).

Therefore, a note saying something like "Using catalogs with different plural forms may lead to unexpected order of precedences" if this behavior is deemed appropriate.

- -

Although this policy will bring plural forms customization out-of-the-box (i.e. for convenience when including third-party messages files or using your plural forms), it will have the undesired results previously described.

The same goals can be achieved without those "side effects" by the path I was working on.

The main concerns raised about was it length and changing they way Django handles translation files fundamentally.

Indeed is lengthy, but mostly because of the tools, not because of the changes to the existing Django code. Now that is has been separated into commits, it can be seen more clearly:

The plural forms consistency mode is just run-time warning raised at merge and collected at a system-check for the available languages enabled by a setting.

I don't find it too long (besides the code for plural form parsing, taken from the tool), and I don't find it changes the way Django handle translation files.

Even under the "no-merge" policy - which I proposed at the begin - it didn't, it just added a "stop" to encourage user action (which nobody objected previously). Files were handled the same way once the assumption was confirmed.

I added the setting disabled by default because of Tim's comment on another thread (~~"raising a warning about a documented behavior may be rude to developers who knew about it and chose to use it").

This mode will also act as the "Release Notes for plural forms", once Django updates them, it will trigger the warning to make you check (changes of the number of plurals may lead to broken translations, changes in the equation should be bugfixes).

The LOCALE_ROOT doesn't change the way Django handle files, it only allows to customize where the merging process starts. As if you set it, you set it locally in your project, you will be able to customize plural forms.

The tools are lengthy, indeed, but not complicated in my biased opinion.

--collect-bundled runs xgetext over the source files and msgcat with the bundled catalogs to deliver into LOCALE_ROOT:

--update-plural-forms allows to automate plurals expansions, trimming and reordering [*] :

[*] last Michal comment kept me thinking and I think another iteration is needed.

The tools may not be included also.

That was my rationale :)

Matemática A3K

unread,
Jan 28, 2020, 12:42:29 AM1/28/20
to django-d...@googlegroups.com
This can be fixed by checking for each msgid in the catalog to be merged if it exists on the already merged catalogs with higher precedence and remove it

אורי

unread,
Mar 4, 2020, 8:20:10 AM3/4/20
to Django developers (Contributions to Django itself)
Django developers,

PR #12332 is waiting to be reviewed and approved.

I would also like to suggest to include this PR, if accepted, into the next version of 2.2. Otherwise I will not be able to use 2.2 or I will have to fork Django to use 2.2, and then I will have to apply every patch manually to my fork, which is a harass and I don't think should be necessary.

Carlton Gibson

unread,
Mar 4, 2020, 8:42:52 AM3/4/20
to Django developers (Contributions to Django itself)
HI Uri.

Yes, we know. :) Please don't bump. It's just adds noise. (If you MUST bump, a comment on the PR is more than enough, so not everyone of the thousands on this list needs a notification.) Thanks.

Ref backport: it's a possibility but you should phrase it more like, "this is serious bug for any user of i18n supporting non-latin languages"[*] — or similar. (That one user has a bug wouldn't qualify it.) — It is under consideration.


On Wednesday, 4 March 2020 14:20:10 UTC+1, Uri wrote:
Django developers,

PR #12332 is waiting to be reviewed and approved.

I would also like to suggest to include this PR, if accepted, into the next version of 2.2. Otherwise I will not be able to use 2.2 or I will have to fork Django to use 2.2, and then I will have to apply every patch manually to my fork, which is a harass and I don't think should be necessary.

On Tue, Jan 28, 2020 at 7:42 AM Matemática A3K <matema...@gmail.com> wrote:


On Fri, Jan 24, 2020 at 2:17 AM Matemática A3K <matema...@gmail.com> wrote:

On Mon, Jan 6, 2020 at 4:17 PM Matemática A3K <matema...@gmail.com> wrote:


On Fri, Dec 13, 2019 at 1:04 AM Matemática A3K <matema...@gmail.com> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

אורי

unread,
Mar 4, 2020, 12:14:31 PM3/4/20
to Django developers (Contributions to Django itself)
Hi Carlton,

I think this issue is important for the Django developers mailing list. I already commented on the PR about 3 weeks ago. Yes I know that most Django users use either English, or a language which its plural forms were not changed recently. But users who use a language whose plural forms changed, for example Hebrew in Django 2.2, have problems upgrading Django to 2.2 without serious consequences. I don't know how many users like this there are, except the users who commented on this ticket, which are only about 4 or 5 users as far as I remember. But think about it this way - if the language affected was English, would you act differently? The ticket is from 10 months ago (May 2019), since then there is a problem with Django 2.2, and I noticed you released recently a new Django version. I think you (the Django developers) should take this issue seriously and not like "That one user has a bug".

Carlton Gibson

unread,
Mar 4, 2020, 12:23:07 PM3/4/20
to django-d...@googlegroups.com
We are taking it seriously Uri. It’s difficult to resolve instantly when there’s a super long mailing list thread, and several PRs in play, and a major discussion on the ticket. It’s only recently that the correct approach has become clear. It’s taken a community effort, from i18n experts to get there.  

Before that there was pressure to quickly merge incorrect solutions. 

Then we have security issues to resolve, such as two SQL injection issues in the last couple of months. These take priority.  

Sorry if things go slower than you’d like. There’s more to it than perhaps there seems. (Not least the danger of introducing regressions, especially if we do decide to backport, as you’d like.)


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABD5YeGCfRT07uRpM3phmQFpNJz%2BFouKt1N4Ty%3D9tC0-Uz-Bjw%40mail.gmail.com.

אורי

unread,
Mar 4, 2020, 12:35:32 PM3/4/20
to Django developers (Contributions to Django itself)
Yes, I understand. Don't merge incorrect solutions. But notice that it is a regression, in the affected languages. Things just stop working after upgrading Django. If I didn't have the unit tests which failed, I might have upgraded Django to production and my users would have received messages in an incorrect language. There is not even an exception raised when the messages are not translated. It just happens that the language I'm using is not the most popular language Django users are using, but it is a regression. It's also not documented in the release notes, as I would expect it to be documented there.


Carlton Gibson

unread,
Mar 4, 2020, 12:42:26 PM3/4/20
to Django developers (Contributions to Django itself)
Right, but now we come right back to the beginning: it's not a regression, and not judged one by Django i18n team, who are as knowledgeable as anyone about these things. I see no value in your continuing to press this point. 

If it had have been it would have been triaged as a Release Blocker. 

As it is, on strictest terms it probably doesn't qualify for a backport, but given that it does affect all users of i18n for non-latin languages we're considering backporting it the "severity" clause. 

FWIW I'm in favour of the backport, and have argued for that in discussions. 

We will get back to the PR again as soon as we can, and will resolve it as soon as we can. More than that I cannot say. 

I appreciate that's not as quick as you want. You're welcome to use a patched version in the meantime if your schedule demands. (Adam posting instructions on how to do that in another thread just today.) 

I'm going to unsubscribe from this thread now.


On Wednesday, 4 March 2020 18:35:32 UTC+1, Uri wrote:
Yes, I understand. Don't merge incorrect solutions. But notice that it is a regression, in the affected languages. Things just stop working after upgrading Django. If I didn't have the unit tests which failed, I might have upgraded Django to production and my users would have received messages in an incorrect language. There is not even an exception raised when the messages are not translated. It just happens that the language I'm using is not the most popular language Django users are using, but it is a regression. It's also not documented in the release notes, as I would expect it to be documented there.


On Wed, Mar 4, 2020 at 7:22 PM Carlton Gibson <carlto...@gmail.com> wrote:
We are taking it seriously Uri. It’s difficult to resolve instantly when there’s a super long mailing list thread, and several PRs in play, and a major discussion on the ticket. It’s only recently that the correct approach has become clear. It’s taken a community effort, from i18n experts to get there.  

Before that there was pressure to quickly merge incorrect solutions. 

Then we have security issues to resolve, such as two SQL injection issues in the last couple of months. These take priority.  

Sorry if things go slower than you’d like. There’s more to it than perhaps there seems. (Not least the danger of introducing regressions, especially if we do decide to backport, as you’d like.)

On Wed, 4 Mar 2020 at 18:14, אורי <u...@speedy.net> wrote:
Hi Carlton,

I think this issue is important for the Django developers mailing list. I already commented on the PR about 3 weeks ago. Yes I know that most Django users use either English, or a language which its plural forms were not changed recently. But users who use a language whose plural forms changed, for example Hebrew in Django 2.2, have problems upgrading Django to 2.2 without serious consequences. I don't know how many users like this there are, except the users who commented on this ticket, which are only about 4 or 5 users as far as I remember. But think about it this way - if the language affected was English, would you act differently? The ticket is from 10 months ago (May 2019), since then there is a problem with Django 2.2, and I noticed you released recently a new Django version. I think you (the Django developers) should take this issue seriously and not like "That one user has a bug".

On Wed, Mar 4, 2020 at 3:43 PM Carlton Gibson <carlto...@gmail.com> wrote:
HI Uri.

Yes, we know. :) Please don't bump. It's just adds noise. (If you MUST bump, a comment on the PR is more than enough, so not everyone of the thousands on this list needs a notification.) Thanks.

Ref backport: it's a possibility but you should phrase it more like, "this is serious bug for any user of i18n supporting non-latin languages"[*] — or similar. (That one user has a bug wouldn't qualify it.) — It is under consideration.


On Wednesday, 4 March 2020 14:20:10 UTC+1, Uri wrote:
Django developers,

PR #12332 is waiting to be reviewed and approved.

I would also like to suggest to include this PR, if accepted, into the next version of 2.2. Otherwise I will not be able to use 2.2 or I will have to fork Django to use 2.2, and then I will have to apply every patch manually to my fork, which is a harass and I don't think should be necessary.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

אורי

unread,
Mar 4, 2020, 12:43:09 PM3/4/20
to Django developers (Contributions to Django itself)
By the way, I also had a regression when upgrading to Django 2.1, which I was not aware of until after about 4 or 5 weeks after I upgraded. Something changed with the cookie settings (the default Django setting) and then login/logout didn't work across 2 of my websites, until I changed cookie settings (I think it was SESSION_COOKIE_SAMESITE that changed). It is documented on the release notes but I didn't notice it, no unit tests failed and I introduced a bug to production. It took me about 4 or 5 weeks to fix this.


Reply all
Reply to author
Forward
0 new messages