I18N and locale identification

[utelle]

Recently I had a long discussion with a developer on the wxWidgets forum, who was struggling with identifying the current UI language under Windows (see Localization with current display language). He tweaked his system settings for testing purposes (for example, setting system language to "German (Belgium)" (de-BE) or "German (Italy)" (de-IT). However, the function wxLocale::GetSystemLanguage returned wxLANGUAGE_UNKNOWN in these cases.

IMHO the reason for this behaviour is twofold:

1. the list of languages (as given in misc/languages/langtabl.txt contains an entry for "de-BE", but not for "de-IT" (which is a recent addition to the list of known locales in Windows)
2. Even though that "de-BE" is included in the list the assigned numeric LCID is 0, so that it is not identified in search through the list.

According to the Microsoft document MS-LCID numeric LCIDs are deprecated and the string IDs should be preferred. I guess the new class wxUILocale in the upcoming wxWidgets 3.1.6 serves exactly this purpose. However, the code for identifying the default language still uses the numeric IDs.

This leads to some questions:

1. Is it planned to adjust this code accordingly?
2. Shouldn't the list of languages in wxWidgets (misc/languages/langtabl.txt) be updated? It currently contains 236 entries, while appendix A in the MS-LCID document contains over 800 entries.
3. Wouldn't it be better to generate the list of languages during runtime? At least on Windows it is possible to enumerate all locales/languages.

Regards,

Ulrich

[Vadim Zeitlin]

Hi Ulrich,

I'm glad that someone (else) is at least looking at wxUILocale, although
it would be even better if you (or that other developer) could try to
actually use it.

On Fri, 29 Oct 2021 00:15:01 -0700 (PDT) you wrote:

u> According to the Microsoft document MS-LCID
u> <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/70feba9f-294e-491e-b6eb-56532684c37f>
u> numeric LCIDs are deprecated and the string IDs should be preferred. I
u> guess the new class wxUILocale in the upcoming wxWidgets 3.1.6 serves
u> exactly this purpose.

Let's say that it's supposed to provide API that works in this kind of
situations too.

u> However, the code for identifying the default
u> language still uses the numeric IDs.

Is the goal really to identify the default language? Usually you "just"
want things to work correctly, i.e. use the conventions appropriate for the
current locale. And wxUILocale should work correctly even for the locales
not supported by wxWidgets, such as "de-IT".

u> This leads to some questions:
u>
u> 1. Is it planned to adjust this code accordingly?

This depends on what exactly the problem is, i.e. how critical is it, how
difficult fixing it is and how long will the rest of high resolution
bitmaps support take me to finish...

u> 2. Shouldn't the list of languages in wxWidgets
u> (misc/languages/langtabl.txt) be updated? It currently contains 236
u> entries, while appendix A in the MS-LCID document contains over 800 entries.

No, I don't think we want to maintain this table. I mean, I won't stand in
the way of anybody who wants to do it, but I don't think it's a good idea
for us to maintain this list, which will invariably get out of data again
soon anyhow.

u> 3. Wouldn't it be better to generate the list of languages during
u> runtime? At least on Windows it is possible to enumerate all
u> locales/languages.

Yes, getting this information from the system is a more promising
approach, IMO. But, again, this depends on what exactly do we need to do.
AFAIK wxUILocale works for what it currently does and while I'm fully
cognizant of the fact that it could do much more, I'm not sure what exactly
is missing in it from your (or that developer's) point of view?

So could you please explain what is the actual goal here?

TIA,
VZ

[Eric Jensen]

Hello Vadim,

Friday, October 29, 2021, 5:06:27 PM, you wrote:

VZ> Hi Ulrich,

VZ> I'm glad that someone (else) is at least looking at wxUILocale, although
VZ> it would be even better if you (or that other developer) could try to
VZ> actually use it.

In order to use it they first must know about it. Ulrich's post was
the first time i ever heard of that class. Although afterwards i
faintly remembered a discussion about it here from a few months ago.

I think it should at least be mentioned in changes.txt. Or is that
only updated before a new official release?

Eric

[Vadim Zeitlin]

On Fri, 29 Oct 2021 18:12:13 +0200 Eric Jensen wrote:

EJ> In order to use it they first must know about it. Ulrich's post was
EJ> the first time i ever heard of that class. Although afterwards i
EJ> faintly remembered a discussion about it here from a few months ago.

I think there were several threads about this while I was working on it.
(although, granted, not as many as about wxBitmapBundle). I did mean to
write a blog post about it, and even started on it, but so far I only have
an almost empty _blog/2021-08-27-localized-applications-and-macos-11.md
file in my working copy.

I don't know what would be the way to let people know about new additions
to wx. Most of them are too small to merit a blog post, but some of them
are too large to be explained by just a single line in the change log. How
could we let people know about both kinds of changes? Other projects have
regular "This Week in Whatever" blog posts and while we're not big/active
enough for this, it could be nice to have "This Month in wx" summaries. The
problem is that somebody would have to do it...

EJ> I think it should at least be mentioned in changes.txt. Or is that
EJ> only updated before a new official release?

Yes, it is, I only add git notes until then.

Regards,
VZ

[utelle]

VZ schrieb am Freitag, 29. Oktober 2021 um 18:21:33 UTC+2:

On Fri, 29 Oct 2021 18:12:13 +0200 Eric Jensen wrote:

EJ> In order to use it they first must know about it. Ulrich's post was
EJ> the first time i ever heard of that class. Although afterwards i
EJ> faintly remembered a discussion about it here from a few months ago.

I think there were several threads about this while I was working on it.

I have to admit that I completely missed the discussions in April and August (which I detected only after reading your above remark). Usually I follow the discussions on wx-dev regularly. I'm really sorry, because I18N is a topic that was (and is) of great interest for me all the time.

 

I don't know what would be the way to let people know about new additions
to wx. Most of them are too small to merit a blog post, but some of them
are too large to be explained by just a single line in the change log. How
could we let people know about both kinds of changes? Other projects have
regular "This Week in Whatever" blog posts and while we're not big/active
enough for this, it could be nice to have "This Month in wx" summaries. The
problem is that somebody would have to do it...

IMHO the wx-dev mailing list is not the best place to inform casual readers (like me) about planned additions or current major issues. Due to the fact that the wxTrac messages are also directed to that list other topics easily drop out of sight.

Maybe a separate page on the wxWidgets website could hold a short list of major issues that are currently discussed and/or worked on (with a very short abstract and maybe with pointers to discussion threads and/or wxTrac entries). Entries should be made if discussions about the design of solutions are likely to be helpful/useful/necessary..., but not for minor and/or trivial additions. The effort to maintain the list should be kept low - that is, no long blog posts and/or summaries.

Regards,

Ulrich

[Ian McInerney]

While I know the wx-users group is mainly people asking for help, posting announcements about large changes to that list would probably help get more attention.

-Ian

--

Ian McInerney
mcia...@gmail.com

No electrons were harmed in the making of this message

[Eric Jensen]

Hello Vadim,

Friday, October 29, 2021, 6:21:31 PM, you wrote:

VZ> On Fri, 29 Oct 2021 18:12:13 +0200 Eric Jensen wrote:

EJ>> In order to use it they first must know about it. Ulrich's post was
EJ>> the first time i ever heard of that class. Although afterwards i
EJ>> faintly remembered a discussion about it here from a few months ago.

VZ> I don't know what would be the way to let people know about new additions
VZ> to wx. Most of them are too small to merit a blog post, but some of them
VZ> are too large to be explained by just a single line in the change log.

For me just a short summary of new public classes (and maybe useful new methods
of existing classes) would be enough. I just need to know they exist,
then i can read the documentation if i want to know more.

Regards,
Eric



--

[utelle]

Hi Vadim,

VZ schrieb am Freitag, 29. Oktober 2021 um 17:06:31 UTC+2:

I'm glad that someone (else) is at least looking at wxUILocale, although
it would be even better if you (or that other developer) could try to
actually use it.

It was sheer coincidence that I detected the existence of this new class wxUILocale. However, now I know of it, I intend to take a closer look and to use it. However, I first have to better understand the relation between wxLocale and wxUILocale. Currently, wxLocale is used to load translation message catalogs for a given locale, and if I understood it correctly, wxUILocale does not handle translations, but should possibly replace wxLocale in the long run. So, how should an application load translation message catalogs in the future?

 

On Fri, 29 Oct 2021 00:15:01 -0700 (PDT) you wrote:

u> According to the Microsoft document MS-LCID
u> <https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/70feba9f-294e-491e-b6eb-56532684c37f>
u> numeric LCIDs are deprecated and the string IDs should be preferred. I
u> guess the new class wxUILocale in the upcoming wxWidgets 3.1.6 serves
u> exactly this purpose.

Let's say that it's supposed to provide API that works in this kind of
situations too.

The problem on Windows is that for many of the new allowed locales there are no reasonable numeric IDs defined. IMHO, working with string identifiers conforming to BCP47 syntax is much more convenient anyway - at least from the user's perspective.

 

u> However, the code for identifying the default
u> language still uses the numeric IDs.

Is the goal really to identify the default language? Usually you "just"
want things to work correctly, i.e. use the conventions appropriate for the
current locale. And wxUILocale should work correctly even for the locales
not supported by wxWidgets, such as "de-IT".

If an I18N application is started for the first time, it is nice if it can detect the actual default system locale, so that the default UI language can be used to load the translation message catalog (at least if the default language is supported by the application - which might not be the case for exotic UI languages).

I have not yet tested wxUILocale, but I have no doubt that it will load the correct settings even for locales not in the language/locale list of wxWidgets.

 

u> This leads to some questions:
u>
u> 1. Is it planned to adjust this code accordingly?

This depends on what exactly the problem is, i.e. how critical is it, how
difficult fixing it is and how long will the rest of high resolution
bitmaps support take me to finish...

Quite often it is useful for an I18N application to deduce the language, the country, and sometimes the currency from the default system locale, so that the application can start with default settings that meet the expectations of the user(s). At least on Windows this doesn't seem to always work properly for less common system locale settings (like "de-IT") according to the other developer. (For my own applications I didn't get complaints from users up to now, because the applications simply start in English, if the system UI language can't be detected, and let the user choose the UI language from the list of languages supported by the application.)

However, I think it would be useful to get rid of the mechanism of identifying locales/languages by numeric IDs.

I'm willing to work on this topic, but I don't want to interfere with already ongoing work, or to work on classes like wxLocale which might be deprecated in the near future.

 

u> 2. Shouldn't the list of languages in wxWidgets
u> (misc/languages/langtabl.txt) be updated? It currently contains 236
u> entries, while appendix A in the MS-LCID document contains over 800 entries.

No, I don't think we want to maintain this table. I mean, I won't stand in
the way of anybody who wants to do it, but I don't think it's a good idea
for us to maintain this list, which will invariably get out of data again
soon anyhow.

Maintaining the list manually is certainly not an option. However, the process could be automated at least to a certain degree by extracting the data from official data sources like ISO 639, ISO 3166, and the Unicode CLDR project (CLDR = Common Locale Data Repository).

For my own applications I extracted translations for languages, countries, and currencies from the Unicode CLDR and generated .po message catalog files for the languages supported by the application, so that list of languages, countries, or currencies can be displayed in the UI language selected by the user.

If there is interest I could extend the process to generate such translation message catalog files for languages, countries, currencies for all languages for which currently wxstd message catalog files exist.

 

u> 3. Wouldn't it be better to generate the list of languages during
u> runtime? At least on Windows it is possible to enumerate all
u> locales/languages.

Yes, getting this information from the system is a more promising
approach, IMO. But, again, this depends on what exactly do we need to do.
AFAIK wxUILocale works for what it currently does and while I'm fully
cognizant of the fact that it could do much more, I'm not sure what exactly
is missing in it from your (or that developer's) point of view?

A list of locales is useful for 2 purposes:

1. In detecting the default UI language like in wxLocale::GetSystemLanguage
2. For displaying lists of languages/locales actually supported by an application (for example, the application scans the message catalog directory for available translations and uses the string identifiers (like "de_DE") to determine the English names of language and country resp locale to be displayed in a selection list for the user, possibly translating the English names to the currently selected UI language.

At least for Windows I could provide code to determine the list of locales. The question is where this code should be located - in wxLocale (where the list is currently generated from data in languageinfo.cpp), or in wxUILocale, or elsewhere?

I don't know whether Linux and macOS provide methods to retrieve a list of known locales. If not, one could create a small tool to retrieve the list from Windows and generate the file langtabl.txt from these data.

BTW, on Windows it is possible to retrieve the native display name for a locale. This can be useful for a language selection list.

 

So could you please explain what is the actual goal here?

I hope my above remarks clarify this a bit.

Regards,

Ulrich

[Andriy Byelikov]

VZ>  I don't know what would be the way to let people know about new additions
VZ> to wx. Most of them are too small to merit a blog post, but some of them
VZ> are too large to be explained by just a single line in the change log.

How about making a new GitHub Project for this? You already have the 'High
resolutions bitmaps support on high DPI screens' project there. You could
add a more general backlog of the new additions with a high level desciption
along with the main developer(s) working on it. Maybe organize each column
by version milestone, kind of like Trac does but without mixing defect
tickets in there. Plus it's easy to maintain since it's all cards and not a
single monolithic document and it helps you get organized and the users can
see it. I'm not sure if GitHub lets you have milestones or tag people there
but if not just write it down as plain text. All that would be left to do is
to leave a link to this project for the users on the README or wherever.

--
You received this message because you are subscribed to the Google Groups "wx-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wx-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wx-dev/1684164552.20211029215642%40j-dev.de.

[Vadim Zeitlin]

On Fri, 29 Oct 2021 12:33:47 -0700 (PDT) utelle wrote:

u> Maybe a separate page on the wxWidgets website could hold a short list of
u> major issues that are currently discussed and/or worked on (with a very
u> short abstract and maybe with pointers to discussion threads and/or wxTrac
u> entries). Entries should be made if discussions about the design of
u> solutions are likely to be helpful/useful/necessary..., but not for minor
u> and/or trivial additions. The effort to maintain the list should be kept
u> low - that is, no long blog posts and/or summaries.

On Fri, 29 Oct 2021 20:39:05 +0100 Ian McInerney wrote:

IM> While I know the wx-users group is mainly people asking for help, posting
IM> announcements about large changes to that list would probably help get more
IM> attention.

On Fri, 29 Oct 2021 21:56:42 +0200 Eric Jensen wrote:

EJ> For me just a short summary of new public classes (and maybe useful new methods
EJ> of existing classes) would be enough.

On Sat, 30 Oct 2021 00:05:21 +0200 Andriy Byelikov wrote:

AB> How about making a new GitHub Project for this?


Thanks for all the replies!

I don't know which one of those would be best, but we have to try
something, as current situation is clearly unsatisfactory. Posting to
wx-users is simple enough, but I'm not sure it's really going to help that
much and there won't be a place to link to permanently later (we could link
to the individual posts in the list archives, of course, but there won't be
a single, unique link).

The separate page on the web site idea is somewhat redundant with the
existing blog and news pages there and I think that perhaps the simplest
solution would be to just be more conscientious and post more news there.
And, in fact, I've just create https://github.com/wxWidgets/website/pull/63
which I think would be worth posting -- please let me know if you have any
comments about it.

The idea of a new GitHub project is intriguing, but I'm not really sure
our planning/coordination is good enough for this to work. A lot of things
just get done without having been really planned for, and the high DPI is
probably an exception rather than the rule. I'm also not sure if people are
really going to go to GitHub and git into the "Done" column of some project
to find out what the changes were. This could be useful to provide more
visibility on the development for people who want to know when the new
release will be made (don't we all), but this is a bit different from
disseminating knowledge about the things already done and I don't think it
would work well for this.

To summarize, I guess the most realistic thing to do is to post news items
to the web site more often and post to wx-users to let people know about
the new posts (as I don't think that many wx users go to the web site
regularly).

Please let me know if you have any suggestions about how this could be
improved further and, also, please don't hesitate to remind me to make the
next post in a few months -- or maybe even make a PR like the one above
yourself, if you feel motivated.

Thanks in advance,
VZ

[Vadim Zeitlin]

On Fri, 29 Oct 2021 14:13:53 -0700 (PDT) utelle wrote:

u> It was sheer coincidence that I detected the existence of this new class
u> wxUILocale. However, now I know of it, I intend to take a closer look and
u> to use it. However, I first have to better understand the relation between
u> wxLocale and wxUILocale.

wxLocale is not formally deprecated because it still works as well (or as
badly) as always in wxMSW and wxGTK but shouldn't be used in the new code
because it simply can't be made to work in wxOSX. So wxLocale is legacy,
new code should only use wxUILocale.

u> Currently, wxLocale is used to load translation
u> message catalogs for a given locale, and if I understood it correctly,
u> wxUILocale does not handle translations, but should possibly replace
u> wxLocale in the long run. So, how should an application load translation
u> message catalogs in the future?

Using wxTranslations, please see the updated internat sample. Combining
both the locale functions and translations loading in wxLocale looked like
a good idea at the time, but turned out to be too limiting and it seems
cleaner now to have different classes for the different tasks.

u> The problem on Windows is that for many of the new allowed locales there
u> are no reasonable numeric IDs defined.

Yes, this is why wxUILocale uses wxLocaleIdent which is a (simplified)
form of BCP 47 tag.

u> If an I18N application is started for the first time, it is nice if it can
u> detect the actual default system locale, so that the default UI language
u> can be used to load the translation message catalog (at least if the
u> default language is supported by the application - which might not be the
u> case for exotic UI languages).
u>
u> I have not yet tested wxUILocale, but I have no doubt that it will load the
u> correct settings even for locales not in the language/locale list of
u> wxWidgets.

Yes, this is indeed supposed to work, please let me know if you see any
problems with the internat sample in any of the less usual locales.

u> Quite often it is useful for an I18N application to deduce the language,
u> the country, and sometimes the currency from the default system locale, so
u> that the application can start with default settings that meet the
u> expectations of the user(s).

This should be supported by GetInfo(), I think, but currently we don't
provide as much information as we could.

u> > u> 2. Shouldn't the list of languages in wxWidgets
u> > u> (misc/languages/langtabl.txt) be updated? It currently contains 236
u> > u> entries, while appendix A in the MS-LCID document contains over 800
u> > u> entries.
u> >
u> > No, I don't think we want to maintain this table. I mean, I won't stand in
u> > the way of anybody who wants to do it, but I don't think it's a good idea
u> > for us to maintain this list, which will invariably get out of data again
u> > soon anyhow.
u>
u> Maintaining the list manually is certainly not an option. However, the
u> process could be automated at least to a certain degree by extracting the
u> data from official data sources like ISO 639, ISO 3166, and the Unicode
u> CLDR project (CLDR = Common Locale Data Repository).

I wasn't ready to do the work necessary for this, but, again, if anybody
else is willing to do it, I certainly wouldn't object. Please note that
there still will be a problem with the old wx versions: even if we update
the different files in master regularly, programs using old versions of wx
still won't recognize any new languages/locales. And as I regularly see
programs using ~10 year old wx, this is not just a theoretical problem, it
does, and probably will, happen in practice. So relying on the OS still
looks like a better idea.

u> At least for Windows I could provide code to determine the list of locales.
u> The question is where this code should be located - in wxLocale (where the
u> list is currently generated from data in languageinfo.cpp), or in
u> wxUILocale, or elsewhere?

Definitely in wxUILocale.

u> I don't know whether Linux and macOS provide methods to retrieve a list of
u> known locales.

Under Mac there is [NSLocale availableLocaleIdentifiers] which is already
used in src/osx/core/uilocale.mm, so it would be simple to refactor the
code to make its results publicly available.

Under Linux there doesn't seem to be any standard way of getting this
information, but it should be possible as "locale -a" manages to get them
somehow, so looking at its sources (in glibc) should provide the answer.
FWIW under Debian there is /usr/share/i18n/SUPPORTED which contains what we
need, but I don't know how portable is it.

u> BTW, on Windows it is possible to retrieve the native display name for a
u> locale. This can be useful for a language selection list.

Yes, we should extend GetInfo() to allow returning this.

u> I hope my above remarks clarify this a bit.

To be honest, yes and no. I.e. I understand that some pieces of
information are not provided by the current API, but this should be easy to
address (and could be done after 3.2.0 because it wouldn't require breaking
the ABI, so is not the highest priority for me right now). But I'm still
not sure if there is something more fundamentally missing in wxUILocale,
which would need to be fixed before 3.2.0. Please try using wxUILocale and
let us know if we have to change anything.

TIA!
VZ

[utelle]

Hi Vadim,

VZ schrieb am Samstag, 30. Oktober 2021 um 15:03:34 UTC+2:

wxLocale is not formally deprecated because it still works as well (or as
badly) as always in wxMSW and wxGTK but shouldn't be used in the new code
because it simply can't be made to work in wxOSX. So wxLocale is legacy,
new code should only use wxUILocale.

Hm, how is this supposed to work? In respect to translations the class wxLocale offers several convenience methods. For example, for adding catalogs to the wxTranslations instance (created by the wxLocale instance), adding catalog search paths, automatically loading the wx default catalog. Developers may feel tempted to continue to use those methods.

Maybe those methods should be flagged as deprecated, because they work only on initialized wxLocale instance (with all the negative effects that led to introducing wxUILocale).

 

u> So, how should an application load translation

u> message catalogs in the future?

Using wxTranslations, please see the updated internat sample. Combining
both the locale functions and translations loading in wxLocale looked like
a good idea at the time, but turned out to be too limiting and it seems
cleaner now to have different classes for the different tasks.

In principle, I agree. However, looking at the updated internat sample reveals that wxLocale is still used 2 times:

1. In OnInit to detect the system default language (wxLocale::GetLanguageInfo(wxLANGUAGE_DEFAULT))
2. In OnTestLocaleAvail to detect the availability of a locale (wxLocale::FindLanguageInfo and wxLocale::IsAvailable)

IMHO these static methods of wxLocale should be moved either to wxUILocale or to a separate class (and the static methods of wxLocale should be adjusted to call the separate implementation). Otherwise wxLocale will be around forever.

 

u> I have not yet tested wxUILocale, but I have no doubt that it will load the
u> correct settings even for locales not in the language/locale list of
u> wxWidgets.

Yes, this is indeed supposed to work, please let me know if you see any
problems with the internat sample in any of the less usual locales.

I already compiled the current wxWidgets master in my development environment and I will test the internat sample.

 

u> Quite often it is useful for an I18N application to deduce the language,
u> the country, and sometimes the currency from the default system locale, so
u> that the application can start with default settings that meet the
u> expectations of the user(s).

This should be supported by GetInfo(), I think, but currently we don't
provide as much information as we could.

At least at the moment this information is not provided by GetInfo. However, wxLocale doesn't provide this information either, except for the language name (which may contain the country/region name).

IMHO it would be useful to be able to detect language, region, and currency of the system default locale. The question is, which class should contain such methods?

u> Maintaining the list manually is certainly not an option. However, the
u> process could be automated at least to a certain degree by extracting the
u> data from official data sources like ISO 639, ISO 3166, and the Unicode
u> CLDR project (CLDR = Common Locale Data Repository).

I wasn't ready to do the work necessary for this, but, again, if anybody
else is willing to do it, I certainly wouldn't object.

In principle, I have done most of the work for my own applications. However, the process involves a few extra tools: a Lua command shell, a SQLite3 command shell, and a XSLT processor for extracting relevant data from the Unicode CLDR.

Since the underlying data do not change so often, IMHO it would be sufficient to update the list when a new version of the Unicode CLDR is released.

 

Please note that
there still will be a problem with the old wx versions: even if we update
the different files in master regularly, programs using old versions of wx
still won't recognize any new languages/locales. And as I regularly see
programs using ~10 year old wx, this is not just a theoretical problem, it
does, and probably will, happen in practice.

Yes, that is indeed a problem. Maybe the list of locales could be provided as a separate file that can be loaded at runtime (like a translation catalog), instead of a hardcoded list in a source file.

 

So relying on the OS still looks like a better idea.

In principle, I agree. However, AFAIK Linux and macOS provide only lists of installed/supported, while Windows can be asked for a more or less complete list (although locales are not standardized as languages (ISO639) or countries (ISO 3166)).

 

u> At least for Windows I could provide code to determine the list of locales.
u> The question is where this code should be located - in wxLocale (where the
u> list is currently generated from data in languageinfo.cpp), or in
u> wxUILocale, or elsewhere?

Definitely in wxUILocale.

Ok, I will create a branch and will implement such a method.

u> I don't know whether Linux and macOS provide methods to retrieve a list of
u> known locales.

Under Mac there is [NSLocale availableLocaleIdentifiers] which is already
used in src/osx/core/uilocale.mm, so it would be simple to refactor the
code to make its results publicly available.

I'll take a look. However, I might need a bit support, because I'm absolutely not familiar with macOS.

 

Under Linux there doesn't seem to be any standard way of getting this
information, but it should be possible as "locale -a" manages to get them
somehow, so looking at its sources (in glibc) should provide the answer.
FWIW under Debian there is /usr/share/i18n/SUPPORTED which contains what we
need, but I don't know how portable is it.

I'll try to investigate the various Linux options.

u> I hope my above remarks clarify this a bit.

To be honest, yes and no. I.e. I understand that some pieces of
information are not provided by the current API, but this should be easy to
address (and could be done after 3.2.0 because it wouldn't require breaking
the ABI, so is not the highest priority for me right now). But I'm still
not sure if there is something more fundamentally missing in wxUILocale,
which would need to be fixed before 3.2.0. Please try using wxUILocale and
let us know if we have to change anything.

I will test wxUILocale and will report back, if I find issues.

Regards,

Ulrich

[Vadim Zeitlin]

On Sat, 30 Oct 2021 14:20:00 -0700 (PDT) utelle wrote:

u> VZ schrieb am Samstag, 30. Oktober 2021 um 15:03:34 UTC+2:
u>
u> > wxLocale is not formally deprecated because it still works as well (or as
u> > badly) as always in wxMSW and wxGTK but shouldn't be used in the new code
u> > because it simply can't be made to work in wxOSX. So wxLocale is legacy,
u> > new code should only use wxUILocale.
u> >
u>
u> Hm, how is this supposed to work? In respect to translations the class
u> wxLocale offers several convenience methods. For example, for adding
u> catalogs to the wxTranslations instance (created by the wxLocale instance),
u> adding catalog search paths, automatically loading the wx default catalog.
u> Developers may feel tempted to continue to use those methods.

The convenience methods are just wrappers for wxTranslations methods, so
it's just as simple -- and more direct -- to use wxTranslations directly.
The automatic loading of wx default catalog is considered by many people to
be a misfeature as it forces you to deal with the locale and translations
at the same time (and hence the need for various hacks such as
wxLOCALE_DONT_LOAD_DEFAULT), so this is something that wxUILocale
intentionally doesn't do.

u> Maybe those methods should be flagged as deprecated, because they work only
u> on initialized wxLocale instance (with all the negative effects that led to
u> introducing wxUILocale).

My rationale for not marking them as deprecated is that wxLocale continues
to work for the programs targeting Windows and Unix/Linux systems only, and
it seems disproportionate to force people who don't even target Mac to
change their programs just because Apple is incapable of writing code for
parsing SVG working in any C locale.

u> In principle, I agree. However, looking at the updated internat sample
u> reveals that wxLocale is still used 2 times:
u>
u> 1. In OnInit to detect the system default language
u> (wxLocale::GetLanguageInfo(wxLANGUAGE_DEFAULT))

This one should be indeed replaced with wxUILocale::GetInfo(wxLOCALE_NAME)
or something like this.

u> 2. In OnTestLocaleAvail to detect the availability of a locale
u> (wxLocale::FindLanguageInfo and wxLocale::IsAvailable)

These methods are much less useful IMO, but we could also refactor them to
move their implementation in wxUILocale, I guess.

u> IMHO these static methods of wxLocale should be moved either to wxUILocale
u> or to a separate class (and the static methods of wxLocale should be
u> adjusted to call the separate implementation). Otherwise wxLocale will be
u> around forever.

I'm fine with this, as I wrote above, I don't seriously plan deprecating
nor removing it.

u> IMHO it would be useful to be able to detect language, region, and currency
u> of the system default locale. The question is, which class should contain
u> such methods?

wxUILocale, definitely. The fact that wxLocale remains available doesn't
mean that we should continue improving it.

u> > So relying on the OS still looks like a better idea.
u>
u> In principle, I agree. However, AFAIK Linux and macOS provide only lists of
u> installed/supported, while Windows can be asked for a more or less complete
u> list (although locales are not standardized as languages (ISO639) or
u> countries (ISO 3166)).

But why would you care about a locale which is not supported by the system
anyhow (in the context of a GUI desktop application, at least)?

u> I will test wxUILocale and will report back, if I find issues.

TIA!
VZ

[utelle]

Hi Vadim,

VZ schrieb am Sonntag, 31. Oktober 2021 um 01:43:39 UTC+2:

u> In respect to translations the class wxLocale offers several convenience methods.

u> [...]

u> Developers may feel tempted to continue to use those methods.

The convenience methods are just wrappers for wxTranslations methods, so
it's just as simple -- and more direct -- to use wxTranslations directly.

I see. Indeed, it doesn't require much more effort to separate locale-specific operations from translation-specific operations.

The automatic loading of wx default catalog is considered by many people to
be a misfeature as it forces you to deal with the locale and translations
at the same time (and hence the need for various hacks such as
wxLOCALE_DONT_LOAD_DEFAULT), so this is something that wxUILocale
intentionally doesn't do.

Ok, agreed. The wx I18N documentation should make it clear that handling of translations should be strictly separated from handling of the locale. Currently, the option to handle translations via the wxLocale class is mentioned as equivalent. Maybe this should simply be put into a footnote, with a note that this approach is not recommended for new applications.

 

My rationale for not marking them as deprecated is that wxLocale continues
to work for the programs targeting Windows and Unix/Linux systems only, and
it seems disproportionate to force people who don't even target Mac to
change their programs just because Apple is incapable of writing code for
parsing SVG working in any C locale.

Ok, I understand your reasoning. However, for new applications it should be strongly recommended to use wxUILocale and wxTranslations (instead of wxLocale).

 

u> 1. In OnInit to detect the system default language
u> (wxLocale::GetLanguageInfo(wxLANGUAGE_DEFAULT))

This one should be indeed replaced with wxUILocale::GetInfo(wxLOCALE_NAME)
or something like this.

 

Adding some items to the wxLocaleInfo enumeration to retrieve additional useful information should be easy enough and wouldn't affect the ABI - as you already stated.

I will start to modify the implementation and will then make a PR. You can then decide how to proceed.

u> 2. In OnTestLocaleAvail to detect the availability of a locale
u> (wxLocale::FindLanguageInfo and wxLocale::IsAvailable)

These methods are much less useful IMO, but we could also refactor them to
move their implementation in wxUILocale, I guess.

In Windows a user has sometimes to take action to make certain locales available by installing language/locale support. Therefore it can be useful to find out which locales are directly supported without trying to actually activate them. This could influence the selection list presented to the user.

These functions are certainly not critical in any way, but IMHO locale-related methods should be located in wxUILocale.

 

u> IMHO it would be useful to be able to detect language, region, and currency
u> of the system default locale. The question is, which class should contain
u> such methods?

wxUILocale, definitely. The fact that wxLocale remains available doesn't
mean that we should continue improving it.

Ok, I'll start to implement such methods.

u> However, AFAIK Linux and macOS provide only lists of

u> installed/supported, while Windows can be asked for a more or less complete
u> list (although locales are not standardized as languages (ISO639) or
u> countries (ISO 3166)).

But why would you care about a locale which is not supported by the system
anyhow (in the context of a GUI desktop application, at least)?

Fair comment.  Probably on Windows this list should then also contain only the supported locales.

Having access to a more or less complete list of languages and countries can be useful to present a list of available translations, even if the corrresponding locale is not supported on the system.  From the message catalogs you can usually deduce only the abbreviation (like "en_US", "de_DE" etc), but displaying "English (USA)", "German (Germany)" etc would be preferrable. I have not yet checked whether wxUILocale allows to access the full locale information for locales not supported by the system.

u> I will test wxUILocale and will report back, if I find issues.

 

I will now start to test wxUILocale in more detail and to implement some of the functionality that would be useful from my point of view. I'll let you know as soon as there is something available to look at.

Regards,

Ulrich

[Vadim Zeitlin]

On Sun, 31 Oct 2021 08:09:51 -0700 (PDT) utelle wrote:

u> Ok, agreed. The wx I18N documentation should make it clear that handling of
u> translations should be strictly separated from handling of the locale.
u> Currently, the option to handle translations via the wxLocale class is
u> mentioned as equivalent. Maybe this should simply be put into a footnote,
u> with a note that this approach is not recommended for new applications.

I did add a note about wxUILocale to the beginning of wxLocale
documentation, do you think it's insufficient or if, perhaps, this should
be mentioned somewhere else?

u> > u> 1. In OnInit to detect the system default language
u> > u> (wxLocale::GetLanguageInfo(wxLANGUAGE_DEFAULT))
u> >
u> > This one should be indeed replaced with wxUILocale::GetInfo(wxLOCALE_NAME)
u> > or something like this.
u>
u> Adding some items to the wxLocaleInfo enumeration to retrieve additional
u> useful information should be easy enough and wouldn't affect the ABI - as
u> you already stated.
u>
u> I will start to modify the implementation and will then make a PR. You can
u> then decide how to proceed.

Thanks! FWIW we can still add new functions (and, in fact, wxUILocale was
written in an ABI-compatibility-friendly way, i.e. new functions can be
added to it without breaking the ABI compatibility) if you think it would
be more natural. I suspect it might because the name of the locale/language
is special as it can be available either in the language itself or in
English and so perhaps we should have a separate function or a pair of
functions for retrieving it, as GetInfo() doesn't have such "axis" (we
could overload the category parameter for this, but it probably would be a
bad idea).

u> u> 2. In OnTestLocaleAvail to detect the availability of a locale
u> > u> (wxLocale::FindLanguageInfo and wxLocale::IsAvailable)
u> >
u> > These methods are much less useful IMO, but we could also refactor them to
u> > move their implementation in wxUILocale, I guess.
u>
u> In Windows a user has sometimes to take action to make certain locales
u> available by installing language/locale support.

Is this still the case with Windows 10 and more recent versions? I thought
all locales were always available now, making worrying about this
unnecessary.

u> Having access to a more or less complete list of languages and countries
u> can be useful to present a list of available translations, even if the
u> corrresponding locale is not supported on the system. From the message
u> catalogs you can usually deduce only the abbreviation (like "en_US",
u> "de_DE" etc), but displaying "English (USA)", "German (Germany)" etc would
u> be preferrable.

This information is returned by Win32 API and maybe nl_langinfo(), but I'm
not sure about Mac.

Regards,
VZ

[utelle]

Hi Vadim,

VZ schrieb am Sonntag, 31. Oktober 2021 um 17:13:01 UTC+1:

u> [...] Maybe this should simply be put into a footnote,

u> with a note that this approach is not recommended for new applications.

I did add a note about wxUILocale to the beginning of wxLocale
documentation, do you think it's insufficient or if, perhaps, this should
be mentioned somewhere else?

IMHO  the note in the wxLocale documentation is sufficient. However, I would recommend to add a note to the documentation of wxTranslations recommending to use wxTranslations directly - if necessary accompanied by wxUILocale -, but not via wxLocale due to the known problems with wxLocale.

 

u> Adding some items to the wxLocaleInfo enumeration to retrieve additional
u> useful information should be easy enough and wouldn't affect the ABI - as
u> you already stated.
u>
u> I will start to modify the implementation and will then make a PR. You can
u> then decide how to proceed.

Thanks! FWIW we can still add new functions (and, in fact, wxUILocale was
written in an ABI-compatibility-friendly way, i.e. new functions can be
added to it without breaking the ABI compatibility) if you think it would
be more natural. I suspect it might because the name of the locale/language
is special as it can be available either in the language itself or in
English and so perhaps we should have a separate function or a pair of
functions for retrieving it, as GetInfo() doesn't have such "axis" (we
could overload the category parameter for this, but it probably would be a
bad idea).

I don't think we need additional methods. Similar to the values used in the Windows API we can simply define additional wxLocaleInfo enum values like

• wxLOCALE_ENGLISHLANGUAGENAME - Name of the language in English
  
• wxLOCALE_NATIVELANGUAGENAME - Name of the language in the native locale language
  
• wxLOCALE_ENGLISHCOUNTRYNAME - Name of the country in English
  
• wxLOCALE_NATIVECOUNTRYNAME - Name of the country in the native locale language
  
• wxLOCALE_ENGLISHDISPLAYNAME - Name of the locale (usually in the form "Language (Country)") in English
  
• wxLOCALE_NATIVEDISPLAYNAME - Name of the locale in the native locale language
  
• ...
  

u> In Windows a user has sometimes to take action to make certain locales
u> available by installing language/locale support.

Is this still the case with Windows 10 and more recent versions? I thought
all locales were always available now, making worrying about this
unnecessary.

I have to admit that I haven't tested this yet. At least when calling the enumeration function one can ask for ALL locales or for SUPPORTED locales. Additionally, in the selection box for the Windows display language in the Windows 10 settings for the language I can only select the languages that I added previously to the list of preferred languages.

 

u> Having access to a more or less complete list of languages and countries
u> can be useful to present a list of available translations, even if the
u> corrresponding locale is not supported on the system. From the message
u> catalogs you can usually deduce only the abbreviation (like "en_US",
u> "de_DE" etc), but displaying "English (USA)", "German (Germany)" etc would
u> be preferrable.

This information is returned by Win32 API and maybe nl_langinfo(), but I'm
not sure about Mac.

According to the documentation of nl_langinfo I can use non-standard selectors to retrieve this information. I will test that in my OpenSuSE Linux environment.

I have no idea how to do it in macOS. Maybe nl_langinfo can be used there, too. Or there are system functions available that can be used for this purpose.

Regards,

Ulrich

[Vadim Zeitlin]

On Sun, 31 Oct 2021 10:04:05 -0700 (PDT) utelle wrote:

u> IMHO the note in the wxLocale documentation is sufficient. However, I
u> would recommend to add a note to the documentation of wxTranslations
u> recommending to use wxTranslations directly - if necessary accompanied by
u> wxUILocale -, but not via wxLocale due to the known problems with wxLocale.

Yes, looking at wxTranslations summary, it should be updated to speak less
of wxLocale and at least mention wxUILocale...

u> I don't think we need additional methods. Similar to the values used in the
u> Windows API we can simply define additional wxLocaleInfo enum values like
u>
u>
u> - wxLOCALE_ENGLISHLANGUAGENAME - Name of the language in English
u> - wxLOCALE_NATIVELANGUAGENAME - Name of the language in the native
u> locale language
u> - wxLOCALE_ENGLISHCOUNTRYNAME - Name of the country in English
u> - wxLOCALE_NATIVECOUNTRYNAME - Name of the country in the native locale
u> language
u> - wxLOCALE_ENGLISHDISPLAYNAME - Name of the locale (usually in the form
u> "Language (Country)") in English
u> - wxLOCALE_NATIVEDISPLAYNAME - Name of the locale in the native locale
u> language
u> - ...

We don't _need_ them, but IMO having both ENGLISH and NATIVE constants is
not very nice. I think either of the alternatives I mentioned would be
better:

1. Use 2 different categories, wxLOCALE_CAT_NATIVE_NAME and
wxLOCALE_CAT_ENGLISH_NAME, for wxLOCALE_{LANGUAGE,COUNTRY,DISPLAY}_NAME
wxLocaleInfo constants with GetInfo(). This is simple, and while it
might be somewhat of a hack, I think it would still be better than
doubling all name constants as Win32 API does.

2. Use a different function taking values of different enum, e.g.

wxString GetName(wxLocaleName what, wxLocaleForm how) const;

with

enum wxLocaleName { wxLOCALE_NAME_LANGUAGE, wxLOCALE_NAME_COUNTRY, ... };
enum wxLocaleForm { wxLOCALE_FORM_NATIVE, wxLOCALE_FORM_ENGLISH };

(this would require renaming/doing something with the existing GetName()
to avoid confusion).

I don't really have time/mental energy to think about this in depth now,
so I don't know what to recommend but, again, either solution seems better
than just copying Win32 API to me.

One thing that we should also keep in mind is that we might (will?) want
to add access to the names of weekday and month names via wxUILocale too
and it could be nice to have an API that could be generalized to do it. Or
maybe not, because it probably isn't very useful to return the English
names of those, as they could be just hardcoded.


u> According to the documentation of nl_langinfo I can use non-standard
u> selectors to retrieve this information. I will test that in my OpenSuSE
u> Linux environment.

I'm pretty confident that it works on all non-ancient Linux systems,
support for this has been available in glibc since quite some time.

u> I have no idea how to do it in macOS. Maybe nl_langinfo can be used there,

No, definitely not. Apple doesn't care about POSIX locale APIs and even
when they're provided, they don't work or, worse, don't work correctly for
all the locales, and should be avoided on this platform.

u> Or there are system functions available that can be used for this
u> purpose.

The docs at https://developer.apple.com/documentation/foundation/nslocale?language=objc
list countryCode and languageCode and then localizedStringForCountryCode:
and localizedStringForLanguageCode: so it looks like we should have no
problem with obtaining at least the native strings. And we should be able
to get the English ones by simply calling the same methods on the "en_US"
locale.

Regards,
VZ

[Alec Teal]

Prominently in the documentation would be great.

We're using some more modern C++ stuff now - I feel we can afford to use
namespaces, wxExp (experimental) maybe, and stick the docs there?

This way we can use new features without them potentially polluting the
namespace and having to be re-worked (the worst case would be a change
with the same argument types but different meanings that broke stuff -
separate namespace help here)

This way a feature could evolve, and even exist in v1, v2, v3...
versions totally separate without breaking anything that used them, and
when happy and stable, moved into the main namespace as basically an
alias (or make the last version an alias) whatever


A "what's new" section in the docs (and a contact email to complain to,
make complaining about the API easy) - best you can do IMO.


Alec

[utelle]

Hi Vadim,

VZ schrieb am Sonntag, 31. Oktober 2021 um 19:29:16 UTC+1:

u> - wxLOCALE_ENGLISHLANGUAGENAME - Name of the language in English
u> - wxLOCALE_NATIVELANGUAGENAME - Name of the language in the native
u> locale language

u> - [...]

We don't _need_ them, but IMO having both ENGLISH and NATIVE constants is
not very nice.

True. Windows defines separate constants, but that doesn't mean that wxWidgets has to do it in the same way.

 

I think either of the alternatives I mentioned would be
better:

1. Use 2 different categories, wxLOCALE_CAT_NATIVE_NAME and
wxLOCALE_CAT_ENGLISH_NAME, for wxLOCALE_{LANGUAGE,COUNTRY,DISPLAY}_NAME
wxLocaleInfo constants with GetInfo(). This is simple, and while it
might be somewhat of a hack, I think it would still be better than
doubling all name constants as Win32 API does.

Certainly one possible option.

2. Use a different function taking values of different enum, e.g.

wxString GetName(wxLocaleName what, wxLocaleForm how) const;

with

enum wxLocaleName { wxLOCALE_NAME_LANGUAGE, wxLOCALE_NAME_COUNTRY, ... };
enum wxLocaleForm { wxLOCALE_FORM_NATIVE, wxLOCALE_FORM_ENGLISH };

(this would require renaming/doing something with the existing GetName()
to avoid confusion).

If we decide to introduce a new method, it could be named GetLocalizedName.

 

I don't really have time/mental energy to think about this in depth now,
so I don't know what to recommend but, again, either solution seems better
than just copying Win32 API to me.

Well, I brought up this topic, but I don't expect you to do the work. Taking your comments into account I will try to make a proposal, after taking a closer look about available interfaces on the various platforms (Windows, Linux, macOS).

 

One thing that we should also keep in mind is that we might (will?) want
to add access to the names of weekday and month names via wxUILocale too
and it could be nice to have an API that could be generalized to do it. Or
maybe not, because it probably isn't very useful to return the English
names of those, as they could be just hardcoded.

The main question is for which purpose those names are required/useful. If the goal is to retrieve translations matching the current locale, I think hardcoded English names would be sufficient. Translations can be extracted from the Unicode CLDR data and can then be made available via message catalogs.

Actually, the same is true for language/country/currency names. However, while the list of weekday or month names is short, lists for languages/countries/currencies are longer and can change over time, so that the question of how to update the lists with English names should be handled.

If the operating system provides easy access to localized strings, using them seems to be preferrable. However, under Linux building the lists may be quite expensive, because a locale for each language would have to be created, before calling nl_langinfo_l, and destroyed thereafter.

 

u> I have no idea how to do it in macOS. Maybe nl_langinfo can be used there,

[...]

u> Or there are system functions available that can be used for this
u> purpose.

The docs at https://developer.apple.com/documentation/foundation/nslocale?language=objc
list countryCode and languageCode and then localizedStringForCountryCode:
and localizedStringForLanguageCode:

Thanks for the pointer.

 

Regards,

Ulrich

[utelle]

Hi Vadim,

in the meantime I made a few preliminary tests with the internat sample and analyzed the wxUILocale and wxTranslations code to understand how it works.

For wxUILocale testing the internat sample in its current form is not so well suited, because the UI locale can't be chosen freely on starting the sample - only English or the default user locale is used. I intend to adjust the sample code a bit.

However, analyzing the code I came across a few issues that need to be clarified, before I really start implementing anything.

1. The header file language.h defines the enumeration wxLanguage, which is generated from the file misc/languages/langtabl.txt. Since this enumeration needs to be available at compile time, generating the list of known languages/locales at runtime is basically not feasible. Since the wxLanguage type is used in wxLocale, wxUILocale, and wxTranslations and most likely in many user applications, it will be necessary to support the wxLanguage enumeration.
   
   However, IMHO the list in langtabl.txt should be updated to reflect the greater number of locales supported by modern operating systems. I'm ready to tackle this task. However, it would be difficult to update the list in such a way that the order of enumeration entries do not change for languages defined in the current version of langtabl.txt. As long as developers always used the enumeration constants this should not impose a problem.
   
   Instead of updating langtabl.txt an alternative could be to add additional languages/locales at runtime as user-defined languages to the language database, but I think that would be quite a hack.
   
   Another alternative could be to generate the full list of supported languages at runtime and map the legacy language database to the new list using the string language IDs.
   
   
2. In the light of issue 1 I wonder whether generating the full list of supported locales/languages at runtime is really needed. Actually, I guess most applications will only check whether the locales corresponding to the translations provided by the application can be set successfully. And this could be done based on the list of available translations already provided by wxTranslations.
   
   Additionally one has to take into account that generating the locale list at runtime might be time consuming. For example, under Linux it will be necessary to scan a system directory for available locales.
   

VZ>  Please note that
VZ>  there still will be a problem with the old wx versions: even if we update
VZ>  the different files in master regularly, programs using old versions of wx
VZ>  still won't recognize any new languages/locales. And as I regularly see
VZ>  programs using ~10 year old wx, this is not just a theoretical problem, it
VZ>  does, and probably will, happen in practice. So relying on the OS still
VZ>  looks like a better idea.

For applications that use old wx versions and wxLocale it can't be helped. Nevertheless, IMHO langtabl.txt should be updated from time to time. For applications that use wxUILocale the static language database should not play a role, that is, the methods of wxUILocale and wxTranslations should not rely on wxLanguage values, but should use string identifiers consequently. AFAICT this would require to add a method AddCatalog to wxTranslations that uses a string identifier instead of a wxLanguage value for the msgIdLanguage. This still leaves the question what should be done with class wxLanguageInfo for languages not in the language database.

Regards,

Ulrich

[Vadim Zeitlin]

On Mon, 1 Nov 2021 15:16:19 -0700 (PDT) utelle wrote:

u> For wxUILocale testing the internat sample in its current form is not so
u> well suited, because the UI locale can't be chosen freely on starting the
u> sample - only English or the default user locale is used. I intend to
u> adjust the sample code a bit.

No, sorry, you can't do this -- this is just not something that can be
done under macOS and so it's intentionally not supported by wxUILocale.
I.e. the UI locale is really fixed there and can't be changed on program
startup, which is why your only choice is to use this locale in your
program or not do it.

In principle, you can still propose using some other language for the
translations, but I don't know how much sense does it make to use German
labels in English locale or vice versa.

u> However, analyzing the code I came across a few issues that need to be
u> clarified, before I really start implementing anything.
u>
u> 1. The header file language.h defines the enumeration wxLanguage, which
u> is generated from the file misc/languages/langtabl.txt. Since this
u> enumeration needs to be available at compile time, generating the list of
u> known languages/locales at runtime is basically not feasible. Since the
u> wxLanguage type is used in wxLocale, wxUILocale, and wxTranslations and
u> most likely in many user applications, it will be necessary to support the
u> wxLanguage enumeration.

Note that the new code is supposed to use wxLocaleIdent rather than
wxLanguage enum.

u> 2. In the light of issue 1 I wonder whether generating the full list of
u> supported locales/languages at runtime is really needed.

I don't really see much need for this personally...

u> Actually, I guess most applications will only check whether the
u> locales corresponding to the translations provided by the application
u> can be set successfully. And this could be done based on the list of
u> available translations already provided by wxTranslations.

Yes, exactly.

u> Nevertheless, IMHO langtabl.txt should be updated from time to time.

I don't have any objections to doing it, of course.

u> AFAICT this would require to add a method AddCatalog to wxTranslations
u> that uses a string identifier instead of a wxLanguage value for the
u> msgIdLanguage.

Note that msgIdLanguage is the language of the IDs, i.e. keys, not
translations and in 99% of the cases it will be English. IOW this doesn't
prevent you from having translations to a language unknown to wx (I think;
I have to admit I didn't test this).

Regards,
VZ

[utelle]

Hi Vadim,

VZ schrieb am Montag, 1. November 2021 um 23:30:52 UTC+1:

u> For wxUILocale testing the internat sample in its current form is not so
u> well suited, because the UI locale can't be chosen freely on starting the
u> sample - only English or the default user locale is used. I intend to
u> adjust the sample code a bit.

No, sorry, you can't do this -- this is just not something that can be
done under macOS and so it's intentionally not supported by wxUILocale.
I.e. the UI locale is really fixed there and can't be changed on program
startup, which is why your only choice is to use this locale in your
program or not do it.

Thanks for the clarification. I was not aware of this limitation under macOS.

However, under Windows and Linux changing the locale is possible. So, why not allowing it on those platforms in the internat sample?

In principle, you can still propose using some other language for the
translations, but I don't know how much sense does it make to use German
labels in English locale or vice versa.

At least while testing during the development of an i18n application, I found it very important to be able to change the UI language, even if the locale (i.e. formatting of numbers and dates) could not be changed (for uninstalled locales).

 

u> 1. The header file language.h defines the enumeration wxLanguage, which
u> is generated from the file misc/languages/langtabl.txt. Since this
u> enumeration needs to be available at compile time, generating the list of
u> known languages/locales at runtime is basically not feasible. Since the
u> wxLanguage type is used in wxLocale, wxUILocale, and wxTranslations and
u> most likely in many user applications, it will be necessary to support the
u> wxLanguage enumeration.

Note that the new code is supposed to use wxLocaleIdent rather than
wxLanguage enum.

Hm, for example in src/common/appcmn.cpp the layout direction is determined implicitly via wxLanguage. I didn't check whether there are other places in the wx code where still wxLanguage is used either directly or indirectly.

 

u> Nevertheless, IMHO langtabl.txt should be updated from time to time.

I don't have any objections to doing it, of course.

Ok. I will give it a try.

Note that msgIdLanguage is the language of the IDs, i.e. keys, not
translations

Yes, I know.

and in 99% of the cases it will be English.

True. And personally, I stick to English messages in the code, if there is the slightest chance that the application needs to support different languages.

However, out there are developers with minor English skills choosing their native language (German or whatever) as the msgId language.

Adding a AddCatalog variant which uses a string language id or wxLocaleIdent should not be a big deal.

IOW this doesn't prevent you from having translations to a language unknown to wx (I think;
I have to admit I didn't test this).

Yes, loading translations to a language unknown to wx works (as long as you explicitly set the target language). However, that's not the point. The language of the msgId strings (the language used in the source code) can still be different from English. Since wxTranslations::LoadCatalog uses string ids anyway, a method AddCatalog which allows to use a string based language identifier would be easy to implement.

Regards,

Ulrich

[Vadim Zeitlin]

On Mon, 1 Nov 2021 16:26:26 -0700 (PDT) utelle wrote:

u> However, under Windows and Linux changing the locale is possible. So, why
u> not allowing it on those platforms in the internat sample?

My rationale is that we want to explain to people who want to implement
choosing the application language that they need to relaunch their program
after changing the language for this to work under macOS. So, if anything,
it would be nice to show how to do this in the internat sample -- this
would be much more useful than showing how to do something which doesn't
work on one of the Tier-1 platforms.

Also notice that even though we can change the locale in wxMSW and wxGTK,
I'm not sure how well does it work in the latter, it's not impossible that
there could be some problems there too. And we don't support re-translating
the UI on the fly in wx API in any case. So re-executing the application to
switch to a different language could be a good idea even if you don't care
about macOS.

u> > Note that the new code is supposed to use wxLocaleIdent rather than
u> > wxLanguage enum.
u>
u> Hm, for example in src/common/appcmn.cpp the layout direction is determined
u> implicitly via wxLanguage.

Yes, this is bad, we should provide a way to get this information from
wxUILocale directly, thanks for noticing this. It's probably worth opening
a ticket to remind us to do it -- unless you could just do it as part of
your changes, if it's not too difficult.

u> Yes, loading translations to a language unknown to wx works (as long as you
u> explicitly set the target language). However, that's not the point. The
u> language of the msgId strings (the language used in the source code) can
u> still be different from English. Since wxTranslations::LoadCatalog uses
u> string ids anyway, a method AddCatalog which allows to use a string based
u> language identifier would be easy to implement.

Again, I definitely don't mind doing it, but IMO it's just not a problem
in practice: even though there are some people who use other languages (to
be fair, I've seen all of German, French and Italian myself), they're still
all relatively common languages covered by wxLanguage constants and the
chances of someone using some unsupported language for the message IDs
seems very, very low.

Regards,
VZ

[utelle]

Hi Vadim,

VZ schrieb am Dienstag, 2. November 2021 um 12:57:24 UTC+1:

u> However, under Windows and Linux changing the locale is possible. So, why
u> not allowing it on those platforms in the internat sample?

My rationale is that we want to explain to people who want to implement
choosing the application language that they need to relaunch their program 

after changing the language for this to work under macOS.

I'm a bit confused now.  In a previous post you wrote that under macOS you can activate the system default locale or no locale at all (which is equivalent to the C locale). That is, relaunching a program wouldn't change anything, unless you change the UI locale of macOS itself inbetween.

Nevertheless, the UI language could be changed to one that differs from the language tied to the default locale. Even if the number formatting still conforms to the UI default locale (say German for example), it would make sense to be able to change the UI language to something else (say French for example, if a colleague from France is visiting).

Typically, on first start an application will try to switch to the default UI locale if its UI language belongs to the supported languages. If the latter is not the case, usually the best option is to not set any locale and use English as the UI language.

Within the application there can be a settings dialog which allows to select a different UI language that would then be used when the application is relaunched. To be frank, afetr relaunch I would then try to set not only the UI language, but also the UI locale. Yes, that doesn't work under macOS, but under Windows and Linux it does.

So, if anything, it would be nice to show how to do this in the internat sample -- this
would be much more useful than showing how to do something which doesn't
work on one of the Tier-1 platforms.

You mean a dialog for selecting a different language (for loading a different message catalog) which would be applied after relaunch?

Also notice that even though we can change the locale in wxMSW and wxGTK,
I'm not sure how well does it work in the latter, it's not impossible that
there could be some problems there too.

Hm, AFAIK wxLocale always changes the locale, when a different language is selected for the message catalogs. If there were serious problems under wxGTK, I guess that there would have been plenty of bug reports and complaints. I'm not aware of such reports, but I may be wrong.

 

And we don't support re-translating the UI on the fly in wx API in any case.

There is no direct support in wxWidgets for re-translating the UI on the fly, but it is doable (and it's a nice feature from the user's perspective.

 

So re-executing the application to switch to a different language could be a good idea

even if you don't care about macOS.

Such a feature could be added to the internat sample, and it should work under macOS, too, because it doesn't affect the locale, but the message catalog that will be loaded.

 

u> Hm, for example in src/common/appcmn.cpp the layout direction is determined
u> implicitly via wxLanguage.

Yes, this is bad, we should provide a way to get this information from
wxUILocale directly, thanks for noticing this. It's probably worth opening
a ticket to remind us to do it -- unless you could just do it as part of
your changes, if it's not too difficult.

Under Windows the layout direction can be queried from the locale. I have to check how to do it under Linux and macOS.

u> Since wxTranslations::LoadCatalog uses

u> string ids anyway, a method AddCatalog which allows to use a string based
u> language identifier would be easy to implement.

Again, I definitely don't mind doing it, but IMO it's just not a problem
in practice:

For the sake of completeness I would add such a method, even if it will most likely not be a problem to use a wxLanguage constant most of the time.

 

even though there are some people who use other languages (to
be fair, I've seen all of German, French and Italian myself), they're still
all relatively common languages covered by wxLanguage constants and the
chances of someone using some unsupported language for the message IDs
seems very, very low.

I agree, it is extremely unlikely that someone would use for example Colognian ("Kölsch") as the base language for messages in source code. :)

So, here is a summary, what I intend to do/implement (the order of the below list does not imply priorities):

• Update the file langtabl.txt in misc/languages;
• Implement the language database in wxUILocale and redirect the static wxLocale methods to the counterparts in wxUILocale. IMHO wxUILocale is the more logical place, but let me know if you have a different opinion;
• Add wxUILocale method GetLocalizedName to retrieve language/country... names in English and in localized language;
• Add method to wxUILocale to retrieve the layout direction of a locale (depending on the locale Windows returns LTR, RTL or TTB - the latter should probably be interpreted as LTR for the time being), since we keep the language database, this information could be retrieved from there;
  
• Change src/common/appcmn.cpp to query the layout direction via wxUILocale;
• Enhance the internat sample to allow changing the translation language
  

If I missed any important item, just let me know.

Regards,

Ulrich

[Vadim Zeitlin]

On Tue, 2 Nov 2021 13:07:58 -0700 (PDT) utelle wrote:

u> I'm a bit confused now. In a previous post you wrote that under macOS you
u> can activate the system default locale or no locale at all (which is
u> equivalent to the C locale). That is, relaunching a program wouldn't change
u> anything, unless you change the UI locale of macOS itself inbetween.

Sorry for being imprecise. You can actually specify the locale of the
program under Mac by using special (i.e. handled by the system itself)
command-line options. But you can't change the locale of the process once
it's running.

u> You mean a dialog for selecting a different language (for loading a
u> different message catalog) which would be applied after relaunch?

Yes.

u> There is no direct support in wxWidgets for re-translating the UI on the
u> fly, but it is doable (and it's a nice feature from the user's perspective.

Everything is doable, of course, but I don't think it is that simple. And
this would impose extra overhead on all wx programs, even those not needing
this, as we'd need to store all menu strings, which we don't do currently.

u> So, here is a summary, what I intend to do/implement (the order of the
u> below list does not imply priorities):
u>
u> - Update the file langtabl.txt in misc/languages;
u> - Implement the language database in wxUILocale and redirect the static
u> wxLocale methods to the counterparts in wxUILocale. IMHO wxUILocale is the
u> more logical place, but let me know if you have a different opinion;
u> - Add wxUILocale method GetLocalizedName to retrieve language/country...
u> names in English and in localized language;
u> - Add method to wxUILocale to retrieve the layout direction of a locale
u> (depending on the locale Windows returns LTR, RTL or TTB - the latter
u> should probably be interpreted as LTR for the time being), since we keep
u> the language database, this information could be retrieved from there;
u> - Change src/common/appcmn.cpp to query the layout direction via
u> wxUILocale;
u> - Enhance the internat sample to allow changing the translation language
u>
u> If I missed any important item, just let me know.

No, this looks very complete to me, thanks in advance!
VZ

[utelle]

Hi Vadim,

VZ schrieb am Dienstag, 2. November 2021 um 23:58:15 UTC+1:

u> [...] change the UI locale of macOS itself [...]

Sorry for being imprecise. You can actually specify the locale of the
program under Mac by using special (i.e. handled by the system itself)
command-line options. But you can't change the locale of the process once
it's running.

Thanks for the clarification.

u> There is no direct support in wxWidgets for re-translating the UI on the
u> fly, but it is doable (and it's a nice feature from the user's perspective.

Everything is doable, of course, but I don't think it is that simple.

No, it isn't. I've done it for a large application. Although it would be possible to re-translate static strings quite easily, this would cover the re-translation task only partially. There are composite and/or parameterized message for which automating re-translation would be difficult. And then there are dynamic selection lists in combo boxes and the like ... So, I'm fine with the current wxWidgets approach.

 

u> So, here is a summary, what I intend to do/implement (the order of the
u> below list does not imply priorities):

u> [...]

u> If I missed any important item, just let me know.

No, this looks very complete to me, thanks in advance!

While preparing the update of the language list (langtabl.txt) I stumbled over a small subtlety. The current list contains a few entries with modifiers like "@latin" or "@valencia". The current implementation of wxUILocaleImpl::CreateForLanguage uses under Windows the LCID to identify the requested locale and under other systems (Linux, macOS) the unmodified modifier. Under Windows I would like to get rid of using LCID in wxUILocale as far as possible. However, if the modifier designates a script like "latin" the correct 4-letter script tag would be "Latn" - this is true also under macOS. Only under Linux I see in my OpenSuSE installation that the modifier "latin" is used in such a case.

In the updated language list there will be some entries with script tags. However, I'm undecided how to decode them to the identifier used in langtabl.txt:

Latn => latin, Cyrl => ?

Actually, I would prefer to use the 4-letter script codes and decode them under Linux if necessary. What's your opinion?

Regards,

Ulrich

[Vadim Zeitlin]

On Thu, 4 Nov 2021 04:36:53 -0700 (PDT) utelle wrote:

u> While preparing the update of the language list (langtabl.txt) I stumbled
u> over a small subtlety. The current list contains a few entries with
u> modifiers like "@latin" or "@valencia". The current implementation of
u> wxUILocaleImpl::CreateForLanguage uses under Windows the LCID to identify
u> the requested locale and under other systems (Linux, macOS) the unmodified
u> modifier. Under Windows I would like to get rid of using LCID in wxUILocale
u> as far as possible. However, if the modifier designates a script like
u> "latin" the correct 4-letter script tag would be "Latn" - this is true also
u> under macOS. Only under Linux I see in my OpenSuSE installation that the
u> modifier "latin" is used in such a case.
u>
u> In the updated language list there will be some entries with script tags.
u> However, I'm undecided how to decode them to the identifier used in
u> langtabl.txt:
u>
u> Latn => latin, Cyrl => ?
u>
u> Actually, I would prefer to use the 4-letter script codes and decode them
u> under Linux if necessary. What's your opinion?

We should definitely try to accept standard (i.e. 4 letter ISO 15924)
script codes and translate them to whatever is used by the system if
necessary, so if it's possible to use "Latn" and map it to "latin" under
Linux, we should do it.

TIA!
VZ

[Vadim Zeitlin]

On Sun, 31 Oct 2021 18:51:20 +0000 Alec Teal wrote:

AT> Prominently in the documentation would be great.

We do include a link to the changes since the last major version on the
main docs page, which is quite prominent IMHO, but this link will need to
be updated for 3.2 (currently it still points to the page describing
changes in 3.0 compared to 2.8).

I'll try to do it, and for now I've posted this

https://www.wxwidgets.org/news/2021/10/roundup-of-the-recent-changes/

and will also post a link to this post to wx-users.

AT> We're using some more modern C++ stuff now - I feel we can afford to use
AT> namespaces, wxExp (experimental) maybe, and stick the docs there?

Sorry, I don't understand what do you mean here. Just to be clear, we're
speaking about disseminating information about the changes already done,
not about actually making more changes, so I really don't see how do the
namespaces could possibly be relevant here.

Regards,
VZ

[al...@unifiedmathematics.com]

With all this stuff around the icon bundle you may want to consider it,
if it does turn out to be in some way unsuitable you will need /another/
name.

As for the documentation, Doxygen lets you create your own pages and
that (as you probably know for examples and concept pages) - use those!
But there's certainly no link at the top to find what's new.

---

[utelle]

I started to work on updating the file langtabl.txt with the list of languages/locales known in wx. I implemented a small Windows program to enumerate the locales known under Windows 10. This results in a list of 856 entries. Under OpenSuSE Linux 15.2 I used the command "locale -a" to retrieve a list of known locales (484 entries - with many duplicates which differ only in the modifier (no modifier vs modifier ".utf8")). Under macOS I used also the command "locale -a" resulting in a list with 203 entries - again many duplicates with different modifiers (.UTF-8, .ISO8859-1 ...). That is, the list under Windows seems to be the most complete, so that I intend to use it for updating langtabl.txt.

However, there are 2 problems for which I would like to get a second opinion.

1. The identifiers wxLANGUAGE_* which are collected in enum wxLanguage need to be generated somehow. I use English language, country, and script names to deduce these names. In principle, this works well, but in several cases this leads to inconsistencies with the prior list of names. Example:
   
   The current file langtabl.txt contains the entry
   
   wxLANGUAGE_AFRIKAANS associated with the canonical name "af_ZA"
   
   Windows delivers 3 entries for the language Afrikaans (country neutral, South Africa, Namibia) from which I generate
   
   wxLANGUAGE_AFRIKAANS                                canonical name "af"
   wxLANGUAGE_AFRIKAANS_NAMIBIA              canonical name "af_NA"
   wxLANGUAGE_AFRIKAANS_SOUTH_AFRICA  canonical name "af_ZA"
   
   That is, wxLANGUAGE_AFRIKAANS would be associated with a different canonical name.
   
   Usually, langtabl.txt uses canonical names which include the country/region identifier (like "de_DE"), but this is not always so (for example for Arabic ("ar")).
   
   In most situations this should not impose a problem, because the right language is identified. However, in conjunction with loading message catalogs for translations this can lead to the second problem
   
   
2. When loading a message catalog the used algorithm looks first for a subdirectory named as the full canonical name. If no subdirectory is found, the subdirectory using only the language part (before the first underscore in the canonical name) is checked. Now, if a user used for the above mentioned example subdirectory "af_ZA" for Afrikaans translations it would not be found in the new version which would return for wxLANGUAGE_AFRIKAANS the canonical name "af"

One way to solve the problem would be to associate a default region for locales for which the canonical name consists only of the language part. This would require to extend wxLanguageInfo, so that for example the canonical name "af" could be associated with "af_ZA". Opinions?

BTW, I have the impression that the algorithm for locating the translation subdirectories fails when the canonical name contains a script name. wxUILocale returns a canonical name which uses the minus sign as a separator. The separator is replaced by an underscore, but the current logic for the subdirectory names uses the at sign (@) to append the script name.

Regards,

Ulrich

[Vadim Zeitlin]

On Sat, 6 Nov 2021 07:09:42 -0700 (PDT) utelle wrote:

u> I started to work on updating the file langtabl.txt with the list of
u> languages/locales known in wx. I implemented a small Windows program to
u> enumerate the locales known under Windows 10. This results in a list of 856
u> entries. Under OpenSuSE Linux 15.2 I used the command "locale -a" to
u> retrieve a list of known locales (484 entries - with many duplicates which
u> differ only in the modifier (no modifier vs modifier ".utf8")). Under macOS
u> I used also the command "locale -a" resulting in a list with 203 entries -

The list returned by [NSLocale availableLocaleIdentifiers] is much larger,
because POSIX APIs are just a redheaded stepchild under macOS.

u> However, there are 2 problems for which I would like to get a second
u> opinion.
u>
u> 1. The identifiers wxLANGUAGE_* which are collected in enum wxLanguage
u> need to be generated somehow. I use English language, country, and script
u> names to deduce these names. In principle, this works well, but in several
u> cases this leads to inconsistencies with the prior list of names.

I'm afraid we'd have to remain compatible with the existing constants,
just because I'm not sure it's safe to change their meaning and there
doesn't seem to be any good reason to take the risk of doing it if it can
be avoided (and I hope it can, if only by maintaining a small list of
exceptions).

u> 2. When loading a message catalog the used algorithm looks first for a
u> subdirectory named as the full canonical name. If no subdirectory is found,
u> the subdirectory using only the language part (before the first underscore
u> in the canonical name) is checked. Now, if a user used for the above
u> mentioned example subdirectory "af_ZA" for Afrikaans translations it would
u> not be found in the new version which would return for wxLANGUAGE_AFRIKAANS
u> the canonical name "af"

We'd definitely like to avoid breaking the existing programs expecting
Afrikaans translations to be found under "af_ZA".

u> One way to solve the problem would be to associate a default region for
u> locales for which the canonical name consists only of the language part.
u> This would require to extend wxLanguageInfo, so that for example the
u> canonical name "af" could be associated with "af_ZA". Opinions?

I don't really see much value in separating the default region like this,
but we could do this.

u> BTW, I have the impression that the algorithm for locating the translation
u> subdirectories fails when the canonical name contains a script name.
u> wxUILocale returns a canonical name which uses the minus sign as a
u> separator. The separator is replaced by an underscore, but the current
u> logic for the subdirectory names uses the at sign (@) to append the script
u> name.

I may have broken this with my recent changes then... Ideally I think we
should look under the directory with the script name (separated by "@") and
then under the directory without script name at all.

If you can fix this as part of your changes, it would be great, otherwise
could you please open a bug about it so that I don't forget to do it later?

TIA,
VZ

[utelle]

VZ schrieb am Samstag, 6. November 2021 um 17:13:03 UTC+1:

u> [...] Under macOS

u> I used also the command "locale -a" resulting in a list with 203 entries -

The list returned by [NSLocale availableLocaleIdentifiers] is much larger,
because POSIX APIs are just a redheaded stepchild under macOS.

Could you provide the list of locale identifiers returned by [NSLocale availableLocaleIdentifiers]? Not urgent, but I'd like to compare the list with that produced by Windows.

 

u> 1. The identifiers wxLANGUAGE_* [...]

I'm afraid we'd have to remain compatible with the existing constants,
just because I'm not sure it's safe to change their meaning and there
doesn't seem to be any good reason to take the risk of doing it if it can
be avoided (and I hope it can, if only by maintaining a small list of
exceptions).

I fully understand that existing wxLANGUAGE constants should point to the same entry (that is, the same canonical name) as before. This will be a bit tricky, because I'd like to automate the process of generating the list as much as possible.

The file langtabl.txt itself already has a few strange entries. Examples:

1. The constants wxLANGUAGE_AZERI, wxLANGUAGE_AZERI_CYRILLIC, wxLANGUAGE_AZERI_LATIN use all the same canonical name, "az", although for the last 2 the script should be appended; same thing with wxLANGUAGE_UZBEK;
   
2. wxLANGUAGE_CHINESE (without script or region) points to the canonical name "zh_TW" (Taiwan), while the canonical name "zh" (as returned by Windows) is linked to mainland China, that is "zh_CN".
3. wxLANGUAGE_ENGLISH points to canonical name "en_GB", while Windows ties it to "en_US"
4. wxLANGUAGE_PORTUGUESE points to canonical name "pt_PT", while Windows ties it to "pt_BR"
   

IMHO at least (1) should be fixed. Not sure about the other 3.

We'd definitely like to avoid breaking the existing programs expecting
Afrikaans translations to be found under "af_ZA".

Yes, therefore the catalog loading algorithm should check location "af_ZA" first, if the canonical name is simply "af". To accomplish this, I would add an internal field for the associated canonical name which includes the default region.

u> This would require to extend wxLanguageInfo, so that for example the
u> canonical name "af" could be associated with "af_ZA". Opinions?

I don't really see much value in separating the default region like this,
but we could do this.

I'm not happy about this either, but the new locale list I extracted from Windows contains regional variations for many of the current entries where at the moment no regional variations are listed at all. It would be difficult to automate the decision, when to keep the non-regional entry for compatibility and when not (also for compatibility). IMHO it is easier to add the default canonical name which includes the default region, like "es_ES" for "es".

u> BTW, I have the impression that the algorithm for locating the translation
u> subdirectories fails when the canonical name contains a script name.
u> wxUILocale returns a canonical name which uses the minus sign as a
u> separator. The separator is replaced by an underscore, but the current
u> logic for the subdirectory names uses the at sign (@) to append the script
u> name.

I may have broken this with my recent changes then...

As far as I can tell the previous implementation didn't handle this case properly either. Most likely no one detected this flaw, because there are only 3 entries in the current list which have script modifiers resp region modifiers (and people using those languages probably always used the full canonical name, in which case no other location is tested). This will change with the updated list, because there are more entries with script modifiers and regional variations, where checking locations without region, but with script, will matter.

Ideally I think we
should look under the directory with the script name (separated by "@") and
then under the directory without script name at all.

If you can fix this as part of your changes, it would be great, otherwise
could you please open a bug about it so that I don't forget to do it later?

 

I think I can fix this in the course of my activities. Are there planned dates for the next wxWidgets releases? I guess it would be nice to include the changes in version 3.2.0, if I manage to get ready in time.

Regards,

Ulrich

[utelle]

VZ schrieb am Samstag, 6. November 2021 um 17:13:03 UTC+1:

I'm afraid we'd have to remain compatible with the existing constants,
just because I'm not sure it's safe to change their meaning and there
doesn't seem to be any good reason to take the risk of doing it if it can
be avoided (and I hope it can, if only by maintaining a small list of
exceptions).

Ok, I tried a different approach which keeps most of the old existing wxLANGUAGE_ identifiers.

There are a few entries in the current version of langtabl.txt which need small adjustments. For example, the entries for wxLANGUAGE_KONKANI and wxLANGUAGE_MANIPURI have no canonical names, but in the list retrieved from Windows they have. The new list will only have entries WITH canonical names.

Then there are a few entries with language or country codes which are not assigned in ISO 639 resp ISO 3166, for example the wxLANGUAGE_AYMARA ("ay"), wxLANGUAGE_BIHARI ("bh"), wxLANGUAGE_BISLAMA ("bi"), wxLANGUAGE_ENGLISH_CARIBBEAN ("en_CB" - "CB" not assigned), wxLANGUAGE_SERBIAN_YU ("sr_YU" - "YU"=Yugoslavia not assigned anymore) ... and roughly 10 more. I will keep the entries on the list for compatibility, although I doubt that any platform offers a locale for them.

For a few languages it will be necessary to adjust the canonical names to include the script, namely wxLANGUAGE_AZERI ("az") and wxLANGUAGE_UZBEK ("uz"). It makes no sense to use the same canonical name for different scripts.

Additionally, I intend to introduce a few aliases (old name => new name):

wxLANGUAGE_AZERI*  = wxLANGUAGE_AZERBAIJANI*
wxLANGUAGE_BENGALI*  = wxLANGUAGE_BANGLA*
wxLANGUAGE_BHUTANI  = wxLANGUAGE_DZONGKHA
wxLANGUAGE_KERNEWEK  = wxLANGUAGE_CORNISH
wxLANGUAGE_ORIYA  = wxLANGUAGE_ODIA
wxLANGUAGE_CHINESE_MACAU  = wxLANGUAGE_CHINESE_MACAO

One point, I'd like to mention is that the new list does not use the symbolic constants LANG_* and SUBLANG_* at the moment, but their hexadecimal values. IMHO this doesn't matter, since the use of LCIDs is deprecated under Windows anyway, and at runtime you also see only the numerical value. And, for a large number of entries no symbolic constants are defined.

If there are no objections I will adjust the generator script and the wx source code accordingly to work with the new list.

Regards,

Ulrich

[Vadim Zeitlin]

On Mon, 8 Nov 2021 03:57:57 -0800 (PST) utelle wrote:

u> Ok, I tried a different approach which keeps most of the old existing
u> wxLANGUAGE_ identifiers.

Great, thanks!

u> One point, I'd like to mention is that the new list does not use the
u> symbolic constants LANG_* and SUBLANG_* at the moment, but their
u> hexadecimal values.

This indeed shouldn't matter much for the generated files.

u> If there are no objections I will adjust the generator script and the wx
u> source code accordingly to work with the new list.

I don't see anything objectionable here, thanks in advance!
VZ

[utelle]

A short status report:

1. Update the file langtabl.txt in misc/languages;
   

1. Completed.
   
   

1. Implement the language database in wxUILocale and redirect the static wxLocale methods to the counterparts in wxUILocale.
   

1. Done.
   However,  I have a question about where to place the code. Up to now the implementation is located in source file intl.cpp. Should the new implementation be moved to uilocale.cpp?
   
   

1. Add wxUILocale method GetLocalizedName to retrieve language/country... names in English and in localized language;
   

1. Done (for Windows and Linux/Unix), Open (for macOS).
   Regarding macOS I would need a bit of guidance, how to call the relevant system functions, because I don't have a mac system for testing and because I'm not familiar with the Objective-C syntax.
   I started with 3 variants for enum wxLocaleForm:
   - wxLOCALE_FORM_NATIVE (for example, "Deutsch (Deutschland)" for locale de_DE)
   - wxLOCALE_FORM_LOCALIZED (for example, "French (France)" for locale fr_FR with currently active locale en_US, "Französisch (Frankreich)" for de_DE)
   - wxLOCALE_FORM_ENGLISH (for example, "German (Germany)" for locale de_DE)
   The variant wxLOCALE_FORM_LOCALIZE would allow to retrieve translations for a locale name based on the currently active locale. However, only Windows and macOS support that; Linux/Unix do not support this feature.Therefore it is maybe better to remove this variant and leave it to the application to handle translations.
   
   

1. Add method to wxUILocale to retrieve the layout direction of a locale
   

1. Open.
   Windows and macOS support to retrieve this information from the locale; however, Linux/Unix does not. Currently, this information is retrieved from the internal language database. Should the locale be used, if possible, with the language database as a fallback? Or should the language database always be used as before?
   
   

1. Change src/common/appcmn.cpp to query the layout direction via wxUILocale
   

1. Open.
   
   

1. Enhance the internat sample to allow changing the translation language
   

1. Open.
   

Regards,

Ulrich

[utelle]

VZ schrieb am Samstag, 30. Oktober 2021 um 15:03:34 UTC+2:

[...] I'm still

not sure if there is something more fundamentally missing in wxUILocale,
which would need to be fixed before 3.2.0. Please try using wxUILocale and
let us know if we have to change anything.

While working on updating the language database I came across one issue that I think should be addressed before 3.2.0.

The language database as well as the class wxLocale use canonical locale names in the form "<language>_<country>@modifier".  wxUILocaleImpl provides a method  CreateForLanguage (used by method UseLanguage), which interprets this form of canonical locale names. However, there is no way back. That is, wxUILocale has just a platform-dependent internal name. This makes it really difficult to use the language database as a fallback. IMHO wxUILocale should provide a platform independent form to identify the locale, for example by constructing a wxLocaleIdent that can be queried.

As an alternative a tag using the subset of BCP 47 as understood by wxLocaleIdent could be added to each language entry in the language database.

Regards,

Ulrich

[Vadim Zeitlin]

On Fri, 19 Nov 2021 01:54:37 -0800 (PST) utelle wrote:

u> A short status report:
u>
u> 1. Update the file langtabl.txt in misc/languages;
u> *Completed.*

Great!

u> 2. Implement the language database in wxUILocale and redirect the static
u> wxLocale methods to the counterparts in wxUILocale.
u> Done.
u> However, I have a question about where to place the code. Up to now the
u> implementation is located in source file *intl.cpp*. Should the new
u> implementation be moved to *uilocale.cpp*?

I don't think it matters that much, but, in principle, wxUILocale methods
should be in uilocale.cpp. But if they contain mostly the same code that
was in wxLocale methods before, it might be better to leave them in
intl.cpp to be able to use "git blame" etc more efficiently.

u> 3. Add wxUILocale method GetLocalizedName to retrieve
u> language/country... names in English and in localized language;
u> *Done (*for Windows and Linux/Unix*), Open (*for macOS*).*
u> Regarding macOS I would need a bit of guidance, how to call the relevant
u> system functions, because I don't have a mac system for testing

We have a Mac used for the CI builds which can also be used for other wx
development tasks, please ask Stefan if you'd like to get an account on it.

u> I'm not familiar with the Objective-C syntax.

I'm not going to try to explain it because I'm pretty bad with Objective-C
myself, but it's relatively simple (compared to some other languages, at
least) and personally I often find it helpful to look at the existing code
to find out how to do something.

Also, if you have any concrete questions, please try asking them here and
hopefully someone knowledgeable about Objective-C (or, in the worst case,
me) would answer them.

u> I started with 3 variants for enum wxLocaleForm:
u> - wxLOCALE_FORM_NATIVE (for example, "Deutsch (Deutschland)" for locale
u> de_DE)
u> - wxLOCALE_FORM_LOCALIZED (for example, "French (France)" for locale
u> fr_FR with currently active locale en_US, "Französisch (Frankreich)" for
u> de_DE)
u> - wxLOCALE_FORM_ENGLISH (for example, "German (Germany)" for locale
u> de_DE)
u> The variant wxLOCALE_FORM_LOCALIZE would allow to retrieve translations
u> for a locale name based on the currently active locale. However, only
u> Windows and macOS support that; Linux/Unix do not support this
u> feature.Therefore it is maybe better to remove this variant and leave it to
u> the application to handle translations.

In principle, it's ok to have this in the API and just return empty string
under Linux for it. OTOH I'm not really sure when is it going to be useful,
so omitting would be fine with me too.

u> 4. Add method to wxUILocale to retrieve the layout direction of a locale
u>
u> *Open. *Windows and macOS support to retrieve this information from the
u> locale; however, Linux/Unix does not. Currently, this information is
u> retrieved from the internal language database*. *Should the locale be
u> used, if possible, with the language database as a fallback? Or should the
u> language database always be used as before?

Are there any languages/locales for which the two approaches yield
different results?

u> 5. Change src/common/appcmn.cpp to query the layout direction via wxUILocale
u>
u> *Open. *
u> 6. Enhance the internat sample to allow changing the translation language
u> *Open.*

Thanks again for the update but I'd also like to say that you shouldn't
hesitate to create even partial/in-progress PRs with your changes,
especially if you think that the parts that are already done won't change
[much] any more, as it could be simpler to review the changes in parts than
the entire final PR, which risks to be huge, at once at the end.

Good luck with finishing this!
VZ

[Vadim Zeitlin]

On Fri, 19 Nov 2021 06:01:39 -0800 (PST) utelle wrote:

u> While working on updating the language database I came across one issue
u> that I think should be addressed before 3.2.0.
u>
u> The language database as well as the class wxLocale use canonical locale
u> names in the form "<language>_<country>@modifier". wxUILocaleImpl provides
u> a method CreateForLanguage (used by method UseLanguage), which interprets
u> this form of canonical locale names. However, there is no way back. That
u> is, wxUILocale has just a platform-dependent internal name. This makes it
u> really difficult to use the language database as a fallback.

Do you mean inside wxUILocale code or in the application?

u> IMHO wxUILocale should provide a platform independent form to identify
u> the locale, for example by constructing a wxLocaleIdent that can be
u> queried.

I have no objections to doing it, I only didn't do this myself because I
try to not add things to the API prematurely, i.e. before I have a use case
for them, and I didn't have one in my own program.

u> As an alternative a tag using the subset of BCP 47 as understood by
u> wxLocaleIdent could be added to each language entry in the language
u> database.

I'm not sure if we can retrieve the _same_ wxLocaleIdent for the same
locale under all platforms using the native API. I hope that this is the
case, but if not, then we would perhaps need to do the above for
portability.

Regards,
VZ

[utelle]

VZ schrieb am Freitag, 19. November 2021 um 16:28:39 UTC+1:

u> While working on updating the language database I came across one issue
u> that I think should be addressed before 3.2.0.
u>
u> The language database as well as the class wxLocale use canonical locale
u> names in the form "<language>_<country>@modifier". wxUILocaleImpl provides
u> a method CreateForLanguage (used by method UseLanguage), which interprets
u> this form of canonical locale names. However, there is no way back. That
u> is, wxUILocale has just a platform-dependent internal name. This makes it
u> really difficult to use the language database as a fallback.

Do you mean inside wxUILocale code or in the application?

Sorry for being too unspecific. I meant within wxUILocale. For example, where supported we can retrieve the layout direction of a locale from the OS (Windows, macOS), but for Linux/Unix we need to use the language database. However, to be able to do that, we need to find the matching entry in the language database. Actually, only the canonical name is suited to identify an entry, but calling wxUILocale::GetName doesn't help because it delivers a platform specific name.

Additionally, I think we need a way to restore a specific locale. For example, an application may want to save a locale identifier, so that the same locale can be restored in the next session. Currently, the application has to take care of remembering the settings used to instantiate a locale on its own, because a wxUILocale can't be queried for a usable identifier. Calling wxUILocale::GetName doesn't work, because the returned value is platform specific and there is no method to create a wxLocaleIdent from it for use in a wxUILocale constructor. Method wxUILocale::FromTag can't be used either, because it expects a tag in BCP 47 format.

u> IMHO wxUILocale should provide a platform independent form to identify
u> the locale, for example by constructing a wxLocaleIdent that can be
u> queried.

I have no objections to doing it, I only didn't do this myself because I
try to not add things to the API prematurely, i.e. before I have a use case
for them, and I didn't have one in my own program.

The big question is how this should be done. The answer depends on whether we need to support all edge cases or whether support for the typical use cases is sufficient. wxLocaleIdent usually assumes that language, region, and script are sufficient to describe a locale. However, under Linux/Unix one can have additional components (charset and modifier). If I'm not mistaken wxLocaleIdent has no method to handle a platform-dependent tag like "en_GB.utf8@euro" which could be the tag returned by GetName. Under Windows there are also special cases like "hu-HU_technl"(where "_technl" designates a special sort order) or "en-US-x-fabricam"(where "-x-fabricam" designates a user-defined locale).

Nevertheless, I think a wxUILocale instance should be able to always provide at least language, script, and region information (script and region only if appropriate).

 

u> As an alternative a tag using the subset of BCP 47 as understood by
u> wxLocaleIdent could be added to each language entry in the language
u> database.

I'm not sure if we can retrieve the _same_ wxLocaleIdent for the same
locale under all platforms using the native API. I hope that this is the
case, but if not, then we would perhaps need to do the above for
portability.

Given a certain wxUILocale instance there is currently no easy way to identify an entry in the language database that corresponds to the wxUILocale instance. IMHO lookup would be easier if tag conversion (tag in language database to/from BCP 47 tag) could be avoided.

Regards,

Ulrich

[utelle]

VZ schrieb am Freitag, 19. November 2021 um 16:25:50 UTC+1:

u> However, I have a question about where to place the code. Up to now the
u> implementation is located in source file *intl.cpp*. Should the new
u> implementation be moved to *uilocale.cpp*?

I don't think it matters that much, but, in principle, wxUILocale methods
should be in uilocale.cpp. But if they contain mostly the same code that
was in wxLocale methods before, it might be better to leave them in
intl.cpp to be able to use "git blame" etc more efficiently.

At the moment, the method bodies are more or less identical. So, for now I will keep them in intl.cpp. The code can be moved to uilocale.cpp later, if that is preferred in the long run.

u> Regarding macOS I would need a bit of guidance, how to call the relevant
u> system functions, because I don't have a mac system for testing

We have a Mac used for the CI builds which can also be used for other wx
development tasks, please ask Stefan if you'd like to get an account on it.

Thanks for the information. I'll get in touch with Stefan, if necessary.

 

u> I'm not familiar with the Objective-C syntax.

I'm not going to try to explain it because I'm pretty bad with Objective-C
myself, but it's relatively simple (compared to some other languages, at
least) and personally I often find it helpful to look at the existing code
to find out how to do something.

I will try to prepare the code as much as I can. Maybe then someone (you or Stefan) can jump in to actually activate the code in macOS build.

u> The variant wxLOCALE_FORM_LOCALIZE would allow to retrieve translations
u> for a locale name based on the currently active locale. However, only
u> Windows and macOS support that; Linux/Unix do not support this
u> feature.Therefore it is maybe better to remove this variant and leave it to
u> the application to handle translations.

In principle, it's ok to have this in the API and just return empty string
under Linux for it. OTOH I'm not really sure when is it going to be useful,
so omitting would be fine with me too.

The only use case which comes to mind is when presenting a list of available locales/translations to the user. It is a nice-to-have to be able to show the locale names in the language of the current locale. However, this can be accomplished also via English names together with translation files.

Since the method will not be available for all major platforms, it may be better to drop it.

 

u> 4. Add method to wxUILocale to retrieve the layout direction of a locale
u>
u> *Open. *Windows and macOS support to retrieve this information from the
u> locale; however, Linux/Unix does not. Currently, this information is
u> retrieved from the internal language database*. *Should the locale be
u> used, if possible, with the language database as a fallback? Or should the
u> language database always be used as before?

Are there any languages/locales for which the two approaches yield
different results?

Hopefully not. For Windows the locale can be queried if the Windows version is Windows 7 or higher. That is, for versions below Windos 7 a lookup via the language database would be necessary. AFAICR currently the LCID is used to lookup the entry in the language database (a method that should be avoided in the future). For Linux/Unix the language database is used to retrieve the layout direction.

 

Thanks again for the update but I'd also like to say that you shouldn't
hesitate to create even partial/in-progress PRs with your changes,
especially if you think that the parts that are already done won't change
[much] any more, as it could be simpler to review the changes in parts than
the entire final PR, which risks to be huge, at once at the end.

The update of the language database could be prepared as a separate PR as the updated version could be used almost without changing any code (except adding methods to access the additional data fields in the language database.

However, the next steps would have the updated language database as a pre-requisite. Not sure, how to handle that properly with different PRs.

Regards,

Ulrich

[Vadim Zeitlin]

On Fri, 19 Nov 2021 13:28:32 -0800 (PST) utelle wrote:

u> VZ schrieb am Freitag, 19. November 2021 um 16:25:50 UTC+1:
u>

u> > u> However, I have a question about where to place the code. Up to now the
u> > u> implementation is located in source file *intl.cpp*. Should the new
u> > u> implementation be moved to *uilocale.cpp*?
u> >
u> > I don't think it matters that much, but, in principle, wxUILocale methods
u> > should be in uilocale.cpp. But if they contain mostly the same code that
u> > was in wxLocale methods before, it might be better to leave them in
u> > intl.cpp to be able to use "git blame" etc more efficiently.
u>
u> At the moment, the method bodies are more or less identical. So, for now I
u> will keep them in *intl.cpp*. The code can be moved to *uilocale.cpp*
u> later, if that is preferred in the long run.

Looks like a good plan.

u> > u> The variant wxLOCALE_FORM_LOCALIZE would allow to retrieve translations
u> > u> for a locale name based on the currently active locale.
[...]
u> Since the method will not be available for all major platforms, it may be
u> better to drop it.

I'd be perfectly fine with it, it would always be possible to add it later
if it's really needed.

u> > u> 4. Add method to wxUILocale to retrieve the layout direction of a locale

u> > u>
u> > u> *Open. *Windows and macOS support to retrieve this information from the

u> > u> locale; however, Linux/Unix does not. Currently, this information is
u> > u> retrieved from the internal language database*. *Should the locale be
u> > u> used, if possible, with the language database as a fallback? Or should
u> > u> the language database always be used as before?
u> >
u> > Are there any languages/locales for which the two approaches yield
u> > different results?
u>
u> Hopefully not.

In this case I don't have any real preference, so you should just do
whichever is more convenient/simpler.

u> For Windows the locale can be queried if the Windows version is Windows
u> 7 or higher. That is, for versions below Windos 7 a lookup via the
u> language database would be necessary. AFAICR currently the LCID is used
u> to lookup the entry in the language database (a method that should be
u> avoided in the future).

It's really unfortunate that we still keep support for XP in 3.2 (and more
time passes, less useful it becomes), things will definitely become simpler
once it's released and we drop support for it...

u> For Linux/Unix the language database is used to retrieve the layout
u> direction.

FWIW GTK gets the default text direction from translations, see
gtk_get_locale_direction(). But I don't think this has any advantages
compared to using the language database.

u> The update of the language database could be prepared as a separate PR as
u> the updated version could be used almost without changing any code (except
u> adding methods to access the additional data fields in the language
u> database.
u>
u> However, the next steps would have the updated language database as a
u> pre-requisite. Not sure, how to handle that properly with different PRs.

Please make the first PR with the database update, it should hopefully be
merged quickly and then you could make the other PR after rebasing your
branch on the latest master (which will have already integrated the
database-related changes).

TIA!
VZ

[Vadim Zeitlin]

On Fri, 19 Nov 2021 13:03:53 -0800 (PST) utelle wrote:

u> VZ schrieb am Freitag, 19. November 2021 um 16:28:39 UTC+1:
u>

u> > u> While working on updating the language database I came across one issue
u> > u> that I think should be addressed before 3.2.0.

u> > u>
u> > u> The language database as well as the class wxLocale use canonical

u> > u> locale names in the form "<language>_<country>@modifier".
u> > u> wxUILocaleImpl provides a method CreateForLanguage (used by method
u> > u> UseLanguage), which interprets this form of canonical locale names.
u> > u> However, there is no way back. That is, wxUILocale has just a
u> > u> platform-dependent internal name. This makes it really difficult to
u> > u> use the language database as a fallback.
u> >
u> > Do you mean inside wxUILocale code or in the application?
u>
u> Sorry for being too unspecific. I meant within wxUILocale. For example,
u> where supported we can retrieve the layout direction of a locale from the
u> OS (Windows, macOS), but for Linux/Unix we need to use the language
u> database. However, to be able to do that, we need to find the matching
u> entry in the language database. Actually, only the canonical name is suited
u> to identify an entry, but calling wxUILocale::GetName doesn't help because
u> it delivers a platform specific name.

Well, by a happy coincidence, under generic Unix system this would be
exactly the canonical name, so GetName() still could be used. But I agree
that it isn't pretty.

u> Additionally, I think we need a way to restore a specific locale. For
u> example, an application may want to save a locale identifier, so that the
u> same locale can be restored in the next session.

I can't really imagine a realistic use case for exactly this situation
("Would you like to sort strings using the rules for the language your
system used the last time you ran this program?" would be an extremely
weird question to ask), but I guess we could want to save the locale being
used in some file, that could later be used on another machine, and in this
case it would indeed be useful to have a way to get a portable locale
identifier.


u> The big question is how this should be done.

Would it be simpler to have only a static GetCurrentIdent() function,
which would return wxLocaleIdent corresponding to the current locale,
instead of non-static GetIdent() that would have to work for any locale?

BTW, I don't know if we want Get[Current]Ident() or Get[Current]Tag()
neither, do you?

u> The answer depends on whether
u> we need to support all edge cases or whether support for the typical use
u> cases is sufficient. wxLocaleIdent usually assumes that language, region,
u> and script are sufficient to describe a locale. However, under Linux/Unix
u> one can have additional components (charset and modifier). If I'm not
u> mistaken wxLocaleIdent has no method to handle a platform-dependent tag
u> like "en_GB.utf8@euro" which could be the tag returned by GetName.

It does have some support for this, it calls it "the modifier".

u> Under Windows there are also special cases like "hu-HU_technl"(where
u> "_technl" designates a special sort order) or "en-US-x-fabricam"(where
u> "-x-fabricam" designates a user-defined locale).

Such locales can't be represented with wxLocaleIdent currently and so are
impossible to construct with the current API. But we could still return
these strings if we had (only) static GetCurrentTag() function.

u> Nevertheless, I think a wxUILocale instance should be able to always
u> provide at least language, script, and region information (script and
u> region only if appropriate).

So it looks like you'd like to have a (non-static) GetIdent()?

Personally I think having GetCurrentTag() might be the best solution:

- It should be the simplest one to implement (as any other one would
require implementing this anyhow).
- It will allow to construct wxLocaleIdent from the returned string if
necessary, but is capable of representing even the locales not currently
supported by wxLocaleIdent, such as those above.
- It should be sufficient for the scenarios described above, as they only
need the identifier for the current locale, and you should already have
either the tag or the identifier for any locale objects you create (as
otherwise how would you create them).

u> Given a certain wxUILocale instance there is currently no easy way to
u> identify an entry in the language database that corresponds to the
u> wxUILocale instance. IMHO lookup would be easier if tag conversion (tag in
u> language database to/from BCP 47 tag) could be avoided.

If we had GetCurrentTag(), we could use wxLocaleIdent::FromTag() to
construct wxLocaleIdent from it and then use GetLanguage(), GetRegion() and
GetModifier() to construct its canonical name (maybe we should add a new
function for this to include/wx/private/uilocale.h).

Regards,
VZ

[utelle]

VZ schrieb am Samstag, 20. November 2021 um 15:01:29 UTC+1:

u> > u> 4. Add method to wxUILocale to retrieve the layout direction of a locale
u> > u>
u> > u> *Open. *Windows and macOS support to retrieve this information from the
u> > u> locale; however, Linux/Unix does not. Currently, this information is
u> > u> retrieved from the internal language database*. *Should the locale be
u> > u> used, if possible, with the language database as a fallback? Or should
u> > u> the language database always be used as before?
u> >
u> > Are there any languages/locales for which the two approaches yield
u> > different results?
u>
u> Hopefully not.

In this case I don't have any real preference, so you should just do
whichever is more convenient/simpler.

IMHO we should query the locale where possible. This is more efficient than looking up the matching entry in the language database, although the language database lookup needs to be implemented, too - at least for Linux/Unix.

 

u> For Windows the locale can be queried if the Windows version is Windows
u> 7 or higher. That is, for versions below Windos 7 a lookup via the
u> language database would be necessary. AFAICR currently the LCID is used
u> to lookup the entry in the language database (a method that should be
u> avoided in the future).

It's really unfortunate that we still keep support for XP in 3.2 (and more
time passes, less useful it becomes), things will definitely become simpler
once it's released and we drop support for it...

Is it known how many wxWidgets users still run XP and really need the latest wx version?

Nevertheless, LCID shouldn't be used for lookup in the language database any longer, at least for all Windows versions >= Windows 7. Many new entries in the updated language database do not have a usable LCID.

 

u> For Linux/Unix the language database is used to retrieve the layout
u> direction.

FWIW GTK gets the default text direction from translations, see
gtk_get_locale_direction(). But I don't think this has any advantages
compared to using the language database.

So GTK must have its own way of determining the layout direction, because the locale data provided via glibc don't include this information.

 

u> The update of the language database could be prepared as a separate PR as
u> the updated version could be used almost without changing any code (except
u> adding methods to access the additional data fields in the language
u> database.

Please make the first PR with the database update, it should hopefully be
merged quickly and then you could make the other PR after rebasing your
branch on the latest master (which will have already integrated the
database-related changes).

Ok.  I'll prepare a PR with the language database update and some small adjustments of the related code.

I will add BCP47-like tags to each entry as this will allow to use the wxLocaleIdent tags for lookup without requiring to reconstruct the current representation of canonical names. Since wxLocaleIdent expects tags in BCP47 representation, this seesm to be the most logical approach.

Regards,

Ulrich

[utelle]

VZ schrieb am Samstag, 20. November 2021 um 15:21:16 UTC+1:

u> I meant within wxUILocale. For example,

u> where supported we can retrieve the layout direction of a locale from the
u> OS (Windows, macOS), but for Linux/Unix we need to use the language
u> database. However, to be able to do that, we need to find the matching
u> entry in the language database. Actually, only the canonical name is suited
u> to identify an entry, but calling wxUILocale::GetName doesn't help because
u> it delivers a platform specific name.

Well, by a happy coincidence, under generic Unix system this would be
exactly the canonical name, so GetName() still could be used. But I agree
that it isn't pretty.

I would prefer that wxUILocale can be queried for a wxLocaleIdent and/or BCP47-like tag.

u> Additionally, I think we need a way to restore a specific locale. For
u> example, an application may want to save a locale identifier, so that the
u> same locale can be restored in the next session.

I can't really imagine a realistic use case for exactly this situation
("Would you like to sort strings using the rules for the language your
system used the last time you ran this program?" would be an extremely
weird question to ask), but I guess we could want to save the locale being
used in some file, that could later be used on another machine, and in this
case it would indeed be useful to have a way to get a portable locale
identifier.

IMHO portable locale identifiers make perfectly sense for a portable framework like wxWidgets. Why should a user want to know the platform-dependent locale name, if he/she is not able to use it anywhere except for maybe in error messages or the like?

 

Would it be simpler to have only a static GetCurrentIdent() function,
which would return wxLocaleIdent corresponding to the current locale,
instead of non-static GetIdent() that would have to work for any locale?

I would prefer a non-static GetIdent().

BTW, I don't know if we want Get[Current]Ident() or Get[Current]Tag()
neither, do you?

As said I'd like to have non-static GetIdent() and/or GetTag(). Additional static methods GetCurrentIdent() and/or GetCurrentTag() are then not really necessary, but may be convenient.

 

u> However, under Linux/Unix

u> one can have additional components (charset and modifier). If I'm not
u> mistaken wxLocaleIdent has no method to handle a platform-dependent tag
u> like "en_GB.utf8@euro" which could be the tag returned by GetName.

It does have some support for this, it calls it "the modifier".

Yes, but you have to split platform-dependent tag yourself into the relevant parts, before you can set the corresponding components of wxLocaleIdent.

u> Under Windows there are also special cases like "hu-HU_technl"(where
u> "_technl" designates a special sort order) or "en-US-x-fabricam"(where
u> "-x-fabricam" designates a user-defined locale).

Such locales can't be represented with wxLocaleIdent currently and so are
impossible to construct with the current API. But we could still return
these strings if we had (only) static GetCurrentTag() function.

Personally, I never had the need for a user-defined locale, but in principle, it would be possible to use the charset and modifier components under Windows to represent such locales. For sort order I would use the charset attribute, and for user-defined locales the modifier attribute.

I don't know whether it's worth the effort, but it is certainly managable in this way.

 

u> Nevertheless, I think a wxUILocale instance should be able to always
u> provide at least language, script, and region information (script and
u> region only if appropriate).

So it looks like you'd like to have a (non-static) GetIdent()?

Yes. :-)

Personally I think having GetCurrentTag() might be the best solution:

- It should be the simplest one to implement (as any other one would
require implementing this anyhow).
- It will allow to construct wxLocaleIdent from the returned string if
necessary, but is capable of representing even the locales not currently
supported by wxLocaleIdent, such as those above.
- It should be sufficient for the scenarios described above, as they only
need the identifier for the current locale, and you should already have
either the tag or the identifier for any locale objects you create (as
otherwise how would you create them).

If I implement a method to construct a wxLocaleIdent, this method can be used for any locale instance, not only the current one. So, for me it's not clear why only implementing GetCurrentIdent / GetCurrentTag would be simpler. However, I may not see the forest for the trees.

For the default system locale you don't need a tag or identifier to create it. However, you may want to be able to reconstruct the very locale.

 

u> Given a certain wxUILocale instance there is currently no easy way to
u> identify an entry in the language database that corresponds to the
u> wxUILocale instance. IMHO lookup would be easier if tag conversion (tag in
u> language database to/from BCP 47 tag) could be avoided.

If we had GetCurrentTag(), we could use wxLocaleIdent::FromTag() to
construct wxLocaleIdent from it and then use GetLanguage(), GetRegion() and
GetModifier() to construct its canonical name (maybe we should add a new
function for this to include/wx/private/uilocale.h).

 

Yes, that may be a reasonable approach, since the canonical name is not only used in the language database, but also for the directory names where translation message catalogs are located. Please correct me, if the latter is not the case on all platforms.

Regards,

Ulrich

[Vadim Zeitlin]

On Sat, 20 Nov 2021 07:38:37 -0800 (PST) utelle wrote:

u> VZ schrieb am Samstag, 20. November 2021 um 15:01:29 UTC+1:
u>

u> > u> > u> 4. Add method to wxUILocale to retrieve the layout direction of a
u> > u> > u> locale
[...]
u> IMHO we should query the locale where possible. This is more efficient than
u> looking up the matching entry in the language database

To be honest, I'm not sure if it's really more efficient, but I also don't
think efficiency really matters here. The only place where it might matter
would be the drawing code and there you'd use the layout direction of the
window and not query the locale directly anyhow. And you could also cache
it if it's really a problem. But you'd only query the layout direction of
the (current) locale at most once on the program startup, so who cares how
long does it take? As long as you don't send a (snail) mail to the
headquarters of some standard organization to get the answer, it should be
fine.

So I'd just do whichever is simpler/faster.

u> Is it known how many wxWidgets users still run XP and really need the
u> latest wx version?

No, but I feel like we're not going to learn about it any time soon and
people currently using 3.1.x could be surprised to not have support for it
any longer in 3.2.0. Of course, still supporting it in 3.2 made much more
sense when I thought it would be released in 2016 and not 2022, but...

u> So GTK must have its own way of determining the layout direction, because
u> the locale data provided via glibc don't include this information.

Yes, they rely on the their translations. I.e. in principle you could have
German RTL translations for GTK too.

u> Ok. I'll prepare a PR with the language database update and some small
u> adjustments of the related code.

TIA!
VZ

[Vadim Zeitlin]

On Sat, 20 Nov 2021 08:25:07 -0800 (PST) utelle wrote:

u> As said I'd like to have non-static GetIdent() and/or GetTag(). Additional
u> static methods GetCurrentIdent() and/or GetCurrentTag() are then not really
u> necessary, but may be convenient.

If we have non-static functions, we don't need the static ones. My idea
was to have _only_ the static ones if it could simplify your life. If it
doesn't, please forget about them.

u> If I implement a method to construct a wxLocaleIdent, this method can be
u> used for any locale instance, not only the current one. So, for me it's not
u> clear why only implementing GetCurrentIdent / GetCurrentTag would be
u> simpler.

I think that there are functions for obtaining the tag/identifier of the
current locale under all platforms, but I'm not sure about the rest. If
it's equally simple to do, sure, let's do it like this, but I thought that
we might have different paths for the current locale (for which we'd just
query the OS) and the other ones (for which we might have to use the tag it
had been initially created from, if the OS doesn't provide the API needed
to get it). Again, if this is not the case and you can easily get this
under all platforms, please forget about the static-only functions idea.

Regards,
VZ

[utelle]

VZ schrieb am Samstag, 20. November 2021 um 18:03:47 UTC+1:

u> 4. Add method to wxUILocale to retrieve the layout direction of a locale

[...]
u> IMHO we should query the locale where possible. This is more efficient than
u> looking up the matching entry in the language database

To be honest, I'm not sure if it's really more efficient, but I also don't
think efficiency really matters here.

Since the language database is currently searched sequentially, it will definitely be more efficient to call just one system function. However, you are right that it doesn't really matter, because this value will typically be queried only once on initialization of the application.

 

u> Is it known how many wxWidgets users still run XP and really need the
u> latest wx version?

No, but I feel like we're not going to learn about it any time soon and
people currently using 3.1.x could be surprised to not have support for it
any longer in 3.2.0. Of course, still supporting it in 3.2 made much more
sense when I thought it would be released in 2016 and not 2022, but...

Not long ago I read an article which mentioned that Windows XP is still in use in many technical devices like elevators (for example at the new Berlin airport) and even the British navy uses it in submarines. So maybe still supporting XP is not such a bad idea after all.

 

Regards,

Ulrich

[utelle]

VZ schrieb am Samstag, 20. November 2021 um 18:06:57 UTC+1:

u> ... non-static GetIdent() and/or GetTag() ...
u> static methods GetCurrentIdent() and/or GetCurrentTag() ...

If we have non-static functions, we don't need the static ones. My idea
was to have _only_ the static ones if it could simplify your life. If it
doesn't, please forget about them.

I doubt that having static functions only would simplify the implementation much. However, I will keep the idea in my mind.

 

I think that there are functions for obtaining the tag/identifier of the
current locale under all platforms, but I'm not sure about the rest.

For Windows (Vista and above) a locale can be queried for the identifier, and it is immediately in the form we need.  In fact, it is the value returned by GetName(). For XP GetName() returns the language name in English - which is only a partial description of the locale. I think for XP it would be better to query the LCID value and then look it up in the language database.

For macOS GetName() returns an identifier which is almost in the form we need, but it could also be something like "zh-Hant_HK_POSIX@collation=pinyin;currency=CNY". 

For Linux/Unix it can be tricky to construct the tag from the Unix identifier.

So, for macOS and Linux/Unix it may be the simplest approach to use the identifier used to create the locale.

Regards,

Ulrich

[utelle]

u> Ok. I'll prepare a PR with the language database update and some small
u> adjustments of the related code.

Arrgh, preparing the PR may take a bit longer than I first thought. The reason is that I decided to use the script codes of BCP47 instead of the Unix script codes. This affects only a single entry of the existing language database, namely Serbian, but the Unix script code is used in the directory path to locate translation message catalogs. That is, the mechanism for loading message catalogs needs to be adjusted to convert from BCP47 script codes to Unix script codes. Since the updated language database contains a greater number of locales with script modifiers, we need a mapping for the 2 script code representations.

And it has to be decided whether the translation loader should look only in the directory with the Unix script code (for example "SR@latin") or also in the directory with the BCP47 script code (for example "SR@Latn").

Looks like I opened a can of worms. ;-)

Regards,

Ulrich

[Vadim Zeitlin]

On Sat, 20 Nov 2021 13:34:25 -0800 (PST) utelle wrote:

u> For Windows (Vista and above) a locale can be queried for the identifier,
u> and it is immediately in the form we need. In fact, it is the value
u> returned by GetName(). For XP GetName() returns the language name in
u> English - which is only a partial description of the locale. I think for XP
u> it would be better to query the LCID value and then look it up in the
u> language database.

IMO it would be perfectly fine to just document that this is not
implemented under XP and not do anything for it.

u> For macOS GetName() returns an identifier which is almost in the form we
u> need, but it could also be something like "zh-Hant_HK
u> _POSIX@collation=pinyin;currency=CNY".

Out of curiosity, how is it possible for this to happen, i.e. how do you
need to configure the system to get the extra part?

VZ

[Vadim Zeitlin]

On Sat, 20 Nov 2021 14:15:00 -0800 (PST) utelle wrote:

u> Arrgh, preparing the PR may take a bit longer than I first thought. The
u> reason is that I decided to use the script codes of BCP47 instead of the
u> Unix script codes. This affects only a single entry of the existing
u> language database, namely Serbian, but the Unix script code is used in the
u> directory path to locate translation message catalogs. That is, the
u> mechanism for loading message catalogs needs to be adjusted to convert from
u> BCP47 script codes to Unix script codes. Since the updated language
u> database contains a greater number of locales with script modifiers, we
u> need a mapping for the 2 script code representations.
u>
u> And it has to be decided whether the translation loader should look only in
u> the directory with the Unix script code (for example "SR@latin") or also in
u> the directory with the BCP47 script code (for example "SR@Latn").

FWIW on my (Debian) system I have some message catalogs under both
/usr/share/locale/sr@Latn and sr@latin, but the latter contains many more
of them and the former just a few. So I think that we could look in just
the latter directory, as it is clearly much more popular.

Does this help?
VZ

[utelle]

VZ schrieb am Samstag, 20. November 2021 um 23:26:32 UTC+1:

u> For macOS GetName() returns an identifier which is almost in the form we
u> need, but it could also be something like "zh-Hant_HK
u> _POSIX@collation=pinyin;currency=CNY".

Out of curiosity, how is it possible for this to happen, i.e. how do you
need to configure the system to get the extra part?

Well, I found this example in the macOS documentation, and there was a note telling "The value held in the property may differ from the identifier used to initialize the locale because NSLocale may canonicalize it during initialization." So, I suspect that if the system is configured for a non-default collation and/or currency the extra attributes will be added by the OS.

[Vadim Zeitlin]

On Sat, 20 Nov 2021 14:55:57 -0800 (PST) utelle wrote:

u> Well, I found this example in the macOS documentation
u> <https://developer.apple.com/documentation/foundation/nslocale/1416263-localeidentifier?language=objc>,
u> and there was a note telling "The value held in the property may differ
u> from the identifier used to initialize the locale because NSLocale may
u> canonicalize it during initialization." So, I suspect that if the system is
u> configured for a non-default collation and/or currency the extra attributes
u> will be added by the OS.

Yes, I understand, but I'm just not sure how to achieve this, i.e. what
would I need to do in the "Regional settings" to get a string like that.
It's not very important, but it might be useful for testing or even for
just interpreting bug reports later.

Regards,
VZ

[utelle]

VZ schrieb am Samstag, 20. November 2021 um 23:37:56 UTC+1:

u> And it has to be decided whether the translation loader should look only in
u> the directory with the Unix script code (for example "SR@latin") or also in
u> the directory with the BCP47 script code (for example "SR@Latn").

FWIW on my (Debian) system I have some message catalogs under both
/usr/share/locale/sr@Latn and sr@latin, but the latter contains many more
of them and the former just a few. So I think that we could look in just
the latter directory, as it is clearly much more popular.

Does this help?

Most likely it depends on the application which installs the message catalogs. @latin seems to be the classic Unix way, while @Latn is based on ISO 15924, where "latin" is listed as an alias for "Latn". And since you have both variants on your Debian system, maybe it would be better to look in both directories, thus leaving it to the application developer to decide where to place the message catalogs. I have not yet inspected the translations related code, what effort would be required to achieve that.

Regards,

Ulrich

[utelle]

utelle schrieb am Sonntag, 21. November 2021 um 00:13:29 UTC+1:

VZ schrieb am Samstag, 20. November 2021 um 23:37:56 UTC+1:

FWIW on my (Debian) system I have some message catalogs under both
/usr/share/locale/sr@Latn and sr@latin, but the latter contains many more
of them and the former just a few. So I think that we could look in just
the latter directory, as it is clearly much more popular.

Most likely it depends on the application which installs the message catalogs. @latin seems to be the classic Unix way, while @Latn is based on ISO 15924, where "latin" is listed as an alias for "Latn". And since you have both variants on your Debian system, maybe it would be better to look in both directories, thus leaving it to the application developer to decide where to place the message catalogs. I have not yet inspected the translations related code, what effort would be required to achieve that.

Since the classic Unix way is using the script code alias "latin" (instead of the ISO 15924 code "Latn") and the existing language database uses that convention and most likely the Unix convention was/is used currently by wx applications on other platforms as well, I get the impression we would be better off, if we use the Unix convention for the updated language database, too. This would allow to update the language database without requiring additional changes in the translation loading code (at least in the first step). Nevertheless, I intend to add conversion maps for the script code to languageinfo.cpp, so that it will be easier to handle conversion to/from wxLocaleIdent representation.

Regards,

Ulrich

[Vadim Zeitlin]

On Sun, 21 Nov 2021 01:36:56 -0800 (PST) utelle wrote:

u> Since the classic Unix way is using the script code alias "latin" (instead
u> of the ISO 15924 code "Latn") *and *the existing language database uses
u> that convention and most likely the Unix convention was/is used currently
u> by wx applications on other platforms as well, I get the impression we
u> would be better off, if we use the Unix convention for the updated language
u> database, too.

Just to confirm again: I'd be perfectly fine with this too.

u> This would allow to update the language database without
u> requiring additional changes in the translation loading code (at least in
u> the first step). Nevertheless, I intend to add conversion maps for the
u> script code to languageinfo.cpp, so that it will be easier to handle
u> conversion to/from wxLocaleIdent representation.

If you'd like to do it, you can do it, of course, but I honestly don't
think it's something we really need. There must be not that many
translations of wx applications into Serbian in the first place, and those
that do exist must be using "@latin" right now, as you say. So the main
(only?) reason to support both "@latin" and "@Latn" I see is the
intellectual satisfaction of having done the most correct thing possible.
Which is a perfectly fine motivation, of course, but, again, not something
that's going to change much in practice.

So please feel free to do whatever you prefer (including nothing)!
VZ

[utelle]

VZ schrieb am Sonntag, 21. November 2021 um 15:45:39 UTC+1:

u> [...] I get the impression we

u> would be better off, if we use the Unix convention for the updated language
u> database, too.

Just to confirm again: I'd be perfectly fine with this too.

Ok, then the updated langtabl.txt will use the Unix convention for now.

u> [script codes] 

If you'd like to do it, you can do it, of course, but I honestly don't
think it's something we really need.

I added a list of script codes and modified the generator script to add this data to the language database initialization.

 

There must be not that many
translations of wx applications into Serbian in the first place, and those
that do exist must be using "@latin" right now, as you say. So the main
(only?) reason to support both "@latin" and "@Latn" I see is the
intellectual satisfaction of having done the most correct thing possible.
Which is a perfectly fine motivation, of course, but, again, not something
that's going to change much in practice.

So please feel free to do whatever you prefer (including nothing)!

IMHO the interaction between wxLocaleIdent, wxUILocale, and wxTranslation will be smoother, if we have the mapping between 4-letter script identifiers (BCP 47) and alias names (Unix convention) available.

I made a first PR with the update of the language database.

Unfortunately, not all checks go through. Interestingly, some tests fail for macOS builds only. Failure seems to be linked to setting a French locale. However, I have no idea, what's going on. The tests work for all Windows and Unix builds. I'm at a loss, why those tests fail on macOS. Any hint would be welcome.

Regards,

Ulrich

[Vadim Zeitlin]

On Tue, 23 Nov 2021 01:00:20 -0800 (PST) utelle wrote:

u> I made a first PR with the update of the language database.

Thanks again, I've finally had a look at it -- sorry for the delay, but
I'm very busy right now as I have to finish something urgently and so don't
really have any time for anything else. I'll get back to this as soon as I
can, but it might take a few more days until I do...

u> Unfortunately, not all checks go through. Interestingly, some tests fail
u> for macOS builds only. Failure seems to be linked to setting a French
u> locale.

To me it rather looks like they fail to load the message catalog.

Regards,
VZ

[utelle]

VZ schrieb am Donnerstag, 25. November 2021 um 15:14:27 UTC+1:

u> I made a first PR with the update of the language database.

Thanks again, I've finally had a look at it -- sorry for the delay, but
I'm very busy right now as I have to finish something urgently and so don't
really have any time for anything else. I'll get back to this as soon as I
can, but it might take a few more days until I do...

No problem, take your time to complete your urgent tasks.

u> Unfortunately, not all checks go through. Interestingly, some tests fail
u> for macOS builds only. Failure seems to be linked to setting a French
u> locale.

To me it rather looks like they fail to load the message catalog.

No, as I found out, it is the function setlocale  that fails, if the locale name does not contain a region specifier. IMHO this is a misbehaviour, but probably there isn't much we can do about it.

Regards,

Ulrich