Porting gwt-i18n module to gwt3

[Ahmad Bawaneh]

Dears

I am working on porting the i18n module, so far all i did is extract the module and the tests into and external repository and make the tests pass, not real change to the code have been yet. but the i18n module is an important module and is being used by so many gwt application so porting this one should be taken with care, this why i am opining this discussion so that i can hear and share idea's.

i would like to start this discussion by sharing some of my thoughts, which are limited to my knowledge of the gwt i18n, and i would love to get corrections and feedback to what ever wrong or good thinking i have.

so first in a gwt 2.x application using i18n with generators assumes that the set of locales available is known during the code generation, for example if you an application that has a module lets call it module A that define a messages interface (MessagesA) and defines the locales to be `en`, `fr` for example, then once the application is compiled, the generator should produce MessagesA class for each locale and we end up with MessagesA_en and MessagesA_fr having that we provide properties files for both.

now assume we add a new module to our application module B and B defines another messages interface MessagesB and extends the locale property to add a third locale `it` and provide properties files for MessagesB en,fr and it and also provide a properties file for MessagesA it, then when we compile we will end up with MessagesA_en, MessagesA_fr, and MessagesA_it, MessagesB_en, MessagesB_fr, and MessagesB_it.

with this i could extends the application messages by adding a new module thanks to the Property oracle that provide information about all defined and extended locales.

now in the APT world with multi modules things are different, the processors does not rum for all modules as one unit instead the processor will run for a single module and will have knowledge limited to that module,  so  module A processor wont know about locales defined in module B, and module B processor wont know about locale values provided by module A, so and expected result that i will end up with MessagesA_en, MessagesA_fr for module A and MessagesB_it for module B.

what do you think?

[Colin Alworth]

I would be interested in hearing about concrete cases where one jar adds .properties files for a class declared in a completely different jar - I haven't seen that done in the wild before, it sounds like a bit of a stretch to me. It might also be something that we could formally discourage in the "new and improved i18n tool" docs. Alternatively, I've heard of cases where the upstream jar doesn't run the processor, but just defines the sources and lets the downstream jar run it when it provides the rest of the impl details.

At the risk of sounding like I'm speaking for the team that built the original i18n module, I see it as trying to replicate PropertyResourceBundle behavior in the JVM, without the need to have runtime access to the classpath. I could be mistaken, but using PropertyResourceBundle in the way you describe (loading .properties files from a different jar) may no longer work correctly in Java 9 anyway, at least without custom module permissions for access (somewhat akin to running or re-running the processor on the downstream project)? Might be wrong on that. Reusing that .properties format for localized files and BundleName_locale.properties file name format seems like an express design goal to me, no matter what other changes happen along the way.

[Ahmad Bawaneh]

Then how do we expect the users of gwt 3.0 to define locales in a multi module project in a way that make i18n consistence across all modules and still allow an easy migration from GWT 2.x application?

[Learner Evermore]

We are not using GWT I18N at all to avoid the permutations this causes but we absolutely do separate translation *.properties from the main code. Translations are done by a different team, following a different schedule and may come *after* the code release (in our case even after the product is released - customers can get more language packs later or supply their own). For these reasons (this is how translators work and this is how customers need to get the translations) the translations are bundled based on locale, not jars of code that use them.

Finally, not all translations/properties *belong* to a specific code. We also have runtime/dynamic metadata-driven functionality requiring additional, translations. These and other similar cases require translation *.properties files that do not have code that references them statically at all.

--
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-contributors+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/e40829d8-fbda-47b9-9516-27ea7995cc81%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Thomas Broyer]

On Friday, January 5, 2018 at 6:33:48 PM UTC+1, Ahmad Bawaneh wrote:

Dears

I am working on porting the i18n module, so far all i did is extract the module and the tests into and external repository and make the tests pass, not real change to the code have been yet. but the i18n module is an important module and is being used by so many gwt application so porting this one should be taken with care, this why i am opining this discussion so that i can hear and share idea's.

Thanks for taking this initiative; i18n support indeed is a must-have.

I however think we need to critically reevaluate our current design (particularly in light of the new JS Intl APIs that already have wide browser support: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl).

Constants, ConstantsWithLookup, Messages, and CustomDateTimeFormat however are likely to stay similar though (and have their implementation rely on those new APIs).

 

i would like to start this discussion by sharing some of my thoughts, which are limited to my knowledge of the gwt i18n, and i would love to get corrections and feedback to what ever wrong or good thinking i have.

so first in a gwt 2.x application using i18n with generators assumes that the set of locales available is known during the code generation, for example if you an application that has a module lets call it module A that define a messages interface (MessagesA) and defines the locales to be `en`, `fr` for example, then once the application is compiled, the generator should produce MessagesA class for each locale and we end up with MessagesA_en and MessagesA_fr having that we provide properties files for both.

now assume we add a new module to our application module B and B defines another messages interface MessagesB and extends the locale property to add a third locale `it` and provide properties files for MessagesB en,fr and it and also provide a properties file for MessagesA it, then when we compile we will end up with MessagesA_en, MessagesA_fr, and MessagesA_it, MessagesB_en, MessagesB_fr, and MessagesB_it.

with this i could extends the application messages by adding a new module thanks to the Property oracle that provide information about all defined and extended locales.

now in the APT world with multi modules things are different, the processors does not rum for all modules as one unit instead the processor will run for a single module and will have knowledge limited to that module,  so  module A processor wont know about locales defined in module B, and module B processor wont know about locale values provided by module A, so and expected result that i will end up with MessagesA_en, MessagesA_fr for module A and MessagesB_it for module B.

what do you think?

I think anything that supports localization should expose ways to replace the constants or messages being used, like SimplePager or CellTree do, or possibly commit to providing localization for as many locales as possible.

That way, applications can pass their own messages.

An alternative is to re-run the annotation processor in the downstream module for the upstream class (call javac passing the interface name rather than –or in addition to– compiling files) so it has access to your new properties files; and then rely on classpath shadowing to use your version rather than the original. This doesn't fit well with Java 9 JPMS as far as I can tell, but it at least provides an alternative until APIs change to allow "injecting" your own messages as suggested above.

[Thomas Broyer]

I completely forgot that we could also possibly have Messages and Constants compile down to Closure Library's goog.getMsg() for later processing by the Closure Compiler. This obviously means that it wouldn't work with GWT 2 though, and tightly bounds us to the Closure Compiler (not just for goog.module/goog.require module loading), but the upside is that the translations are provided at the very final Closure Compiler stage (but then, this raises a new question: how libraries would easily contribute their translations to applications?)

[Colin Alworth]

In response to both Learner and Thomas, I think those "dynamic" solutions would be great to have in later releases, so as to first support GWT 2's existing functionality, and second optimize for use cases like runtime strings changing or leaning on closure directly to improve compile times (and directly interop with other js libs that also happen to use closure, though in my experience this sadly tends not to be a long list).

I've been playing with the new Intl for both NumberFormat and DateTimeFormat, and both seem somewhat lacking compared to what GWT offers - there is no option to provide your own format string as far as I can tell, only request various options. This might work for 80% or so of cases, but I can imagine things getting very frustrating for that last 20% without each project writing their own parser if GWT's i18n doesn't support it. DateTimeFormat.formatToParts looks like it could potentially be used as a primitive in building this functionality (if we wanted to forgo any JVM support) perhaps. That said, formatToParts is still not well supported yet (no Edge, no IE11, no FF ESR).

--

  Colin Alworth

  co...@colinalworth.com

--

You received this message because you are subscribed to the Google Groups "GWT Contributors" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/d3466879-fa8d-4ea3-840f-c57aabb171bf%40googlegroups.com.

[Ahmad Bawaneh]

Dears

I am trying to simplify my understanding of the i18n module, and now i focus on the constants part since it looks to be simple to start with, here are my findings :

Having a constants interface 

1.  Values will be retrieved from a properties file, if a key is missing from the properties file, then value is retrieved from @DefaultXXX annotation value, and if missing from both compilation fails.
2. for each extended locale value a class will be generated for each corresponding existed .properties file.
   
3. if locale property is set then only a class for that locale will be generated.

what i am looking for here is a set of rules where i can implement a simple annotation processor that replicate the same behavior and generates the same set of classes following the same naming convention,

knowing that here i am ignoring the fact of the global locale knowledge currently presented with generators.

so if you have any point/note/suggestion to add i will appreciate that.

Thanks

[Ahmad Bawaneh]

Dears
here a repository where i experiment with constants generation with apt instead of generators, i am doing this to find what problems i might face

it is not yet complete but it covers so for the most basic use of constants.

https://github.com/vegegoku/gwt-constants-apt

Thanks

On Friday, January 5, 2018 at 7:33:48 PM UTC+2, Ahmad Bawaneh wrote:

[Ahmad Bawaneh]

a ported version of the gwt-cldr importer and gwt-cldr is now available on github 

https://github.com/vegegoku/gwt-i18n-cldr

https://github.com/vegegoku/gwt-cldr-importer

On Friday, January 5, 2018 at 7:33:48 PM UTC+2, Ahmad Bawaneh wrote:

[Learner Evermore]

I think I18N is important. However, we never liked or used the GWT 2.x style of it because it requires dev time knowledge of locales and multiplies permutations (compile time). It was also inflexible another way - e.g. if a user wants to switch or update the language the code is reloaded as well and the state lost.

Instead we load localizations separately from the main code (and have the ability to automaticall, at runtime, merge it with the code, should we wish to do so). 

A serious GWT must have a serious I18N system, *not* like GWT 2.x.

Now, we still need to address porting of existing code - if there remains a desire for porting today. We can make that easier. However, since some work will be involved anyway, the compatibility system does not have to do it the GWT 2.x way, just needs to be more-or-less compatible with it from the coding perspective.

Learner Evermore

--

You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/ab91963d-dc97-40eb-a8f0-fa51bfd32495%40googlegroups.com.

[Thomas Broyer]

On Saturday, April 7, 2018 at 3:03:39 PM UTC+2, Learner Evermore wrote:

I think I18N is important. However, we never liked or used the GWT 2.x style of it because it requires dev time knowledge of locales and multiplies permutations (compile time). It was also inflexible another way - e.g. if a user wants to switch or update the language the code is reloaded as well and the state lost.

Instead we load localizations separately from the main code (and have the ability to automaticall, at runtime, merge it with the code, should we wish to do so). 

A serious GWT must have a serious I18N system, *not* like GWT 2.x.

Now, we still need to address porting of existing code - if there remains a desire for porting today. We can make that easier. However, since some work will be involved anyway, the compatibility system does not have to do it the GWT 2.x way, just needs to be more-or-less compatible with it from the coding perspective.

I 100% agree (as I said earlier in this thread) that we need to critically reevaluate and rethink how we do i18n. One thing is that we need to evaluate messages and constants separately from formatting/collation/etc. The latter could use generation from CLDR (like in GWT 2), or possibly delegate to the browser's native Intl (https://caniuse.com/#feat=internationalization), or even possibly support both (with minimal to no change to the code).

IFF we can get something that allows for dynamically switching locale without reloading, that would be even better (though this has –probably, maybe you could tell us more about how you did it– huge impacts in how you architecture your application)

One thing appears quite clear to me though, that in terms of application architecture, everything should be "injected" into the components et al. I.e. in your application, you define your messages, etc. and most importantly which locales you're going to support, this would then generate appropriate code to handle it, and you pass the required instances (DateTimeFormat or whatever) down to the components coming from libraries.

I believe we should brainstorm rather than rush into implementations, or we're going to simply copy what GWT 2 is already doing. We should aim for maximum backwards compatibility, but we have the opportunity of doing things better and introduce a few breaking changes, and we should seize it!

[Ahmad Bawaneh]

I also one of those who replaced gwt-i18n with my own implementation which was a simple dictionary like, that loads the labels and messages at runtime, but in my case i didnt need more than plain simple text translation without parameters or anything else.

with that said, one of the goals of porting gwt 2 modules is to provide a smooth transition for applications that uses gwt into gwt 3.0, this does not mean we cant introduce something new or implement things differently, but we can have both, while we are porting the old module and ensure a smooth transition, we give ourselves a better chance and freedom to brainstorm, introduce and discuss new things.

while working on porting i18n module i am trying to break it into smaller modules that are more specific to one thing,  so :

1- i started by extracting the i18n module from the monolith gwt code, and make sure the test cases works.

2- i have draft implementation of an APT based constants generator https://github.com/vegegoku/gwt-constants-apt

3- extracted  CLDR into its own module with no JSNI and JSO, the generator is also moved out of the gwt tools into its own module and now works with gradle instead of ant.

and i am pretty sure there are many gwt i18n users who will appreciate this non evolutionary porting, but also i believe this will give us a better ground when we start introducing new things, and for such a large module will help us with coordination and get more involved.

Thanks

[Thomas Broyer]

I didn't mean to be dismissive of your work (sorry if that might have sounded like this); on the contrary this is valuable work, but IMO more as an "experimentation" exploring what can be done than a definitive answer to how is should be done now.

On Saturday, April 7, 2018 at 8:39:32 PM UTC+2, Ahmad Bawaneh wrote:

I also one of those who replaced gwt-i18n with my own implementation which was a simple dictionary like, that loads the labels and messages at runtime, but in my case i didnt need more than plain simple text translation without parameters or anything else.

with that said, one of the goals of porting gwt 2 modules is to provide a smooth transition for applications that uses gwt into gwt 3.0, this does not mean we cant introduce something new or implement things differently, but we can have both, while we are porting the old module and ensure a smooth transition, we give ourselves a better chance and freedom to brainstorm, introduce and discuss new things.

while working on porting i18n module i am trying to break it into smaller modules that are more specific to one thing,  so :

1- i started by extracting the i18n module from the monolith gwt code, and make sure the test cases works.

2- i have draft implementation of an APT based constants generator https://github.com/vegegoku/gwt-constants-apt

For example, here, you pass the locales as a constant on the annotation, which is not a “non evolutionary porting” then.

This implies that when/if you want to add a new locale, you need to go through all your constants interfaces to not only add properties files (see below about that too) but also add the locale to the annotation. I wonder if we couldn't instead pass an option to the annotation processor (we could also have the processor load the properties files from the processor path; in Gradle this would mean that changing a properties file would trigger a recompile; unfortunately this is impractical at best in Maven). An alternative would be a generator that's not an annotation processor (no idea how you'd call it; that idea maybe not worth pursuing as that could be too cumbersome for users; unless maybe the generator generates intermediate Java source that's only processed by the annotation processor later and never actually used at runtime; the goal only being that the input to javac –maven-compiler-plugin or Gradle JavaCompile task– is only made of Java source files, for more accurate incremental builds –i.e. correctly trigger a recompile when needed).

Wrt properties files, current GWT 2.x implementation will look for properties files for the superinterfaces too up to Localizable; which provides a way to centralize the values across the entire application. When adding a new locale, you could then give a single properties file to the translator, and have a single new properties file to add to your application (and not one per constants interfaces; but with your proposed implementation, you'd still have to go through all of them to add the locale to the annotation).

Ideally, we should also plan our annotation processors so they work great with Gradle's experimental incremental compilation: https://docs.gradle.org/release-candidate/userguide/java_plugin.html#sec:incremental_annotation_processing

3- extracted  CLDR into its own module with no JSNI and JSO, the generator is also moved out of the gwt tools into its own module and now works with gradle instead of ant.

and i am pretty sure there are many gwt i18n users who will appreciate this non evolutionary porting, but also i believe this will give us a better ground when we start introducing new things, and for such a large module will help us with coordination and get more involved.

I agree that we should aim for maximum backwards compatibility, but there will be breaking changes anyway (if only to replace GWT.create() with calls to a –generated– static factory).

I'll try to setup a document today with what I have in mind so we can discuss.

[Thomas Broyer]

On Sunday, April 8, 2018 at 9:39:24 AM UTC+2, Thomas Broyer wrote:

I'll try to setup a document today with what I have in mind so we can discuss.

Here's a first shot: https://docs.google.com/document/d/18kaSQ3MXstFtqqzFmLZatzMNkLxDWt3NwxkLam8VD40/edit?usp=sharing

Open for comments/suggestions to any member of this group.

[Ahmad Bawaneh]

oh no, i am the one who is sorry if i made it look like that,

On Sunday, April 8, 2018 at 10:39:24 AM UTC+3, Thomas Broyer wrote:

I didn't mean to be dismissive of your work (sorry if that might have sounded like this); on the contrary this is valuable work, but IMO more as an "experimentation" exploring what can be done than a definitive answer to how is should be done now.

On Saturday, April 7, 2018 at 8:39:32 PM UTC+2, Ahmad Bawaneh wrote:

I also one of those who replaced gwt-i18n with my own implementation which was a simple dictionary like, that loads the labels and messages at runtime, but in my case i didnt need more than plain simple text translation without parameters or anything else.

with that said, one of the goals of porting gwt 2 modules is to provide a smooth transition for applications that uses gwt into gwt 3.0, this does not mean we cant introduce something new or implement things differently, but we can have both, while we are porting the old module and ensure a smooth transition, we give ourselves a better chance and freedom to brainstorm, introduce and discuss new things.

while working on porting i18n module i am trying to break it into smaller modules that are more specific to one thing,  so :

1- i started by extracting the i18n module from the monolith gwt code, and make sure the test cases works.

2- i have draft implementation of an APT based constants generator https://github.com/vegegoku/gwt-constants-apt

For example, here, you pass the locales as a constant on the annotation, which is not a “non evolutionary porting” then.

This implies that when/if you want to add a new locale, you need to go through all your constants interfaces to not only add properties files (see below about that too) but also add the locale to the annotation. I wonder if we couldn't instead pass an option to the annotation processor (we could also have the processor load the properties files from the processor path; in Gradle this would mean that changing a properties file would trigger a recompile; unfortunately this is impractical at best in Maven). An alternative would be a generator that's not an annotation processor (no idea how you'd call it; that idea maybe not worth pursuing as that could be too cumbersome for users; unless maybe the generator generates intermediate Java source that's only processed by the annotation processor later and never actually used at runtime; the goal only being that the input to javac –maven-compiler-plugin or Gradle JavaCompile task– is only made of Java source files, for more accurate incremental builds –i.e. correctly trigger a recompile when needed).

Wrt properties files, current GWT 2.x implementation will look for properties files for the superinterfaces too up to Localizable; which provides a way to centralize the values across the entire application. When adding a new locale, you could then give a single properties file to the translator, and have a single new properties file to add to your application (and not one per constants interfaces; but with your proposed implementation, you'd still have to go through all of them to add the locale to the annotation).

Ideally, we should also plan our annotation processors so they work great with Gradle's experimental incremental compilation: https://docs.gradle.org/release-candidate/userguide/java_plugin.html#sec:incremental_annotation_processing

Yes you are right, for this draft is not completed yet and is exactly as you said and experiment, for example reading values from annotation is not yet implemented, and walking the inheritance tree up to localizable, but those already in mind, thats why i didnt move the code to org.gwtproject yet, and for gradle, i am really new to gradle so i dont have the full insight, but i know there is people involved who will correct me when i go wrong or miss something.

 

3- extracted  CLDR into its own module with no JSNI and JSO, the generator is also moved out of the gwt tools into its own module and now works with gradle instead of ant.

and i am pretty sure there are many gwt i18n users who will appreciate this non evolutionary porting, but also i believe this will give us a better ground when we start introducing new things, and for such a large module will help us with coordination and get more involved.

I agree that we should aim for maximum backwards compatibility, but there will be breaking changes anyway (if only to replace GWT.create() with calls to a –generated– static factory).

I'll try to setup a document today with what I have in mind so we can discuss.

having such document and have more brains getting involved is really a good thing.