Type-safe message bundles powered by Qute

[Martin Kouba]

Hi all,

while thinking how to add some i18n support to Qute [1] I had an idea
that it could be useful to introduce type-safe message bundles powered
by Qute. I've sent a draft PR [2] which is a working POC (some
validation parts are still missing).

The basic idea is that each message is potentially a very simple
template. And how does it work?

First you define the default message bundle interface:

@MessageBundle("msg") // -> msg is a namespace that can be used in templates
public interface MyMessages {

// each method represents a message
@Message("Hello {name}!") // value is a qute template
String hello_name(String name); // name is a param that can be used
in the template
}

You can also define a localized bundle:

@Localized("cs")
public interface CsMessages extends MyMessages {

@Message("Ahoj {name}!") // localized template
@Override
String hello_name(String name);

}

Quarkus generates the implementations during build and you can use the
bundles:

1. directly in your code
MessageBundles.get(MyMessages.class).hello_name("Jachym")
2. @Inject MyMessages messages
3. in the templates: {msg:hello_name('Jachym')}

See also the test [3].

This approach has some advantages and also disadvantages. However, I
believe that overall it fits the "Quarkus way".

WDYT?

Martin

[1]
https://github.com/quarkusio/quarkus/issues/7665

[2]
https://github.com/quarkusio/quarkus/pull/8516

[3]
https://github.com/mkouba/quarkus/blob/issue-7665/extensions/qute/deployment/src/test/java/io/quarkus/qute/deployment/i18n/MessageBundleTest.java

[George Gastaldi]

I haven't seen the poc yet, but you'll need to create a subinterface and override methods for each additional language? 

That sounds too much work compared to adding an entry in a properties file.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/fa8305fd-bc04-2b15-d92d-0b255350f104%40redhat.com.

[George Gastaldi]

But I like it :)

[Max Rydahl Andersen]

if the actual messages are inside classes is there any way for existing
toolchains for translation to integrate with this ?
And how does this handle pluralisation ? as parameters to the method or
?

Can the actual messages be in traditional i18n resource
bundles/properties and we byte code generate these classes ?

/max
https://xam.dk/about

[Stephane Epardaud]

I was initially thinking properties files are shorter than this, but they're actually not much shorter, and you get zero support for completion, refactoring and validation of parameters, so perhaps we should support both options, but this one is about as type-safe as possible while also being as concise as possible when type-safe. Which is what makes it very valuable IMO.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/C37AA40E-3224-4551-9EDC-239D3537617A%40redhat.com.

--

Stéphane Épardaud

[Ken Finnigan]

What about just using https://jboss-logging.github.io/jboss-logging-tools/ for defining messages with Interfaces and then generating the classes at build time?

It's what we are planning to use in SmallRye for defining messages for product that is needed to support translation

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKU9E9unomXXtRzq7%3DjFAoOBAKoG-PcMRiqoXVt8NQoHVLhEwg%40mail.gmail.com.

[Martin Kouba]

Dne 10. 04. 20 v 13:54 George Gastaldi napsal(a):

> I haven't seen the poc yet, but you'll need to create a subinterface and
> override methods for each additional language?

Yes. You'll need to override the messages you'd like to localize.

>
> That sounds too much work compared to adding an entry in a properties file.

Yes, the initial cost is higher. But it starts to pay off once you have
several localizations and modify/refactor the messages. Normally, you
would need a special tool to keep all messages in sync etc. Here you can
rely on the compiler, ide refactorings and quarkus validation (to be
implemented ;-).

>
> Em Sex, 10 de abr de 2020 05:44, Martin Kouba <mko...@redhat.com

> <mailto:mko...@redhat.com>> escreveu:

> <mailto:quarkus-dev%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/quarkus-dev/fa8305fd-bc04-2b15-d92d-0b255350f104%40redhat.com.
>

--
Martin Kouba
Senior Software Engineer
Red Hat, Czech Republic

[Martin Kouba]

Dne 10. 04. 20 v 15:54 Max Rydahl Andersen napsal(a):

>
> if the actual messages are inside classes is there any way for existing
> toolchains for translation to integrate with this ?

No integration so far. What toolchains do you have in mind? We could add
the support later...

> And how does this handle pluralisation ? as parameters to the method or ?

No builtin support so far. Bu since the message template can contain any
qute logic, you can simply add {#if numberOfPeople == 1} etc.

>
> Can the actual messages be in traditional i18n resource

> bundles/properties and we byte code generate these classes ?.

Nope. Actually, that would kill the purpose. If you want to stick with
properties files than you could just use the ResourceBundle.

[Martin Kouba]

Dne 10. 04. 20 v 17:18 Ken Finnigan napsal(a):

> What about just using
> https://jboss-logging.github.io/jboss-logging-tools/ for defining
> messages with Interfaces and then generating the classes at build time?

I think that this would be a bit confusing. These annotations make use
of positional parameter values and java.util.Formatter-type and
java.text.MessageFormat-type formats. We would like to use param names
and qute (including control flow constructs). Moreover, we'd like to use
the messages directly in templates. Of course, it's possible to use the
bundles generated by jboss-logging (through the annotation processor) in
the app code even now...

> <mailto:quarkus-dev%2Bunsu...@googlegroups.com>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/quarkus-dev/fa8305fd-bc04-2b15-d92d-0b255350f104%40redhat.com.
>
> --
> You received this message because you are subscribed to the
> Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to quarkus-dev...@googlegroups.com

> <mailto:quarkus-dev%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/quarkus-dev/C37AA40E-3224-4551-9EDC-239D3537617A%40redhat.com.
>
>
>
> --
> Stéphane Épardaud
>
> --
> You received this message because you are subscribed to the Google
> Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to quarkus-dev...@googlegroups.com

> <mailto:quarkus-dev...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/quarkus-dev/CAKU9E9unomXXtRzq7%3DjFAoOBAKoG-PcMRiqoXVt8NQoHVLhEwg%40mail.gmail.com
> <https://groups.google.com/d/msgid/quarkus-dev/CAKU9E9unomXXtRzq7%3DjFAoOBAKoG-PcMRiqoXVt8NQoHVLhEwg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Max Rydahl Andersen]

>> if the actual messages are inside classes is there any way for
>> existing toolchains for translation to integrate with this ?
>
> No integration so far. What toolchains do you have in mind? We could
> add the support later...

I'm by no means and expert on these but all the ones I've had contact
with are all text file bases (.properties or similar) and then
tooling made to wire in at build/runtime.

>> And how does this handle pluralisation ? as parameters to the method
>> or ?
>
> No builtin support so far. Bu since the message template can contain
> any qute logic, you can simply add {#if numberOfPeople == 1} etc.

k

>> Can the actual messages be in traditional i18n resource
>> bundles/properties and we byte code generate these classes ?.
>
> Nope. Actually, that would kill the purpose. If you want to stick with
> properties files than you could just use the ResourceBundle.

what is the purpose then ? if I (and my lang translators) have to
maintain a full class per language as opposed to just at build time
generate them what is the win ?

/max

[Martin Kouba]

Dne 14. 04. 20 v 11:56 Max Rydahl Andersen napsal(a):

>>> if the actual messages are inside classes is there any way for
>>> existing toolchains for translation to integrate with this ?
>>
>> No integration so far. What toolchains do you have in mind? We could
>> add the support later...
>
> I'm by no means and expert on these but all the ones I've had contact
> with are all text file bases (.properties or similar) and then
> tooling made to wire in at build/runtime.
>
>>> And how does this handle pluralisation ? as parameters to the method
>>> or ?
>>
>> No builtin support so far. Bu since the message template can contain
>> any qute logic, you can simply add {#if numberOfPeople == 1} etc.
>
> k
>
>>> Can the actual messages be in traditional i18n resource
>>> bundles/properties and we byte code generate these classes ?.
>>
>> Nope. Actually, that would kill the purpose. If you want to stick with
>> properties files than you could just use the ResourceBundle.
>
> what is the purpose then ? if I (and my lang translators) have to
> maintain a full class per language as opposed to just at build time
> generate them what is the win ?

So there are cases where all you want is to have a "source of messages"
which is easy to maintain/refactor, which is java source for java
developers ;-). And of course, I can use the interface directly (in a
type-safe manner). That wouldn't be possible if you generate the stuff
from files.

Of course, if you work on a large project translated to N languages and
have N dedicated translators then you'll prefer some externalized source
(such as properties files, projectfluent.org, etc.).

I'm not saying that it's a universal solution. But it offers nice
features such as refactoring capabilities, e.g. I can rename the method
on the "default" interface and all localizations are renamed
automatically by my IDE. Or validation - quarkus tells me that my
message template defines an unknown property (because it compares the
method parameters and the template).

[Ladislav Thon]

út 14. 4. 2020 v 13:58 odesílatel Martin Kouba <mko...@redhat.com> napsal:

I'll tell you it sounds really weird. I'd expect one interface (with default messages) to be enough -- all translations should be external. If I want to add a language, I should be able to stick a .properties file somewhere, and Quarkus should build it into the application. (How does it know which key corresponds to which method? I don't know, but it should be possible to either generate unique and meaningful keys automatically, or let me define them.)

LT

 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/1ffa108f-ae6e-7e44-98c0-fc140a62c028%40redhat.com.

[Martin Kouba]

Dne 14. 04. 20 v 15:00 Ladislav Thon napsal(a):

> út 14. 4. 2020 v 13:58 odesílatel Martin Kouba <mko...@redhat.com

> <mailto:mko...@redhat.com>> napsal:

> <http://projectfluent.org>, etc.).

>
> I'm not saying that it's a universal solution. But it offers nice
> features such as refactoring capabilities, e.g. I can rename the method
> on the "default" interface and all localizations are renamed
> automatically by my IDE. Or validation - quarkus tells me that my
> message template defines an unknown property (because it compares the
> method parameters and the template).
>
>
> I'll tell you it sounds really weird. I'd expect one interface (with
> default messages) to be enough -- all translations should be external.

Well, "weird" does not necessarily mean "bad" ;-). I get your point and
I do understand that most developers only worked with external
translations.

On the other hand, it does not mean that you would never use another
approach if it was available and functional.

Maybe it's not worth it. We'll see.

I personally like the idea that it's all about types. E.g. if I want to
find out whether a message is localized I use my IDE to find "all types
implementing a method" and then just jump into the right location to
modify the template. Of course, it's also doable with properties files
but it would require additional tools and more complex logic to find the
correct lines.

> <mailto:quarkus-dev%2Bunsu...@googlegroups.com>.

> >>>> To view this discussion on the web visit
> >>>>
> https://groups.google.com/d/msgid/quarkus-dev/fa8305fd-bc04-2b15-d92d-0b255350f104%40redhat.com.
>
> >>>>
> >>>
> >>
> >> --
> >> Martin Kouba
> >> Senior Software Engineer
> >> Red Hat, Czech Republic
> >
>
> --
> Martin Kouba
> Senior Software Engineer
> Red Hat, Czech Republic
>
> --
> You received this message because you are subscribed to the Google
> Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to quarkus-dev...@googlegroups.com

> <mailto:quarkus-dev%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/quarkus-dev/1ffa108f-ae6e-7e44-98c0-fc140a62c028%40redhat.com.
>

[Christian Köberl]

Just tried the feature in 1.7.0.CR1 - here is my feedback to the new implementation for type-safe message bundles.

The idea with the type-safe implementation is good and will help a lot using message bundles.

But there are some issues with the current implementation:

• There should be the ability to define all messages in a properties (or PO, or XLIFF) file - also the default ones. This is needed for translation systems to work properly.
  An alternative could be exporting/importing those files as a build step.
• Dot '.' should be allowed in the message key. This is used by some translations tools to group texts into different sections (modules) for translation. For example "app-maintenance.settings.admin-email-label"
  
• There should be the ability to adapt/extend the loading of messages via your own code. Most Translation systems (like Weblate) have a REST API where you can get your translations on demand. With this ability the app developer can load the translations from the API so the translator can see the immediate effect of her translations in the running application. This is a small library doing this for Spring's MessageSource with Zanata: https://github.com/porscheinformatik/zanata-spring
  
• Changing the msg_xx.properties files should reload the app in dev mode
• Pluralization (and other adoption like gender) should be added - ideally using MessageFormat since it is well established in Java (and JavaScript)
  

Cheers,

Chris

[Martin Kouba]

Hi Christian,

thanks a lot for your feedback!

Note that "type-safe message bundles" are not intended to cover all
complex i18n use cases. Instead, it provides a type-safe alternative to
solve common tasks.

I added some comments inline.

Dne 04. 08. 20 v 11:40 Christian Köberl napsal(a):

> Just tried the feature in 1.7.0.CR1 - here is my feedback to the new
> implementation for type-safe message bundles.
>
> The idea with the type-safe implementation is good and will help a lot
> using message bundles.
>
> But there are some issues with the current implementation:
>

> * There should be the ability to define all messages in a properties

> (or PO, or XLIFF) file - also the default ones. This is needed for
> translation systems to work properly.

That kind of defeats the purpose of the type-safe message bundles. That
way we wouldn't be able to validate the messages properly.

> An alternative could be exporting/importing those files as a build step.

Well, the interface could be generated but that would not help with
validation.

> * Dot '.' should be allowed in the message key. This is used by some

> translations tools to group texts into different sections (modules)
> for translation. For example
> "app-maintenance.settings.admin-email-label"

The problem with dots is that we use them to split expressions in Qute.
We also support the bracket notation but I need to verify whether it
works with namespaces, i.e.
msg:['app-maintenance.settings.admin-email-label']... added to my TODO list.

> * There should be the ability to adapt/extend the loading of messages

> via your own code. Most Translation systems (like Weblate

> <https://weblate.org/>) have a REST API where you can get your

> translations on demand. With this ability the app developer can load
> the translations from the API so the translator can see the
> immediate effect of her translations in the running application.
> This is a small library doing this for Spring's MessageSource with
> Zanata: https://github.com/porscheinformatik/zanata-spring

That's probably beyond the capabilities we would like to introduce at
this stage.

> * Changing the msg_xx.properties files should reload the app in dev mode

That's a bug or rather an oversight. I've created
https://github.com/quarkusio/quarkus/issues/11199.

> * Pluralization (and other adoption like gender) should be added -

> ideally using MessageFormat since it is well established in Java

> (and JavaScript <https://github.com/messageformat/messageformat>)

Right now, you can use basic control flow constructs to handle similar
use cases ({#if}). In theory, nothing prevents you from using
MessageFormat directly in your messages, but since it is not thread-safe
you would need to synchronize externally.

> <https://github.com/mkouba/quarkus/blob/issue-7665/extensions/qute/deployment/src/test/java/io/quarkus/qute/deployment/i18n/MessageBundleTest.java>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to quarkus-dev...@googlegroups.com

> <mailto:quarkus-dev...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/quarkus-dev/aeda7538-7aa4-4382-85ae-862e69f5d249o%40googlegroups.com
> <https://groups.google.com/d/msgid/quarkus-dev/aeda7538-7aa4-4382-85ae-862e69f5d249o%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Christian Köberl]

Hi Martin!

Here are some remarks to your answers.

 

>   * There should be the ability to define all messages in a properties
>     (or PO, or XLIFF) file - also the default ones. This is needed for
>     translation systems to work properly.

That kind of defeats the purpose of the type-safe message bundles. That
way we wouldn't be able to validate the messages properly.

You could just declare the message in the Java code and provide the text like this

AppMessages.java

@Message String hello(); msg.properties helllo=Hello With this you could use a translation tool - without that you simply can't.

>     An alternative could be exporting/importing those files as a build step. Well, the interface could be generated but that would not help with validation.

I thought about generating the base properties file from the Java code (not the other way around) so that a translation tool can use it as input.

The problem with dots is that we use them to split expressions in Qute.

I think that would be a nice-to-have (probably also a need-to-have when you migrate from another framework or use messages in a multi-app setup).

Right now, you can use basic control flow constructs to handle similar use cases ({#if}). In theory, nothing prevents you from using MessageFormat directly in your messages, but since it is not thread-safe you would need to synchronize externally.

{#if} does not work for pluralization (see Russian for an example http://www.russianlessons.net/lessons/lesson11_main.php).

>   * There should be the ability to adapt/extend the loading of messages >     via your own code. Most Translation systems (like Weblate >     <https://weblate.org/>) have a REST API where you can get your >     translations on demand. With this ability the app developer can load >     the translations from the API so the translator can see the >     immediate effect of her translations in the running application. >     This is a small library doing this for Spring's MessageSource with >     Zanata: https://github.com/porscheinformatik/zanata-spring That's probably beyond the capabilities we would like to introduce at this stage.

Maybe this can be added later but it would be great to have extension points for app developers in the API.

Please have a look at other framework's approaches to localization to see the needs for applications (locale fallback, pluralization, combining bundles, loading translations from the database, etc.). I've started by dev career in 1999 with implementing message bundles/i18n in our in-house framework and am still learning new stuff in this area every other month.

Cheers,

Chris

[Martin Kouba]

On 04. 08. 20 22:57, Christian Köberl wrote:
> Hi Martin!
>
> Here are some remarks to your answers.
>
> >   * There should be the ability to define all messages in a
> properties
> >     (or PO, or XLIFF) file - also the default ones. This is
> needed for
> >     translation systems to work properly.
>
> That kind of defeats the purpose of the type-safe message bundles. That
> way we wouldn't be able to validate the messages properly.
>
>
> You could just declare the message in the Java code and provide the text
> like this
>
> AppMessages.java
> @Message
> String hello();
>
> msg.properties
> helllo=Hello
>

This might work, although it could complicate things a little. E.g. we
would have to decide what happens if an interface mixed things up. I've
created https://github.com/quarkusio/quarkus/issues/17857 to track the
issue.

> With this you could use a translation tool - without that you simply can't.
>
> >     An alternative could be exporting/importing those files as a
> build step.
> Well, the interface could be generated but that would not help with
> validation.
>
>
> I thought about generating the base properties file from the Java code
> (not the other way around) so that a translation tool can use it as input.

Sure, this might be useful:
https://github.com/quarkusio/quarkus/issues/17859

>
> The problem with dots is that we use them to split expressions in Qute.
>
>
> I think that would be a nice-to-have (probably also a need-to-have when
> you migrate from another framework or use messages in a multi-app setup).

The bracket notation can be used, i.e.
msg:['app-maintenance.settings.admin-email-label']

>
> Right now, you can use basic control flow constructs to handle similar
> use cases ({#if}). In theory, nothing prevents you from using
> MessageFormat directly in your messages, but since it is not
> thread-safe
> you would need to synchronize externally.
>
>
> {#if} does not work for pluralization (see Russian for an example
> http://www.russianlessons.net/lessons/lesson11_main.php).

My point is that these constructs can be used together with nested
messages. Take a look at this test, which tries to solve the same
problem as described in
https://docs.oracle.com/javase/tutorial/i18n/format/choiceFormat.html:

https://github.com/quarkusio/quarkus/pull/17855/files#diff-18e70586ac7ef2a7a27493eb49d40800e1f3ddda2a9bb9e019ecce19c90a7168R34-R43

Of course, we should improve the UX, i.e. support multi-line messages in
the properties files. But in general, this way you can handle any
"choice" problem in one bundle without the need for some special constructs.

>
> >   * There should be the ability to adapt/extend the loading of
> messages
> >     via your own code. Most Translation systems (like Weblate

> >     <https://weblate.org/ <https://weblate.org/>>) have a REST

> API where you can get your
> >     translations on demand. With this ability the app developer
> can load
> >     the translations from the API so the translator can see the
> >     immediate effect of her translations in the running application.
> >     This is a small library doing this for Spring's MessageSource
> with
> >     Zanata: https://github.com/porscheinformatik/zanata-spring
> <https://github.com/porscheinformatik/zanata-spring>
>
> That's probably beyond the capabilities we would like to introduce at
> this stage.
>
>
> Maybe this can be added later but it would be great to have extension
> points for app developers in the API.

Yep, we could definitely add more stuff. Feel free to open issues and
send pull requests :-).

>
>
> Please have a look at other framework's approaches to localization to
> see the needs for applications (locale fallback, pluralization,
> combining bundles, loading translations from the database, etc.). I've
> started by dev career in 1999 with implementing message bundles/i18n in
> our in-house framework and am still learning new stuff in this area
> every other month.

Yes, localization and internationalization is a complex topic. But like
I mentioned before, the type-safe messages are not intended to cover all
complex i18n use cases in the world. Instead, it provides a type-safe
alternative to solve common tasks. Anyway, Quarkus is OSS so let's work
on it together and add one feature at a time ;-).

>
> Cheers,
> Chris

>
> --
> You received this message because you are subscribed to the Google
> Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to quarkus-dev...@googlegroups.com
> <mailto:quarkus-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/quarkus-dev/5e4b1238-1f5a-4dc0-94e8-d86f45c514e9o%40googlegroups.com
> <https://groups.google.com/d/msgid/quarkus-dev/5e4b1238-1f5a-4dc0-94e8-d86f45c514e9o%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Martin Kouba

[Christian Köberl]

Hi Martin!

Thx for your answer and all the good work!

As I said: I like the idea of type-safe messages but I think it should also integrate well with the translation tooling - otherwise only developers can translate their own products.

Anyway I've created a simple demo that shows different ways for i18n in Quarkus/Qute: https://github.com/derkoe/quarkus-i18n-demo (including Java and ICU message format). I will add the template syntax for pluralization with type-safe messages to the demo.

--

Christian

[Martin Kouba]

Awesome, thanks!

>
> --
> Christian

>
> --
> You received this message because you are subscribed to the Google
> Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to quarkus-dev...@googlegroups.com
> <mailto:quarkus-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/quarkus-dev/f393ced9-0843-43d1-8cc2-dd375789475an%40googlegroups.com
> <https://groups.google.com/d/msgid/quarkus-dev/f393ced9-0843-43d1-8cc2-dd375789475an%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Max Rydahl Andersen]

This repo could use a companion blog entry - Christian, you interested in making one ?

/max

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/f393ced9-0843-43d1-8cc2-dd375789475an%40googlegroups.com.

/max
https://xam.dk/about