rails-i18n fullstack gettext support?

[Sven Fuchs]

There's been a discussion about better Gettext support for Rails in
the rails-core mailing list: http://groups.google.com/group/rubyonrails-core/browse_thread/thread/e9e219ff318fa6e7

To prove that fullstack Gettext support using the Rails I18n API is
possible and the way to go I've played with an experimental Gettext
helper layer and backend: github.com/svenfuchs/i18n/tree/gettext

As I don't have a real usecase for this right now I'd like to ask
Gettext folks to help improving this. :)

The goal basically is:

- have standard gettext'ish helpers like _(), n_() etc.
- have a backend that reads po/mo files and stores them in the
standard translation format
- somehow announce expected translations for computed keys in a
gettext'ish manner (so source code parsers can pick them up)

That would use the standard API both on the input and output side and
thus allow us to combine this with other tools like a caching layer or
whatever else.

So, any takers?

[Masao Mutoh]

Hi Sven,

I read "Enterprise Rails" thread.

IMO,
There are 3 areas to consider about localization.

(a) Rails internal localization
* numeric/date/..
* messages
* locale handling
-> There is a Simple backend to use, don't need other
backend, don't need pluggable.
(b) Application localization
-> People can select l10n libs. Simple backend or others
such as gettext
(c) Plugins/Engines localization
-> People can select l10n libs. Simple backend or others
such as gettext

(b), (c) is ignorable even if (a)'s "module I18n" doesn't have
perfect feature.
# But most important thing is I18n don't break (b)(c) l10n libs.
# So I highly recommand not to have gettext APIs in rails-i18n.

Then, in (a) area, the considerations are:
* Linguistically correct translations (by geekQ)
Every language should be supported.
1. The sentence may be RTL not LTR
2. Subject may not be the first or omiitable.
3. Sentence isn't separate with white space

e.g.) title + " is blank" is bad.
It should be "{{title}} is blank".
It will be translated to "blankis{{title}}" in a language.
This is well known issue to develop I18n apps.
* Translate messages lazily as possible.
1. Class/Module level localization (by geekQ)
validates_format_of account, :message => I18n.t("foo")
This "foo" can't be localized correctly, can it?
2. Calling same methods in different languages
I18n.t = :en
a.foo #=> English
I18n.t = :ja
a.foo #=> Japanese
AFAIK, ActiveRecord errors (full_messages/on) can't do this

* Improve I18n simple backend
IMO, Only simple backend should be the backend of (a) area.
Other backend is not needed. Because others are used in (b)(c).
The issuei is the speed because it's too much slow.

If these thing (or some other person recommanded) are supported whole
rails((a) area), people can use their own l10n libs(such as gettext) freely
in (b)(c) area and they don't have any complaints.

gettext developers have complaints because of a lot of monkey patching.
But (a)area is improved, the monkey patching will be reduced.

HTH,

--
Masao Mutoh <muto...@gmail.com>

[mateo]

Hi,

Just coming over from the other thread...

The attribute name concatenation issue is one that is frequently
mentioned, but it seems to me that the solution is trivial; since
error translations already support attribute interpolation, simply
change the default error messages to include {{attribute}} at the
beginning, and turn off concatenation. For backwards compatibility, a
config option could be added to turn concatenation back on, or even a
"smart" mode could be implement that only concatenates if
{{attribute}} isn't found in the translation value. I can write a
patch with tests for this, if you'd like.

On Apr 30, 1:54 am, Masao Mutoh <mutom...@gmail.com> wrote:

>       1. Class/Module level localization (by geekQ)
>            validates_format_of account, :message => I18n.t("foo")
>            This "foo" can't be localized correctly, can it?

If you pass a symbol as message, it will use that as a key to look up
the appropriate translation in the activerecord.errors scope.

You can see the relevant docs here: http://api.rubyonrails.org/classes/ActiveRecord/Errors.html

This should be sufficient for your needs, no?

[Masao Mutoh]

Hi,

On Thu, 30 Apr 2009 20:49:47 -0700 (PDT)
mateo <mateo....@gmail.com> wrote:

>
> Hi,
>
> Just coming over from the other thread...
>
> The attribute name concatenation issue is one that is frequently
> mentioned, but it seems to me that the solution is trivial; since
> error translations already support attribute interpolation, simply
> change the default error messages to include {{attribute}} at the
> beginning, and turn off concatenation. For backwards compatibility, a
> config option could be added to turn concatenation back on, or even a
> "smart" mode could be implement that only concatenates if
> {{attribute}} isn't found in the translation value. I can write a
> patch with tests for this, if you'd like.

Yeah, great.

> On Apr 30, 1:54 am, Masao Mutoh <mutom...@gmail.com> wrote:
>
> >       1. Class/Module level localization (by geekQ)
> >            validates_format_of account, :message => I18n.t("foo")
> >            This "foo" can't be localized correctly, can it?
>
> If you pass a symbol as message, it will use that as a key to look up
> the appropriate translation in the activerecord.errors scope.
>
> You can see the relevant docs here: http://api.rubyonrails.org/classes/ActiveRecord/Errors.html
>
> This should be sufficient for your needs, no?

Seems nice, but I want to write :message as the message not symbol.
If there is a hook method(for translation) when full_messages/on/... calls,
it can be fixed (work with both of symbol/message).
# Please note what I want to say is "lazy localization support". This is an example.

> >


--
Masao Mutoh <muto...@gmail.com>

[Sven Fuchs]

Hi Masao,

thanks a lot for getting involved here, too!

Please don't worry about anybody breaking stuff by publishing this. It
would break your libraries when it's release as part of the I18n gem
and enabled by default. We'll definitely not do that. Nobody's
interested in making things harder for other people.

But concurring implementations aren't necessarily a bad thing as it
allows us all to look at different solutions from different angles and
learn from that. Just like there are at least three different
approaches to Gettext support in the open right now: your libs, Sam
Lown's backend and Michael's fast_gettext. They certainly solve pretty
different needs and that's a good thing IMO.

I have a few notes:

# AR messages helpers

I totally agree with your and Vladimir's concerns with the
full_messages helper. (Thanks for your great examples!)

We should fix that rather soon. We've been left it as is because we
were told that there were other people working on it (we have to find
out what happened to that) and that this helper is meant to be
reimplemented for special needs anyway. So one could argue that this
helper only provides some default functionality and should be left as
is for backwards compat reasons (we always have to keep in mind that
there are tons of Rails users who aren't interested in I18n/L10n at
all). But I'm sure the Rails core team is open to fixing this helper
in Core, so let's take a look at that.

Similar to these helpers I agree that there's a ton of improvements
possible in what you call area A, mostly regarding number/time/
date/... localization, not so much regarding translations.

# Pluggable backends

I do not agree that we do not need pluggable backends. I see the
abstraction that the I18n API provides as similar to the Rack API. It
is well defined and thus enables a so far unknown power of
extensibility. That does not mean one absolutely must use it, but it
means it's a good thing for Rails to support it. The fact that Rails
does not yet use Rack for routing and url_generation (afaik it will in
Rails 3.0) does not mean Rails should stop supporting the Rack API.
And similar the fact that Rails does not perfectly support all
features required for proper I18n/L10n does not mean it should stop
supporting the I18n API.

The benefits we get through using the API is that in the end we have

- better extensibility and exchangeability
- more targeted and focussed feature sets
- thus: less work, less bugs, better tools, better data ...

# Different approaches

Obviously we have a few different approaches to integrate Gettext with
Rails I18n. I do not agree with that we must pick one of them. Instead
we should try to support all of them as smoothly as possible because
all of them target different needs. (I'm referring to Gettext as one
example of third-party solutions.)

1. Rails I18n for Rails core, Gettext for everything else not using
the I18n API
2. Rails I18n with a Gettext backend that falls back to the Simple
backend
3. Rails I18n with a pure Gettext backend

As far as I understand it (please correct me if I'm wrong) the
following libraries take these approaches:

1. rails-gettext (your library) and fast_gettext (Michael Grosser)
2. i18n_gettext (Sam Lown)
3. (there's no solid implementation)

Let me say clearly that I see all of these approaches as perfectly
valid. They obviously solve problems for people so in my opinion
there's no point in arguing that away.

Now one of the reasons for geekQ to post their proposal that sparked
the discussion on rails-core was that they want to manage *all* of
their translations through Gettext (e.g. in order to enable
translators to use poedit everywhere). It seems to me that there is no
other way to fullfil that requirement to either implement approach 3)
or just throw away Rails I18n, go back two years and try to implement
pure Gettext right into Rails core (which isn't an option). And that's
the reason for my previous experiments :)

[Sven Fuchs]

Hi Mateo,

thanks for joining the discussion here :)

On 01.05.2009, at 05:49, mateo wrote:
> The attribute name concatenation issue is one that is frequently
> mentioned, but it seems to me that the solution is trivial; since
> error translations already support attribute interpolation, simply
> change the default error messages to include {{attribute}} at the
> beginning, and turn off concatenation.

Yeah, I agree with that. But I also agree with the sentiment that the
implementation of the full_messages helper is just wrong from the
perspective of proper I18n, we just should not concatenate anything
like that in Rails core.

> For backwards compatibility, a
> config option could be added to turn concatenation back on, or even a
> "smart" mode could be implement that only concatenates if
> {{attribute}} isn't found in the translation value. I can write a
> patch with tests for this, if you'd like.

The "smart mode" doesn't sound too bad to me, but the API currently
does not allow to check whether or not a value was interpolated to it.
So no matter how that's solved, it'd be a bit ugly.

Not sure, but why not not control this solely through the availability
of translations?

[Sven Fuchs]

On 01.05.2009, at 11:33, Masao Mutoh wrote:
>>> 1. Class/Module level localization (by geekQ)
>>> validates_format_of account, :message => I18n.t("foo")
>>> This "foo" can't be localized correctly, can it?
>>
>> If you pass a symbol as message, it will use that as a key to look up
>> the appropriate translation in the activerecord.errors scope.
>>
>> You can see the relevant docs here: http://api.rubyonrails.org/classes/ActiveRecord/Errors.html
>>
>> This should be sufficient for your needs, no?
>
> Seems nice, but I want to write :message as the message not symbol.
> If there is a hook method(for translation) when full_messages/on/...
> calls,
> it can be fixed (work with both of symbol/message).
> # Please note what I want to say is "lazy localization support".
> This is an example.

Can you explain why you want to write out the message? Because you
want to use a gettext source parser for identifying your messages? Or
is there another reason?

I've had a look at this issue yesterday.

For one thing with Clemens' Lambda support it should now already be
possible to do:

validates_format_of :account, :messages => lambda { _("foo") }

That should make both source parsers happy and work as expected.

Another option to solve this situation might be:

validates_format_of :account, :messages => gettext_noop("foo").to_sym

WDYT?

[Masao Mutoh]

Hi,

On Fri, 1 May 2009 13:52:41 +0200
Sven Fuchs <sven...@artweb-design.de> wrote:

>
> Hi Masao,

> # Different approaches
>
> Obviously we have a few different approaches to integrate Gettext with
> Rails I18n. I do not agree with that we must pick one of them. Instead
> we should try to support all of them as smoothly as possible because
> all of them target different needs. (I'm referring to Gettext as one
> example of third-party solutions.)
>
> 1. Rails I18n for Rails core, Gettext for everything else not using
> the I18n API
> 2. Rails I18n with a Gettext backend that falls back to the Simple
> backend
> 3. Rails I18n with a pure Gettext backend
>
> As far as I understand it (please correct me if I'm wrong) the
> following libraries take these approaches:
>
> 1. rails-gettext (your library) and fast_gettext (Michael Grosser)
> 2. i18n_gettext (Sam Lown)
> 3. (there's no solid implementation)
>
> Let me say clearly that I see all of these approaches as perfectly
> valid. They obviously solve problems for people so in my opinion
> there's no point in arguing that away.

1. is correct.
I don't know i18n_gettext well, but exactly, 2, 3 approach is needless.
We don't need that anymore.

> Now one of the reasons for geekQ to post their proposal that sparked
> the discussion on rails-core was that they want to manage *all* of
> their translations through Gettext (e.g. in order to enable
> translators to use poedit everywhere). It seems to me that there is no
> other way to fullfil that requirement to either implement approach 3)
> or just throw away Rails I18n, go back two years and try to implement
> pure Gettext right into Rails core (which isn't an option). And that's
> the reason for my previous experiments :)

He may think so.
But in my opinion, the real problem was monkey patches of gettext.
The reason why gettext have monkey patches is ruby-core's i18n
support has been poor yet. But it can improve with current i18n.

--
Masao Mutoh <muto...@gmail.com>

[Erkki Eilonen]

This might still be relevant:
http://groups.google.com/group/rails-i18n/browse_thread/thread/b7268565a891f019/3a0d4be5e871364b?lnk=gst&q=erkki#3a0d4be5e871364b

We didn't get anywhere with this and currently just use an overridden
version of full_messages
but the attribute issue has been discussed previously.

Erkki

[Masao Mutoh]

Hi,

One of the reason of it is fit is gettext.
But don't think it's "only gettext problem". It's better to support both
style in this area.

Other reasons:
1. It's not beautiful to use to_sym or lamda.
Especially, symbol can't solve anything here for gettext.
2. It can't solve my "2. Calling same methods in different languages "
problem.

If ActiveRecord::Errors#generate_message is called
in #full_messages/#on/#each not #add, all problems are fixed.
It's not difficult and also gettext already has this kind of the monkey patch.

--
Masao Mutoh <muto...@gmail.com>

[mateo murphy]

@Sven Fuchs

> Not sure, but why not not control this solely through the availability
> of translations?

Well, I think that ideally concatenation should be removed completely.
Even in english it forces you to write your error messages in a way
that isn't always natural. I think ideally it would be deprecated, but
supported for backwards compatibility, and with 3.0 coming up, now is
probably the best time to do it...

@Masao Mutoh:

> If ActiveRecord::Errors#generate_message is called
> in #full_messages/#on/#each not #add, all problems are fixed.
> It's not difficult and also gettext already has this kind of the monkey patch.

Right, I had trouble understanding what you meant by "lazy
localization support", but this makes it clear

[Sven Fuchs]

Thank you, Erkki!

I've just wanted to ask the list where this discussion was :)

I guess the solution you proposed at that point is the same as what
Mateo proposes as a "smart mode", right?

So in that case AR would still concat strings but only as a default
fallback when no translation that uses interpolation for the attribute
name was used.

[Erkki Eilonen]

Correct, but at that point I was concerned with backwards
compatibility
without any configuration options.

In the light of rails 3 or some i18n gem if we could do this as an
I18n configuration option and lose string concatenation that would be
awesome!

I also remember I figured out some edge cases where the "smart" mode
does the wrong thing
but I'll be damned if I remember any of them right now. Maybe someone
else can pick up
the string of thought :)

On May 2, 2:00 pm, Sven Fuchs <svenfu...@artweb-design.de> wrote:
> Thank you, Erkki!
>
> I've just wanted to ask the list where this discussion was :)
>
> I guess the solution you proposed at that point is the same as what  
> Mateo proposes as a "smart mode", right?
>
> So in that case AR would still concat strings but only as a default  
> fallback when no translation that uses interpolation for the attribute  
> name was used.
>
> On 01.05.2009, at 15:22, Erkki Eilonen wrote:
>
>
>
> > This might still be relevant:

> >http://groups.google.com/group/rails-i18n/browse_thread/thread/b72685...