thanks a lot for you proposal. I think we all agree that rails-i18n
can be improved and input on that is highly appreciated.
This is a very long mail, so I'll cut some of it.
On 21.04.2009, at 11:08, geekQ wrote:
> All our approaches still required extensive monkey patching resulting
> in high, unexpected efforts.
Could you please list the bits you had to monkey patch, maybe
providing some code we can look at? Or does this only refer to AR
messages?
> The solutions work in 95% of all cases, which is
> probably sufficient for most Rails applications, but in our case it
> is not.
Yes. The initial goal of the rails-i18n project was to a) provide a
common API all I18n solutions could build on while b) providing an
implementation (simple backend) that works for English. It turned out
that this implementation seems to work for (as you say) 95% of all
usecases which is much more than we expected.
We are currently seeing concurring implementations (backends) which
IMO is a good thing but doesn't necessarily mean we need to integrate
all of them into Rails core right now.
> Often software developers are not able to or are not allowed to create
> translations of their applications' texts for several reasons:
Agreed.
> Important note: non-developers need [tools][poedit] for editing
> language files!
I agree that currently a solid tool for managing large collections of
translations is missing. There are efforts to build such tools though.
(E.g. see http://github.com/newsdesk/translate)
> Identifying messages by a symbol in a YAML file (like in Rails 2.3) is
> problematic, because it breaks the developer's flow: you have to stop
> coding, come up with a good identifier (symbol) name for your
> message, go to a
> YAML file, and type in the message.
This is certainly a highly debated topic.
We are refering to this as "default translations as keys" vs "symbols
as keys". Gettext also embeds "contexts" (scopes) to the default
translation. There are several variations of these concepts and there
are pros and cons to all of them.
You name one of the problems with "defaults as keys":
> * For commercial-grade applications the exact wording usually is
> controlled by the marketing department.
"You don't want to those guys mess with your code." seems like a good
reason to use Symbols as keys.
Afaik another reason is that the initially picked default translation
might need to be changed during the process so there's a risk for keys
getting out of sync. Also, with "defaults as keys" there's no way to
compute defaults (fallbacks) (which of course affects another highly
debated topic: reusing keys). E.g. you can not fallback
to :"errors.model.invalid" when :"errors.article.invalid" is missing.
In the end we agreed to go with Symbols as keys because we felt that
they a) are a better fit for framework needs and b) provides even
better means for separating roles (i.e. devs mess with Symbols,
translators mess with translations).
There's no reason though why you could not add a helper method _() to
your application and then use the same syntax as in your examples:
> _('Archive is invalid')
> _('%{attribute} must not be empty') % attr
In Rails I18n Strings can be used as keys. The only drawback here is
that you'd need to escape/unescape dots to something else in your _()
implementation because they'd be interpreted as scope (context)
separators. (This might be an opportunity to improve the API. We've
never discussed this further.)
> With command line tools such as [msgfmt -cv][msgfmt] you can also
> check the
> well-formnedess and completeness of your transaltions as part of the
> *continuous build process*.
This again concerns the tools layer and isn't necessarily related to
the API and/or backend implementation.
Afaik these Gettext tools rely on the keys not being computed though.
E.g. devs must stick to using _('Archive is invalid') instead of
_(foo.msg), right? This obviously is a limitation that we might not
want to rely on for Rails itself. It might be a perfect fit for
userland apps though so I can't see what's holding anybody back from
using Rails I18n like this.
> ActiveRecord validations support the concept of error messages and
> full error messages. From a linguistical point of view this does not
> work: there
> is no way to infer a correct full message from its short message
> counterpart
> and vice versa. The string concatenation approach used by Rails
> (almost)
> works for English but rarely for other languages.
I'm actually not sure about the current status of this issue, but it's
been a known issue when we implemented Rails I18n. AR error messages
were subject to an ongoing discussion at that time so we simply ported
the existing functionality even though it's suboptimal.
> A message can only be translated as a whole. Hence, it should be
> possible to provide custom ActiveRecord validation messages at any
> time. For us it
> was only possible with [a dirty hack][custom-validation-messages].
Allowing Procs for AR messages seems like a good idea to me. It's not
the only place where Rails translates stuff though so it's probably
not sufficient for replacing Rails I18n with "something else" (i.e.
Gettext in your case). I feel the more appropriate way would be to use
the API and use a Gettext enabled backend instead.
> *String concatenation should never be used to create human-readable
> messages. Use string interpolation instead (as it has been used in
> other
> frameworks and platforms for decades).*
I think we all agree on this.
> Of course, a robust pluralization implementation, as provided by
> gettext, is important, too.
What's wrong or not robust with the current pluralization API derived
from CLDR?
> All the different localization libraries try to select an appropriate
> locale corresponding to their own rules and in a transparent way. The
> corresponding logic is often buried deep in the library's
> implementation and cannot
> be fixed using monkey patching.
Rails I18n does not ship locale detection/selection, so there's
nothing to monkey patch?
But yeah, you're laying out why we decided to leave features like
these to plugin land for now.
> If the handling of text messages needs to be refactored anyway, it
> would be advantageous to switch to the less invasive, proven, and
> familiar
> gettext syntax:
>
> _("The billing system is not available. Please, try again later.")
>
> instead of
>
> I18n.t(:billing_not_available)
>
> Providing context for translation:
>
> "Gadget|Title" => (German) "Bezeichnung"
Tbh I don't fully understand how this is less invasive than
t('gadget.title', :default => 'Title')
It's a bit shorter, sure, but that comes to a price, too. And you can
always add your own accessor layer/helper on top of I18n#t, no?
> The word "Title" is translated differently depending on its context.
> Hierarchical contexts are not needed, that is YAML files with deeper
> nesting as in Rails 2.3 do not make sense.
You don't need to nest your keys/scopes/contexts if you don't want to.
Even the GNU gettext manual though seems to suggest that there are
situations where this makes sense: http://www.gnu.org/software/hello/manual/gettext/Contexts.html
Am I missing your point here?
> The current interface for plugging in different localization storage
> backends is a nice intention, but in this case flexibility is not
> needed. A
> perfectly designed and working backend would be sufficient.
That's a strong statement as it suggests that there's a silver-bullet
solution for all needs. Our experience with several concurrent I18n
solutions in Rails' history rather seemed to suggest the opposite.
I believe the way forward can't be to force everbody to use Gettext
but instead make sure Rails I18n supports a full stack Gettext
solution through the API as seemlessly as possible. That might mean:
implement a Gettext backend, provide some helper syntax, maybe make
the scope separator configurable.
All that said, thanks again for bringing this up!
Sven
On 26.04.2009, at 23:35, Hongli Lai wrote:
> The remaining issues are *really* subjective.
> - Putting default translations in the code is clutter in your opinion.
I didn't say it's my opinion :) My role in the Rails I18n group was
more the one of being a moderator.
> - You view the fact that SimpleLocalization, Globalite and co are not
> designed with the Gettext style as proof that people want something
> simple and clean. The way I see it is that they haven't seriously
> tried Gettext.
I'm pretty sure they did.
> I think their view of "simple" is like coding a web
> application without using MVC - it's simpler but it gives you more
> headache down the road.
Regarding the API quite the opposite is true. You just don't have this
feature set with gettext's _(). Regarding the tools layer I agree, but
hey, if you want to use poedit for 95% of your messages you can just
do that, no? Just add fast_gettext and use it. Also, I bet a converter
that takes a flat yaml translations file and converts it to po should
not be that hard to do.
> I find it "interesting" that pretty much all
> open source desktop applications use Gettext. Gettext has been used to
> translate hundreds, if not thousands, of desktop applications to
> dozens of languages. Yet the web applications world seems to
> completely ignore Gettext. For PHP I can understand, everybody's
> reinventing the wheel there. But Rails?
Look at the history of Rails. There were tons of concurring
implementations, Gettext being one of them. Gettext hasn't been able
to win the race in any way and I think that's for a reason.
Also, we haven't reinvented the wheel. We've extracted what we (based
on the experience of several implementors) believed the best ideas are.
> Your idea regarding the computability of symbols is interesting. On an
> abstract level it does seem to fit within the Rails philosophy, but it
> remains to be seen how useful it is in practice and whether anyone can
> come up with a good implementation.
Hm? People are doing stuff like this.
flash[:notice] =
t(:"flash.#{controller_name}.#{action}.success", :default
=> :"flash.#{action}.success")
Do that in gettext. Obviously, flash messages are only one place where
computability of keys is quite useful.
Again, there was a reason why so many people weren't happy with
gettext before and invented their own APIs for years.
> In any case, what is clear that at the very least, Rails should have
> better I18n tools. There should be tools that alert translators which
> translations need to be updated, how many strings still need
> translations, for writing the translations, etc.
I agree.
Aside from that though I think some thought should be put into how
integrate a gettext style accessor _('foo') and a gettext backend.
On 27.04.2009, at 18:02, geekQ wrote:
> the most important question is, whether the core team would sacrifice
> *some* parts of humanize, pluralize and other string concatenation
> voodoo, especially in ActiveRecord to allow for 100% linguistically
> correct translations and smooth, enterprise-ready localization
> workflow.
Exactly which parts are you referring to?
> Currently we have to put a lot of effort into monkey patching to work
> around the opinionated decisions baked in into Rails.
Again, it would be great if you could list the exact places that you
found need monkeypatching.
> Missing possibility of changing the default string has never been an
> issue,
> neither in my personal decade of writing international applications
> (different open source platforms, Microsoft.net and pre-dot-net) nor
> for big
> open source projects with longer history.
It has been an issue which is why people implemented key based
solutions.
>> It turned out
>> that this implementation seems to work for (as you say) 95% of all
>> usecases which is much more than we expected.
> So this solution does not qualify for any serious enterprise or
> governmental (European Union) application. 100% linguistically correct
> translations are required. 95% is much less that is expected from us.
Right. Which is why we have a pluggable backend so you can implement
your needs in plugin land. If you need patching to core, please list
the places that need patching. If you need changes to the API, please
do so, too.
> All the kinds of hierarchically organized scopes (computed keys)
> and (optional) translation inheritance do not work in environment
> with role separation. Inheritance and method overriding
> work in OOP. But it does not work for translations.
> A translation agency needs a comprehensive and flat list
> of strings to be translated. To be able to make a
> decision about to override or not to override or where to override
> they would need to analyse the application source code. Only manually
> created and obligatory translation scope makes sense.
Maybe they don't make sense for the most part of translation agencies.
That doesn't mean they don't make sense for the rest.
>> People wanted a simple and clean API, they explicitely did not
>> want to mess with Gettext which was designed, let's face it, in
>> 1994 for C. It feels old and awkward to many.)
> And since then adapted for 20 different programming languages.
> Bindings
> for dynamic language, e.g. Python are very nice. Same API can be used
> for Ruby too.
Yeah, still asuming a C'ish API and compilation stage though.
> * pluralization rules are different for different languages,
> gettext uses a formula in a programming language per language for
> that
> * some languages have 3 or 5 plural forms as opposite to 2 in English
> and German
> * only complete sentence can be pluralized, not a single word
Yup. The API covers that.
> The question remains, whether this really is the direction towards
> Rails
> is heading. If so, we would contribute a solid Gettext based I18n
> implementation that addresses the aforementioned issues. This however
> requires some breaking changes within the Rails core and a consensus
> about the necessity of them being addressed.
I believe this ship has sailed about 1 year ago. It's not the question
anymore whether or not we want that API. The question is if everybody
who has good ideas rolls up their sleeves and implements them *using*
this common API. If you want to do that for Gettext I'm absolutely
sure the community will welcome that with big applause.
Btw I've just pushed some experiments with gettext'ish accessors on
top of Rails I18n:
http://github.com/svenfuchs/i18n/tree/gettext
You might particularly want to look at the helper layer and the tests:
http://github.com/svenfuchs/i18n/blob/49220ce667fe542041c97bd38a0190e27a9581d6/lib/i18n/gettext.rb
http://github.com/svenfuchs/i18n/blob/49220ce667fe542041c97bd38a0190e27a9581d6/test/gettext_test.rb
For a fullstack gettext support that uses the Rails I18n API there
seem to be three things missing:
- complete the helpers (trivial)
- implement a gettext backend (anybody?)
- figure out a gettext'ish way to announce expected translations for
computed keys
Any help and/or feedback would be appreciated!
Nice.
I'd use this even without having gettext as the backend.
With helpers you refer to the ability to accept named arguments? Would
you add that to the _ method or would you do it ruby-gettext style by
extending String with a % method?
(I'd prefer _ to accept the arguments directly, saves polluting the
String object)
Cheers,
Lawrence
On 28.04.2009, at 08:06, Lawrence Pit wrote:
> I'd use this even without having gettext as the backend.
heh :)
> With helpers you refer to the ability to accept named arguments? Would
> you add that to the _ method or would you do it ruby-gettext style by
> extending String with a % method?
No idea, I was just checking this out for some kind of proof of concept.
> (I'd prefer _ to accept the arguments directly, saves polluting the
> String object)
Sure. I guess the question would be whether one wants to rebuild the
exact gettext api with all of its C'ish methods (sgettext, pgettext,
psgettext, ngettext, nsgettext, ...) or not.
Vladimir,
Sven has tried repeatedly to get specifics out of you throughout this
thread. Until this message there's been nothing but vague statements
and rehashing of discussions which came to conclusion months ago.
Sven and the rails-i18n team *do* grasp the issues that you've
mentioned and have their i18n patches applied straight to rails. The
guys on that list are responsible for directing the rails i18n effort
and we listen to them and take their patches. You've cleaerly
identified a few key points where the existing i18n api is lacking,
and the ActiveRecord code is inflexible. Let's address those issues,
and the right place to do those is the rails-i18n list and sven and co
are the ones to talk with.
Rather than throwing your toys out of the cot and feeling
self-satisfied in the superior enterprisiness of your approach, you
should try to work with Sven and the team to iron out all the issues
with the existing api and let everyone benefit from the amount of work
you've clearly put into this. If you're genuinely interested in
enabling 'true gettext' support and removing the string
concatenations in the validations API, then it will be surely be a few
small, targeted patches.
If on the other hand you're looking to dump wiki markup into mailing
list threads and talk dismissively about the work of other
programmers, then perhaps you should do that elsewhere.
We're all working towards the same goal here, just because you've
found some shortcomings doesn't mean that the people who did the
existing work are evil or clueless.
--
Cheers
Koz
On 30.04.2009, at 10:27, geekQ wrote:
> The problem is best visible in the following line:
>
> http://github.com/rails/rails/blob/09a976ac58d2d7637003b92d51637f59f647b53a/activerecord/lib/active_record/validations.rb#L207
>
> full_messages << attr_name +
> I18n.t('activerecord.errors.format.separator', :default => ' ') +
> message
>
> The counterpart in Rails3 is
> http://github.com/rails/rails/blob/bab2bfa69220ca1b6c7b56dccc79cf8e41245306/activemodel/lib/active_model/errors.rb#L65
>
> errors_with_attributes << (attribute.to_s.humanize + " " + error)
Great, thanks for pointing that out. This is a known issue and I agree
that we should get that fixed. There are few options to do that and
discussion about that has already started over at http://groups.google.com/group/rails-i18n
Please join in! We're keen on hearing your opinions.
> The second issue with ActiveRecord validations is using custom
> messages. Gettext can not be used at this place without monkey-
> patching,
> that adds lambda support.
Integrating lamda support to the I18n API has been a request for a
long time. It's also useful for localizing dates to rather funky rules
and such.
I've worked with Clemens yesterday on integrating and polishing his
contributions and pushed it to a branch: http://github.com/svenfuchs/i18n/commits/lambda
So this should then be possible:
validates_format_of :account, :messages => lambda { _("foo") }
Another option to solve this situation might be:
validates_format_of :account, :messages => gettext_noop("foo").to_sym
This is also being discussed on the rails-i18n list. Please let us
know about your opinion.
> = Known Monkeys
>
> * in our project we monkey patched as follows
> http://blog.geekq.net/2009/04/09/i18n-remove-validation-message-prefix/
>
> * Masao Mutoh pointed out, that we do not need any monkey patching,
> if we use N_ from his gettext library because he has already
> monkey patched everything.
>
> * more monkey patching from masao
> http://github.com/mutoh/*
>
> * following library also overrides the full_messages()
> http://github.com/yaroslav/russian/blob/7960596ede5159462c41d5dcd07b137953bf1b3d/lib/russian/active_record_ext/custom_error_message.rb
Great list, this is helpful. Thanks!
>>> * pluralization rules are different for different languages,
>>> gettext uses a formula in a programming language per language for
>>> that
>>> * some languages have 3 or 5 plural forms as opposite to 2 in
>>> English
>>> and German
>>> * only complete sentence can be pluralized, not a single word
>>
>> Yup. The API covers that.
>
> Did not find documentation for that in activesupport-2.3 / I18n.
> Now I've found some hints in the current Rails guide.
>
> But people are still forced to do a lot of programming per language,
> like in
> http://github.com/yaroslav/russian/blob/7960596ede5159462c41d5dcd07b137953bf1b3d/lib/russian/backend/advanced.rb
Sure. Please distinguish the API from their implementations
(backends). This is on purpose.
There are a few more backend implementations in Globalize2:
http://github.com/joshmh/globalize2/tree/a46ab1e885c37aff435823d992cc8c919b0e3c50/lib/globalize/backend
Please think about the I18n API in Rails as similar to the Rack API
support. Rack allows for previously unseen extensibility and
exchangeability of concurrent implementations of rather focussed
features.
Now, even though Rails 2.x now supports that API it doesn't leverage
all of the features it provides. E.g. Rack routing/url_generation is
not supported, yet (will be there in Rails 3, afaik). Nobody's arguing
Rails should stop supporting Rack for this reason though. And similar
the fact that Rails does not perfectly support all features required
for proper I18n/L10n does not mean it should stop supporting the I18n
API.
>> I believe this ship has sailed about 1 year ago. It's not the
>> question
>> anymore whether or not we want that API. The question is if everybody
>> who has good ideas rolls up their sleeves and implements them *using*
>> this common API. If you want to do that for Gettext I'm absolutely
>> sure the community will welcome that with big applause.
> Could you point to at least one complete backend implementation,
> that is entirely based on the Rails.I18n public API, without the
> need for extensive monkey patching?
If by "extensive monkey patching" you mean the bug/shortcoming in
AR#full_messages then, no.
> No, it is not possible to implement serious Gettext or serious
> internationalization on the basis of Rails I18n API, that is why
>
> - gettext_rails is a pure monkey patch solution, without any usage
> of the mentioned API
> - Michael Grosser's fast_gettext does not use the mentioned API
> in any way
> - Sam Lown's i18n_gettext is a Rails plugin, that simply wraps
> the Masao's library and uses it as a fallback in addition to
> the Rails simple backend. i18n_gettext is not a stand alone
> internationalization solution
Yeah, I know these are different approaches from what you have in mind.
I've listed them because I have received some angry private messages
that were based on the perception I wouldn't know about or conceal or
downplay these efforts. Just wanted to make sure people know that I
don't, these are great contributions.
Thanks again!
I am happy to hear, that we totally agree on the important items.
We should tackle the problems, as you are describing:
1. use string interpolation instead of concatenation everywhere
> Great, thanks for pointing that out. This is a known issue and I agree
> that we should get that fixed. There are few options to do that and
> discussion about that has already started over at
> http://groups.google.com/group/rails-i18n
2. introducing lambda support for error messages for maximum flexibility
> Integrating lamda support to the I18n API has been a request for a
> long time. It's also useful for localizing dates to rather funky rules
> and such.
3. discuss/improve the API. My opinion always was, that supporting
different *storage* backends is a good idea. Every developer is
comfortable with yaml files. Others can use gettext
specific .mo or .po (bypassing, as you pointed out,
the dated compilation approach). On the other hand, some things
should not be optional and can not be plugged through the API,
but lets discuss this later, after we succeeded with the first
two things.
> Please think about the I18n API in Rails as similar to the Rack API
> support. Rack allows for previously unseen extensibility and
> exchangeability of concurrent implementations of rather focussed
> features.
This is an excellent example!
It illustrates two things: successful technical design and importance of
experience and solution maturity. Instead of reinventing the wheel, the
Ruby community adopted successful and proven solution from the Python
world, where it is known under the name of WSGI, and further improved it.
I was talking about the gettext whole the time not because I admire the
obscure .mo file format
http://www.gnu.org/software/gettext/manual/gettext.html#MO-Files , but
because the folks at GNU have already seen all the possible problems and
addressed them in the design, the tools, and the best practices.
http://www.gnu.org/software/gettext/manual/gettext.html#Why
Best Regards and see you
on the http://groups.google.com/group/rails-i18n shortly,
Vladimir