Implementing the CLDR for Ruby/Rails
84 views
Skip to first unread message
Clemens Kofler
Sep 7, 2008, 2:58:20 PM9/7/08
to rails-i18n
Hi there,
some of you may be aware of Unicode's Common Locale Data Repository
(CLDR) project and their XML dialect Locale Data Markup Language
(LDML). If not, you can read up on it at http://unicode.org/cldr/.
Here's the thing: I think implementing a way to use the CLDR in Ruby/
Rails is would probably be a good, if not the best, idea to
internationalize both, Ruby and Rails because it seems to be the
standard when it comes to localizing. For simple overview, here's a
couple of things that the current LDML version provides:
- translated languages - so you can look up what Japanese means in
German (to save you the trouble of looking it up - it's
"Japanisch" ;-))
- script names
- country names
- date and time formats (including time zones)
- number formatting (including currencies)
- fallback rules for non-ASCII characters (like "ae" for the German
"ä")
- pluralization rules (although I don't quite get them yet)
- phone codes
- and probably much more stuff that I haven't discovered just yet.
I don't know about you but to me this sounds like most of the stuff
that's used in your everyday localization efforts.
The "problem" is that implementing this stuff with a well-thought-
through API is probably quite a bitch to do. I googled this stuff a
couple of days ago and it seems that hardly anyone bothered doing it
so far - so either it's not that important, after all, or it's really
hard to get it working. Pretty much the only implementations I've
found in _any_ language (not only Ruby) are Ruby-Gettext author Masao
Mutoh's Ruby-CLDR project (http://www.yotabanana.com/hiki/ruby-
cldr.html) which seems to have only one version out and it dates back
to 2006 (plus, like Ruby-Gettext, the documentation isn't all that
great) and PHP localize (http://sourceforge.net/projects/localize)
which was created on SourceForge in 2005 and never left its planning
stage.
Here's what I'd like to see in an implementation:
- A way to parse Unicodes LDML files so you can keep your locale data
up to date.
- Some way to efficiently handle the data. The XML files usually
amount to a couple megabytes in size - definitely not something you
want to keep in memory all the time if you're running a site with a
couple dozen localizations. I'm thinking of a database backend and
maybe even memcache here.
- Some kind of cool API to wrap the guts of this stuff and make it
easily accessible. I think, the Rails i18n project serves as a great
example when it comes to being unobtrusive and slim, yet effective.
- Of course, in the long run, some way to integrate it with Rails.
I really think I want to tackle this but I definitely can't do it all
by myself. I'd really like to create some kind of workgroup here
(again, like the i18n team did) and merge our skills and knowledge for
this complex project. I'd really love if everyone that's interested
(or knows someone who might be interested) leaves a message here and
afterwards - if there's enough people interested - we could meet in
IRC and discuss how to approach this and what the goals for this
project should go.
I'd love to hear your opinions about it - also if you think this whole
thing isn't worth the fuzz.
Best,
- Clemens
some of you may be aware of Unicode's Common Locale Data Repository
(CLDR) project and their XML dialect Locale Data Markup Language
(LDML). If not, you can read up on it at http://unicode.org/cldr/.
Here's the thing: I think implementing a way to use the CLDR in Ruby/
Rails is would probably be a good, if not the best, idea to
internationalize both, Ruby and Rails because it seems to be the
standard when it comes to localizing. For simple overview, here's a
couple of things that the current LDML version provides:
- translated languages - so you can look up what Japanese means in
German (to save you the trouble of looking it up - it's
"Japanisch" ;-))
- script names
- country names
- date and time formats (including time zones)
- number formatting (including currencies)
- fallback rules for non-ASCII characters (like "ae" for the German
"ä")
- pluralization rules (although I don't quite get them yet)
- phone codes
- and probably much more stuff that I haven't discovered just yet.
I don't know about you but to me this sounds like most of the stuff
that's used in your everyday localization efforts.
The "problem" is that implementing this stuff with a well-thought-
through API is probably quite a bitch to do. I googled this stuff a
couple of days ago and it seems that hardly anyone bothered doing it
so far - so either it's not that important, after all, or it's really
hard to get it working. Pretty much the only implementations I've
found in _any_ language (not only Ruby) are Ruby-Gettext author Masao
Mutoh's Ruby-CLDR project (http://www.yotabanana.com/hiki/ruby-
cldr.html) which seems to have only one version out and it dates back
to 2006 (plus, like Ruby-Gettext, the documentation isn't all that
great) and PHP localize (http://sourceforge.net/projects/localize)
which was created on SourceForge in 2005 and never left its planning
stage.
Here's what I'd like to see in an implementation:
- A way to parse Unicodes LDML files so you can keep your locale data
up to date.
- Some way to efficiently handle the data. The XML files usually
amount to a couple megabytes in size - definitely not something you
want to keep in memory all the time if you're running a site with a
couple dozen localizations. I'm thinking of a database backend and
maybe even memcache here.
- Some kind of cool API to wrap the guts of this stuff and make it
easily accessible. I think, the Rails i18n project serves as a great
example when it comes to being unobtrusive and slim, yet effective.
- Of course, in the long run, some way to integrate it with Rails.
I really think I want to tackle this but I definitely can't do it all
by myself. I'd really like to create some kind of workgroup here
(again, like the i18n team did) and merge our skills and knowledge for
this complex project. I'd really love if everyone that's interested
(or knows someone who might be interested) leaves a message here and
afterwards - if there's enough people interested - we could meet in
IRC and discuss how to approach this and what the goals for this
project should go.
I'd love to hear your opinions about it - also if you think this whole
thing isn't worth the fuzz.
Best,
- Clemens
Joshua Harvey
Sep 7, 2008, 3:10:07 PM9/7/08
to rails...@googlegroups.com
Glad you brought that up, Clemens. CLDR has been on our radar for a while, and integrating parts of it into Globalize2 is in our plans. That said, it would be even better to do a general Ruby project that's not limited to Globalize. I'd definitely be interested in getting together with a few people and working on this.
-- Josh
Sven Fuchs
Sep 7, 2008, 3:18:49 PM9/7/08
to rails...@googlegroups.com
Cool. Count me in as well.
Also, there was a guy from Merb at RailsConf who asked me about CLDR
and wanted to ask on this list about it. (I so suck so much at
rembering names ... I really should find some external solution for
that. Polaroid maybe?)
Also, there was a guy from Merb at RailsConf who asked me about CLDR
and wanted to ask on this list about it. (I so suck so much at
rembering names ... I really should find some external solution for
that. Polaroid maybe?)
Yaroslav Markin
Sep 7, 2008, 3:30:26 PM9/7/08
to rails...@googlegroups.com
Count me in.
Also, it seems that a full-blown objective implementation of CLDR in scope of I18n/Rails may be an overkill, what we need is probably a simple converter from LDML to YAML/Ruby hashes for use in I18n. What do you think? Seems that pretty much all functionality of CLDR can just serve as hash data for I18n translations table and all we have is to agree on key naming and experiment with backends.
On Sun, Sep 7, 2008 at 11:18 PM, Sven Fuchs <sven...@artweb-design.de> wrote:
Cool. Count me in as well.
Joshua Harvey
Sep 7, 2008, 3:44:53 PM9/7/08
to rails...@googlegroups.com
Yeah, a lot of it can be Ruby or YAML translation files for i18n.rb. The other part that would be really useful is the pluralization rules, which could be mapped to a Ruby module with lambdas or something. There might also be some good currency and number localization stuff.
One possibility is to create a group of rake tasks that would generate the Ruby/YAML files, and distribute the rake tasks as a gem, but also distribute the resulting files themselves, so that you'd be able to just require them into your app.
-- Josh
Yaroslav Markin
Sep 7, 2008, 3:53:28 PM9/7/08
to rails...@googlegroups.com
On Sun, Sep 7, 2008 at 11:44 PM, Joshua Harvey <jos...@gmail.com> wrote:
Yeah, a lot of it can be Ruby or YAML translation files for i18n.rb. The other part that would be really useful is the pluralization rules, which could be mapped to a Ruby module with lambdas or something. There might also be some good currency and number localization stuff.
Well, I still think that storing rules as lambdas is the simplest thing possible. We'll need to build a simple LDML pluralization rule parser.
One possibility is to create a group of rake tasks that would generate the Ruby/YAML files, and distribute the rake tasks as a gem, but also distribute the resulting files themselves, so that you'd be able to just require them into your app.
Exactly what I was thinking -- a CLDR gem may include:
* CLDR source (LDML release fro unicode.org),
* Libraries and rake tasks to parse it into .rb files with ruby hashes or YAML where possible, and
* Files that are produced from CLDR that can be easily included from Rails or any app using I18n.
Joshua Harvey
Sep 7, 2008, 4:03:29 PM9/7/08
to rails...@googlegroups.com
As long as installing the gem doesn't install the CLDR files. There should be a rake task that downloads the latest version of CLDR.
Clemens Kofler
Sep 7, 2008, 4:27:21 PM9/7/08
to rails-i18n
> As long as installing the gem doesn't install the CLDR files. There should
> be a rake task that downloads the latest version of CLDR.
I was thinking along these lines, too. One could also create a rake
> be a rake task that downloads the latest version of CLDR.
task like rake cldr:current that checks with the web site if the CLDR
is up to date.
> > Well, I still think that storing rules as lambdas is the simplest thing
> > possible. We'll need to build a simple LDML pluralization rule parser.
to what i18n did, i.e. we implement the stuff in Ruby and then provide
a way to easily integrate it with Rails. Anyway, I don't think this is
something that really has to be decided right this instant.
Glad you're all interested!
Pavel Kunc
Sep 9, 2008, 12:06:47 PM9/9/08
to rails-i18n
I'd like to participate too. And I'm that guy from the RailsConf. :-)
My primary target is an separate GEM because I work now more with Merb
than with Rails. And I really like to have CLDR stuff independent and
accesisble to both Rails and Merb.
I don't know how the Merb guys will go for i18n. If they use rails
i18n in its GEM form or they'll build their own stuff. That is also
reason to keep CLDR GEM and API as simple as possible and use it only
as a "data source". So the Rails i18n could have use the gem as one of
the sources (backend) of locale information and Merb could have their
own code to get information from CLRD GEM.
I had basically the same ideas as you all wrote here. So API to get
locale information and tools in form of the rake tasks to update CLDR
distribution (which is not so often) and tasks to pre-process CLDR XML
to someting faster and ready for the API, hash possibly.
Pavel
My primary target is an separate GEM because I work now more with Merb
than with Rails. And I really like to have CLDR stuff independent and
accesisble to both Rails and Merb.
I don't know how the Merb guys will go for i18n. If they use rails
i18n in its GEM form or they'll build their own stuff. That is also
reason to keep CLDR GEM and API as simple as possible and use it only
as a "data source". So the Rails i18n could have use the gem as one of
the sources (backend) of locale information and Merb could have their
own code to get information from CLRD GEM.
I had basically the same ideas as you all wrote here. So API to get
locale information and tools in form of the rake tasks to update CLDR
distribution (which is not so often) and tasks to pre-process CLDR XML
to someting faster and ready for the API, hash possibly.
Pavel
Pavel Kunc
Sep 9, 2008, 12:09:34 PM9/9/08
to rails-i18n
And we also thought about CLDR during the merb_global localization
plugin: http://groups.google.com/group/merb_global/browse_thread/thread/25de0899ea2b661e
plugin: http://groups.google.com/group/merb_global/browse_thread/thread/25de0899ea2b661e
Sven Fuchs
Sep 9, 2008, 6:44:15 PM9/9/08
to rails...@googlegroups.com
Hey Pavel,
welcome to the list, great that you made it :)
I guess we're all mostly on the same page regarding a separate ruby
gem that makes sense for just everyone and regarding to just provide
things in an easily accessible way.
So, that makes already four of us, at least. Maybe we should just meet
up on IRC? I'd suggest #rails-i18n on irc.freenode.net
What's a good date and time?
I'd suggest Friday, Saturday or Sunday at 16:00 UTC:
http://www.timeanddate.com/worldclock/fixedtime.html?month=9&day=12&year=2008&hour=16&min=0&sec=0&p1=0
http://www.timeanddate.com/worldclock/fixedtime.html?month=9&day=13&year=2008&hour=16&min=0&sec=0&p1=0
http://www.timeanddate.com/worldclock/fixedtime.html?month=9&day=14&year=2008&hour=16&min=0&sec=0&p1=0
Joshua Harvey
Sep 9, 2008, 7:15:20 PM9/9/08
to rails...@googlegroups.com
I think Sat or Sun would work for me.
-- Josh
Clemens Kofler
Sep 10, 2008, 6:01:03 AM9/10/08
to rails-i18n
Friday, Saturday and Sunday are all fine with me.
On Sep 10, 1:15 am, "Joshua Harvey" <jos...@gmail.com> wrote:
> I think Sat or Sun would work for me.
> -- Josh
>
>
> >http://www.timeanddate.com/worldclock/fixedtime.html?month=9&day=13&y...
>
> >http://www.timeanddate.com/worldclock/fixedtime.html?month=9&day=14&y...
On Sep 10, 1:15 am, "Joshua Harvey" <jos...@gmail.com> wrote:
> I think Sat or Sun would work for me.
> -- Josh
>
> On Wed, Sep 10, 2008 at 1:44 AM, Sven Fuchs <svenfu...@artweb-design.de>wrote:
>
>
>
> > Hey Pavel,
>
> > welcome to the list, great that you made it :)
>
> > I guess we're all mostly on the same page regarding a separate ruby
> > gem that makes sense for just everyone and regarding to just provide
> > things in an easily accessible way.
>
> > So, that makes already four of us, at least. Maybe we should just meet
> > up on IRC? I'd suggest #rails-i18n on irc.freenode.net
>
> > What's a good date and time?
>
> > I'd suggest Friday, Saturday or Sunday at 16:00 UTC:
>
> >http://www.timeanddate.com/worldclock/fixedtime.html?month=9&day=12&y...
>
>
>
> > Hey Pavel,
>
> > welcome to the list, great that you made it :)
>
> > I guess we're all mostly on the same page regarding a separate ruby
> > gem that makes sense for just everyone and regarding to just provide
> > things in an easily accessible way.
>
> > So, that makes already four of us, at least. Maybe we should just meet
> > up on IRC? I'd suggest #rails-i18n on irc.freenode.net
>
> > What's a good date and time?
>
> > I'd suggest Friday, Saturday or Sunday at 16:00 UTC:
>
>
> >http://www.timeanddate.com/worldclock/fixedtime.html?month=9&day=13&y...
>
> >http://www.timeanddate.com/worldclock/fixedtime.html?month=9&day=14&y...
Yaroslav Markin
Sep 10, 2008, 6:23:15 AM9/10/08
to rails...@googlegroups.com
Same here, Saturday would be nice.
On Wed, Sep 10, 2008 at 2:01 PM, Clemens Kofler <cle...@railway.at> wrote:
Friday, Saturday and Sunday are all fine with me.
Pavel Kunc
Sep 11, 2008, 5:52:08 AM9/11/08
to rails-i18n
This weekend is complicated, but Friday or Sunday, I can make.
On Sep 10, 12:23 pm, "Yaroslav Markin" <yaroslav.mar...@gmail.com>
wrote:
On Sep 10, 12:23 pm, "Yaroslav Markin" <yaroslav.mar...@gmail.com>
wrote:
> Same here, Saturday would be nice.
>
>
Christian Lescuyer
Sep 11, 2008, 6:25:17 AM9/11/08
to rails-i18n
Thanks for the link, Clemens, I was not aware of the CLDR. This seems
really interesting.
I'll try to join you on IRC if I can find what the UTC time is here
(just kidding), and if I find how to do IRC with my brand new Mac :-)
Christian
really interesting.
I'll try to join you on IRC if I can find what the UTC time is here
(just kidding), and if I find how to do IRC with my brand new Mac :-)
Christian
Clemens Kofler
Sep 11, 2008, 7:41:11 AM9/11/08
to rails-i18n
Christian: http://colloquy.info/ - works like a charm! ;-)
On Sep 11, 12:25 pm, Christian Lescuyer <christian.lescu...@gmail.com>
wrote:
On Sep 11, 12:25 pm, Christian Lescuyer <christian.lescu...@gmail.com>
wrote:
Clemens Kofler
Sep 11, 2008, 7:44:04 AM9/11/08
to rails-i18n
Alright. Seems that everyone can do on Sunday - so let's schedule the
first meeting on Sunday, 14 September, at 16h UTC in #rails-i18n. I
suggest this be a pretty informal meeting although I will maybe put
down some topics that should definitely be discussed.
Everyone d'accord with this?
first meeting on Sunday, 14 September, at 16h UTC in #rails-i18n. I
suggest this be a pretty informal meeting although I will maybe put
down some topics that should definitely be discussed.
Everyone d'accord with this?
Joshua Harvey
Sep 11, 2008, 7:56:26 AM9/11/08
to rails...@googlegroups.com
works for me :)
Sven Fuchs
Sep 11, 2008, 8:51:59 AM9/11/08
to rails...@googlegroups.com
+1
Christian Lescuyer
Sep 11, 2008, 8:53:23 AM9/11/08
to rails-i18n
D'accord :-)
Kless
Sep 12, 2008, 5:24:19 AM9/12/08
to rails-i18n
Ago time I had thinked in store all those data into a SQLite database
so it could be used from whatever language and also could be used from
desktop systems -today they are based on gettext-. Note that using a
RDBMS is more efficient since that could be related many data as
languages to countries.
I also had created these relations: country - language, country -
first_subdivision, country - timezone, first_subdivision -
timezone, ...
My idea was to create a service where can add/modify easily those
data, and get the SQLite database, that could be used in whatever
language always when been created an API -for the desired language- to
access to those data.
so it could be used from whatever language and also could be used from
desktop systems -today they are based on gettext-. Note that using a
RDBMS is more efficient since that could be related many data as
languages to countries.
I also had created these relations: country - language, country -
first_subdivision, country - timezone, first_subdivision -
timezone, ...
My idea was to create a service where can add/modify easily those
data, and get the SQLite database, that could be used in whatever
language always when been created an API -for the desired language- to
access to those data.
Pavel Kunc
Sep 12, 2008, 5:32:50 AM9/12/08
to rails-i18n
OK, Sunday, great.
Sven Fuchs
Sep 12, 2008, 12:14:02 PM9/12/08
to rails...@googlegroups.com
limechat, limechat, limechat :)
http://limechat.sourceforge.net
Kless
Sep 12, 2008, 1:39:49 PM9/12/08
to rails-i18n
Well, I've been looking in CLDR and my initial idea for create a web
service where add/modify those data isn't necessary since that all
data
are updated frecuently, and in addition there is a frontend for modify
them, and there are also relations of languages with countries.
Now, the next step would be to convert those XML data to a format
for that can be used easily by a program. And it has already made :)
In Python there is a program called Babel [1] whose author
converted those data to binary, and then use an API to access to them.
So, It would be only necessary create a ruby API to access to it.
[1] http://babel.edgewall.org/
On Sep 12, 10:24 am, Kless <jonas....@googlemail.com> wrote:
> Ago time I had thinked in store all those data into a SQLite database
> so it could be used from whatever language and also could be used from
> desktop systems -today they are based on gettext-. Note that using a
> RDBMS is more efficient since that could be related many data as
> languages to countries.
>
> I also had created these relations: country - language, country -
> first_subdivision, country - timezone, first_subdivision -
> timezone, ...
>
> My idea was to create a service where can add/modify easily those data
service where add/modify those data isn't necessary since that all
data
are updated frecuently, and in addition there is a frontend for modify
them, and there are also relations of languages with countries.
Now, the next step would be to convert those XML data to a format
for that can be used easily by a program. And it has already made :)
In Python there is a program called Babel [1] whose author
converted those data to binary, and then use an API to access to them.
So, It would be only necessary create a ruby API to access to it.
[1] http://babel.edgewall.org/
On Sep 12, 10:24 am, Kless <jonas....@googlemail.com> wrote:
> Ago time I had thinked in store all those data into a SQLite database
> so it could be used from whatever language and also could be used from
> desktop systems -today they are based on gettext-. Note that using a
> RDBMS is more efficient since that could be related many data as
> languages to countries.
>
> I also had created these relations: country - language, country -
> first_subdivision, country - timezone, first_subdivision -
> timezone, ...
>
> My idea was to create a service where can add/modify easily those data
Joshua Harvey
Sep 13, 2008, 12:31:50 PM9/13/08
to rails...@googlegroups.com
Ok, so we're on for Sunday (tomorrow) at 16:00 UTC, just look up the closest city in your country:
Sven and I discussed some items we might want to put on the agenda. Feel free to discuss priorities and additions here. Here's what we have so far:
1. What data format should we convert to? Ruby? YAML?
2. What should the API look like?
3. Fetch function? Standalon app? Rake task?
4. How to package it, probably as a Gem.
5. Which functionality in CLDR to prioritize:
Pluralization rules
Right-to-Left
Date, number, currency formats
Language names in each language
Country names in each language
6. Do we allow converting just a subset of locales?
-- Josh
Clemens Kofler
Sep 13, 2008, 12:40:20 PM9/13/08
to rails-i18n
Thanks Josh.
I'd also like to discuss existing projects/implementations (we already
mentioned Ruby CLDR and Babel for Python) plus the typical project
management stuff (timeline, coordination, project setup, etc.). But
hell - we'll get to that anyway! ;-)
See you guys tomorrow.
On Sep 13, 6:31 pm, "Joshua Harvey" <jos...@gmail.com> wrote:
> Ok, so we're on for Sunday (tomorrow) at 16:00 UTC, just look up the closest
> city in your country:http://www.timeanddate.com/worldclock/fixedtime.html?month=9&day=14&y...
I'd also like to discuss existing projects/implementations (we already
mentioned Ruby CLDR and Babel for Python) plus the typical project
management stuff (timeline, coordination, project setup, etc.). But
hell - we'll get to that anyway! ;-)
See you guys tomorrow.
On Sep 13, 6:31 pm, "Joshua Harvey" <jos...@gmail.com> wrote:
> Ok, so we're on for Sunday (tomorrow) at 16:00 UTC, just look up the closest
>
> Sven and I discussed some items we might want to put on the agenda. Feel
> free to discuss priorities and additions here. Here's what we have so far:
>
> 1. What data format should we convert to? Ruby? YAML?
> 2. What should the API look like?
> 3. Fetch function? Standalon app? Rake task?
> 4. How to package it, probably as a Gem.
> 5. Which functionality in CLDR to prioritize:
> Pluralization rules
> Right-to-Left
> Date, number, currency formats
> Language names in each language
> Country names in each language
> 6. Do we allow converting just a subset of locales?
>
> -- Josh
>
> Sven and I discussed some items we might want to put on the agenda. Feel
> free to discuss priorities and additions here. Here's what we have so far:
>
> 1. What data format should we convert to? Ruby? YAML?
> 2. What should the API look like?
> 3. Fetch function? Standalon app? Rake task?
> 4. How to package it, probably as a Gem.
> 5. Which functionality in CLDR to prioritize:
> Pluralization rules
> Right-to-Left
> Date, number, currency formats
> Language names in each language
> Country names in each language
> 6. Do we allow converting just a subset of locales?
>
> -- Josh
>
Reply all
Reply to author
Forward
0 new messages