Fwd: [OpenTripPlanner] Re: Multilanguage proposal

[Marko Burjek]

I didn't send last message to the list.

@fxpurcell Thanks for your work. They will be useful.

I have another question why isn't time_format from otp localization used?
For example in javascript files there are many instances of time format h:mma ( https://github.com/opentripplanner/OpenTripPlanner/search?l=javascript&q=h%3Amma&type=Code ) which I figured is moment.js time format for 12 hour time format. But in current localization files there is a variable time_format https://github.com/opentripplanner/OpenTripPlanner/blob/27956dfb60f31b051c788adc9385b4abde733562/otp-openlayers-client/src/main/webapp/js/otp/locale/English.js#L238 Which isn't used. And time_format is also a format which isn't moment.js time format. Is there any reason for this? Or can I change other h:mma times to use localized time_format?

2014-03-21 5:26 GMT+01:00 Frank P <fxpu...@gmail.com>:

Hey Marko ... feel free to use some .po files I've been building over at https://github.com/OpenTransitTools/view/tree/master/ott/view/locale.  I'm thinking a fair number of the strings I've localized into Spanish will map to strings within the Leaflet client.

BTW, here you can see these localizations work against an OTP backend here:

  http://trimet.org/ride/planner_form.html?_LOCALE_=es

 - and -

  http://trimet.org/ride/planner_form.html

It would benefit me to know that the .po files that I've been using with Babel / Pyramid / Python also work with other libraries like Pomo / JsGettext.  

On Thursday, March 20, 2014 12:43:02 AM UTC-7, Marko Burjek wrote:

Is anyone working on this?

If not I will look into it because I need localization in local deployment.
I will compare JsGettext and Pomo to see which is better. They both support gettext po files. Pomo seems to be more modern jsgettext is 5 years old. I also found Jed and i18next but first seem to only works on node-js I have to look into i18next. Anybody knows any other js gettext library?

I would like to know which branch to branch for this and on what client to work? Openlayers or Leaflet. And is Openlayers abandoned because it doesn't work on master?

I posted something on the tracker on issue #261

On Monday, 3 May 2010 20:48:29 UTC+2, Karel wrote:

I have created 3 new tickets for this based on this thread (261-263).

karel

On Sat, 2010-05-01 at 05:15 -0700, Frank wrote:
> To clarify, I'm seeing the proposal shaping up like this:
>
> 1. move the exiting English.js and Spanish.js translation content over
> to gettext Portable Object (PO) files
> 2. use jsgettext within the .js webapp to read the PO, and translate
> our strings.
>
> That's great.  I'm +1 on this idea, with the addition that we maintain
> the PO translation files in a single place available for use
> throughout the project, to both java and javascript modules (as
> opposed to locking translations away in the webapp / javascript
> specific).  My question is in what module do we house the PO files?
> Do we use an exiting module (e.g., opentripplanner-utils) or create a
> new module (e.g., opentripplanner-localizations)?
>
> On Apr 26, 8:49 am, Nicholas Bergson-Shilcock
> <nichola...@openplans.org> wrote:
> > Excerpts from jvhigon's message of Mon Apr 26 01:34:58 -0400 2010:
> >
> > > Hello,
> > > I've never used jsgettext so I can't help doing a patch at this moment
> > > (I could contribute with the changes that I commented few days ago).
> > > The approach with gettext seems to be better than the OpenLayers
> > > solution (I don't know if OL have already worked this problem out). I
> > > wiil wait for your answer in order to do the patch.
> >
> > If Dave or I took a pass at moving what's currently localized to the gettext
> > system, would you be able to then help out in bringing some of the remaining
> > items (e.g., those listed in ticket #220http://opentripplanner.org/ticket/220)
> > over to the new system?
> >
> > -Nick
> >
> >
> >
> > > Regards
> >
> > > On 23 abr, 21:36, Karel Novotny <novotny.ka...@gmail.com> wrote:
> > > > On Fri, 2010-04-23 at 13:00 -0400, David Turner wrote:
> > > > > Actually, question:  how does this deal with plurals?  Because I just
> > > > > found myself writing code to deal with plurals and realized that if I
> > > > > had to do very much of that, I would go mad:
> >
> > > > > <tpl if="duration != 1.0">{[this.locale.time.minute]}</tpl>)<tpl
> > > > > if="duration != 1.0">{[this.locale.time.minutes]}</tpl>)</span></tpl>
> >
> > > > > Also, it would not work for languages like Polish which have a separate
> > > > > forms not just for 1 and n, but also for other numbers.
> >
> > > > > See this:
> > > > >http://www.gnu.org/software/hello/manual/gettext/Plural-forms.html
> >
> > > > I am glad you brought this up. I am czech and haven't actually realized
> > > > that just '1 vers. many' system doesn't work for my language either.
> >
> > > > karel
> >
> > > > > On Fri, 2010-04-23 at 12:34 -0400, David Turner wrote:
> > > > > > This sounds good to us.  Would you be willing to write a patch to do
> > > > > > this?  
> >
> > > > > > On Fri, 2010-04-23 at 02:34 -0700, jvhigon wrote:
> > > > > > > Hi everyone,
> > > > > > > I would like to propose a new way to deal with multilanguage in OTP..
> > > > > > > That is, using the Openlayers approach.
> > > > > > > The steps would be the following:
> > > > > > > 1.-  It would be a javascript file for each language. This file would
> > > > > > > extend the OpenLayers file defined for this language. For example:
> >
> > > > > > > OpenLayers.Lang.en_otp = OpenLayers.Util.extend(OpenLayers.Lang.en, {
> > > > > > >  'key' : 'translation_in_otp_code',
> > > > > > >   ...
> > > > > > >    CLASS_NAME: "OpenLayers.Lang.en_otp"
> > > > > > > });
> >
> > > > > > > 2.- The language would be set at the beginning of the application,
> > > > > > > using an OpenLayers function. For example:
> >
> > > > > > > OpenLayers.Lang.setCode('en_otp');
> >
> > > > > > > 3.-  The translatiosn would be done with another OpenLayers function:
> >
> > > > > > > OpenLayers.Lang.translate('key')
> >
> > > > > > > By the way, there is some source code that is not able to be
> > > > > > > translated like "Templates.js" . With this new approach the changes
> > > > > > > would be as described below:
> >
> > > > > > > Now:
> > > > > > >     TP_START : new Ext.XTemplate(
> > > > > > >           '<h4><a href="#">Start at</a> {name}</h4>'
> > > > > > >     ).compile(),
> >
> > > > > > > With OpenLayers approax
> >
> > > > > > >    TP_START = new Ext.XTemplate(
> > > > > > >           '<h4><a href="#">' +  OpenLayers.Lang.translate('start_at')
> > > > > > > + '</a> {name}</h4>'
> > > > > > >     ).compile();
> >
> > > > > > > What do you think?
> >
> > > > > > > Regards
> >
> > > > --
> > > > Subscription settings:http://groups.google.com/group/opentripplanner-dev/subscribe?hl=en

--
You received this message because you are subscribed to a topic in the Google Groups "OpenTripPlanner Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opentripplanner-dev/KLdzJdmx94k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opentripplanner...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marko Burjek]

I tried Jsgettext, Pomo and i18next and I think that i18Next is currently the best. It is MIT licensed and has 31 kb.

JSgettext is old (Last commit in 2012) it is only basic wrapper over gettext (different functions for singular or plural, another function for replacing parameters). Po files are suported as is. But supporting multiple languages files has to be converted to json with Perl. Pomo is quite Ok but is only one man band project.

i18next supports po files, which has to be converted to json with nodejs before use. It is actively maintained supports translating jquery, jquery UI and handlebars and everything else. Only one function. There are projects which can create POT files from source js files. It also supports setting language from query parameter, browser language or cookie.

More features: http://i18next.com/pages/doc_features.html

Before I start porting to i18next I would like to fix hardcoded time format in issue:

https://github.com/opentripplanner/OpenTripPlanner/issues/1382 .

[Marko Burjek]

Translation update.

Currently I fully translated widgets for:
Trip Options,

Itineraries (Also print and I think email too), with all relative travel directions

Stop Viewer,

Stop Finder,
Trip Viewer,
Analyst,
Bikeshare.

Currently there is a 149 strings to translate in total. https://github.com/buma/OpenTripPlanner/blob/po_localization/otp-leaflet-client/src/main/webapp/i18n/messages.pot

Most of it was easy to change with some I had to change templates too, because text was in templates.

Untraslated things currently are:
- errors from trip planner,
- some buttons on widgets (OK, Yes, No, Cancel), because I don't know where they appear. https://github.com/buma/OpenTripPlanner/blob/po_localization/otp-leaflet-client/src/main/webapp/js/otp/widgets/widget-templates.html

- Geocoder stuff (because I don't have a geocoder)

- absolute directions (How would I get trip with absolute directions to test it?)

 And things that are send from server for example in Trip Viewer bus line from A to B. From has to be translated on a server.

I would like to add FieldTrip and CallTaker also, but I can't start them.

When I choose calltaker (after adding it in config.js) I get URL not found:
http://localhost:8080/undefined/auth/newSession

Same with FieldTripPlanner.

I also added a Makefile https://github.com/buma/OpenTripPlanner/blob/po_localization/Makefile for extracting translations and updating it. Currently it works only with my path to pybabel.

All the code for calling i18next is in config.js https://github.com/buma/OpenTripPlanner/blob/po_localization/otp-leaflet-client/src/main/webapp/js/otp/config.js

Next stage is changing Makefile that it also installs necessary programs, try to use as much translations as possible when porting language.js translation files to po.

I think this should be finished in next week.

[Andrew Byrd]

Hello Marko,

Thanks for the update, and thanks for all this work on internationalizing OTP.

It's good that you've created a reproducible process for extracting strings and working on translations, but it seems unlikely to me that the casual translator will want to install all the tooling to go through this process. When I look at the final result (the JSON file that is actually loaded by the JS library) it strikes me that it would be very easy to just duplicate this file and edit it directly:
https://github.com/buma/OpenTripPlanner/blob/po_localization/otp-leaflet-client/src/main/webapp/js/otp/locale/sl.json

Is there any reason why we should discourage people from supplying translations this way?

You mention "changing Makefile that it also installs necessary programs, try to use as much translations as possible when porting language.js translation files to po." Considering the variety of platforms that people are running OTP on, I'm not sure changing the Makefile to install software would be a good idea. It's very likely that only one or two people will be working with your entire PO/POT toolchain and a HOWTO document would be more helpful. I also don't really understand how changing this Makefile would help with porting the old language files to the new system. Can you elaborate?

Unfortunately I can't help you much with setting up the calltaker and fieldtrip interfaces, as I have not worked on them. Work is still continuing on those, so it may be more practical to internationalize them at a later date.

-Andrew

You received this message because you are subscribed to the Google Groups "OpenTripPlanner Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner...@googlegroups.com.

[Marko Burjek]

2014-05-19 16:29 GMT+02:00 Andrew Byrd <and...@fastmail.net>:
Hello Marko,

Thanks for the update, and thanks for all this work on internationalizing OTP.

It's good that you've created a reproducible process for extracting strings and working on translations, but it seems unlikely to me that the casual translator will want to install all the tooling to go through this process. When I look at the final result (the JSON file that is actually loaded by the JS library) it strikes me that it would be very easy to just duplicate this file and edit it directly:
https://github.com/buma/OpenTripPlanner/blob/po_localization/otp-leaflet-client/src/main/webapp/js/otp/locale/sl.json

Is there any reason why we should discourage people from supplying translations this way?

Yes file is easy to edit directly. But I would discourage it for next reasons:

Context
For example you are translator and would like to translate this keys:

"Depart_tripoptions": ""

"Depart_itinerary": "",

They are both Depart in english. How to translate them depends on usage. In json file only context (word behind _) is retained but in po file you have everything you need to have (Translator comments, where this happens if you need to look etc..):

#. Depart on to street / Board at station in itinerary
#: otp-leaflet-client/src/main/webapp/js/otp/modules/planner/ItinerariesWidget.js:536
#, fuzzy
msgctxt "itinerary"
msgid "Depart"

msgstr ""

#. Depart [time dropdown] [date dropdown]. Used in
#. dropdown as a label to choose wanted time/date of departure
#: otp-leaflet-client/src/main/webapp/js/otp/widgets/tripoptions/TripOptionsWidget.js:312

msgctxt "tripoptions"
msgid "Depart"
msgstr ""

Plurals
This is original text in english that translators sees
    "%d min early": "",                                                        
    "%d min early_plural": "",

Slovenian translators have to change that to:

    "every %d min_plural_5": "",
    "every %d min": "",
    "every %d min_plural_2": "",
    "every %d min_plural_3": "",

Because we have 4 different plural forms. (Most languages have less). 

But each translator would need to know how should he change that.
In gettext this is only one command away. (msginit -i messages.pot -o lang.po -l lang).

Updating

Now when new translations are added they are merged with all existing languages with one command.

 And we always now for each file how complete translation is.
Because we have po files. But if we have only json file from a translator this is a manual process.
 And we could have translations we woudln't know are incomplete.


Tooling support

For gettext translation exist a lot of programs. For example:
http://userbase.kde.org/Lokalize
http://virtaal.translatehouse.org/

http://poedit.net/
http://www.gted.org/ (Plugin for Eclipse)
https://localise.biz/free/poedit (freely usable from a browser, you don't even have to register)


Most of them support translation memory (So that you can use other translations if same string is used),
 web translating services to easy translating, setting
 correctness of translation(Fuzzy usually means you are not sure that translation is correct), checking string correctness

 (If original string has space and the end, translation needs to have it too, parameter correctness etc..)

They are GUI programs.

And also we could send translation template (Pot file) to https://www.transifex.com (It's free for OSS projects) or

 https://help.launchpad.net/Translations/StartingToTranslate .
So that making translations is even easier and nobody needs to install anything.


I think that with using json for translation we would have similar problem then now with js files where some are complete,

 some are not, but you are not really sure.

You mention "changing Makefile that it also installs necessary programs, try to use as much translations as possible when porting language.js translation files to po." Considering the variety of platforms that people are running OTP on, I'm not sure changing the Makefile to install software would be a good idea. It's very likely that only one or two people will be working with your entire PO/POT toolchain and a HOWTO document would be more helpful. I also don't really understand how changing this Makefile would help with porting the old language files to the new system. Can you elaborate?

Then I'll just write a HOWTO it is probably better.


Makefile isn't used for old translations. I think you probably misunderstood me. I'm writing a script that will use

 translations from old *.js files to bootstrap new gettext translations.

I also added translations for Geocoder and found that dialogs are mostly used in fieldTrip and callTaker which

 I would leave for now.
I still want to know if you can help me How would I get trip with absolute directions?

Translators actually doesn't need any tooling. (Strictly speaking they would need msginit to initialize translation or change "LANGUAGE" to their language in pot file and add plural forms from http://localization-guide.readthedocs.org/en/latest/l10n/pluralforms.html)
Developer who changes translatable strings or adds new ones would need tooling it he wants to keep translations up to date. But that is also not necessary we could only update translations before new release is out and only that time my tooling is needed.

Most gettext project update translation files in time before a release called string freeze where by agreement new strings shouldn't be added unless it is absolutely necessary.

 

Unfortunately I can't help you much with setting up the calltaker and fieldtrip interfaces, as I have not worked on them. Work is still continuing on those, so it may be more practical to internationalize them at a later date.

-Andrew

I am at an end of what I can translate in a frontend. Errors and notes etc.. can only be translated in Java. I have a question should I change an API to add language parameter to all calls that could return strings? or does anyone have a better idea how to solve this?

- Marko
 

[Andrew Byrd]

Thanks Marko for explaining all of this. I can see how having the translation comments, context, and proper labeling of plural forms is important, so it's not practical for people to just hack directly on the JSON.

You mention that "Now when new translations are added they are merged with all existing languages with one command." At this point you probably understand the process much better than the rest of us, so it will be important for you to contribute step-by-step instructions for updating the template and individual translations.

Can we also count on you to be the i18n maintainer?

I am still a little concerned that we'll see a drop in translation contributions due to perceived complexity. But I suppose that's the price of "doing it right". I guess we can always prepare the PO file for a volunteer translators and point them to a tool, online if necessary.

-Andrew

[Andrew Byrd]

On 05/19/2014 05:49 PM, Marko Burjek wrote:

> I am at an end of what I can translate in a frontend. Errors and notes
> etc.. can only be translated in Java. I have a question should I
> change an API to add language parameter to all calls that could return
> strings? or does anyone have a better idea how to solve this?

I suppose the HTTP Accept-language header is also an option. However, we
require more than just language information -- we need to localize date
formats (ugh) and units.

I don't think we have the spare dev capacity to work on translating
backend strings at the moment. Or are you offering to do the coding?
Perhaps it is most important to get the frontend translation process in
place first.

-Andrew

[Marko Burjek]

2014-05-20 16:55 GMT+02:00 Andrew Byrd <and...@fastmail.net>:

Thanks Marko for explaining all of this. I can see how having the translation comments, context, and proper labeling of plural forms is important, so it's not practical for people to just hack directly on the JSON.

You mention that "Now when new translations are added they are merged with all existing languages with one command." At this point you probably understand the process much better than the rest of us, so it will be important for you to contribute step-by-step instructions for updating the template and individual translations.

Yes no problem. This comes in HOWTO. (Currently you just do "make update" if you have Babel ofcourse).

Can we also count on you to be the i18n maintainer?

Yes. I would love to be i18 maintainer.

I am still a little concerned that we'll see a drop in translation contributions due to perceived complexity. But I suppose that's the price of "doing it right". I guess we can always prepare the PO file for a volunteer translators and point them to a tool, online if necessary.

We can write in a HOWTO if you want to translate to your language post to mailing list. And then I can send them po file or point them to web translating project or something.

[Marko Burjek]

2014-05-20 17:03 GMT+02:00 Andrew Byrd <and...@fastmail.net>:

On 05/19/2014 05:49 PM, Marko Burjek wrote:

I am at an end of what I can translate in a frontend. Errors and notes etc.. can only be translated in Java. I have a question should I change an API to add language parameter to all calls that could return strings? or does anyone have a better idea how to solve this?

I suppose the HTTP Accept-language header is also an option. However, we require more than just language information -- we need to localize date formats (ugh) and units.

I think  HTTP Accept-language could be used for main site (Currently it is in my branch i18next supports it). Together with language chooser, for all people living in different country and don't know how to change language in browser. But then we need to tell server in which language to send "first station to last station" in routes and all "at the corner of x and y" which are currently in a language where server is run.

Units and time/date format settings are now already configurable in a client. Next thing to do would be to look if everything is consistent which currently isn't. All times in ISO 8601, all units in m/s then it is up to a client to convert to nice format for a user. I think units are all in meters.

One option is to send it in English as a template and replace with language specific things on a client. This would be probably little harder for mobile clients.

Another one is to add locale parameter to API if parameter is empty return English otherwise return strings in that language if it exists.

I don't think we have the spare dev capacity to work on translating backend strings at the moment. Or are you offering to do the coding? Perhaps it is most important to get the frontend translation process in place first.

I could do the coding. First I would like to finish frontend and get it merged. Then connect it with fixes I did with time/date.

-Andrew

Marko

--
You received this message because you are subscribed to the Google Groups "OpenTripPlanner Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-dev+unsub...@googlegroups.com.