[TW5] MultiLang

[wimm]

Hi,

translation proof of concept / show case multi language TW5.1.11

The current implementation for interface language rquires 600+ separate tiddlers for each piece of text for each language. 

my goal

- reduce the number of shadow (translation) tiddlers (core plugin was getting very long list of tiddlers) (partial done, language tiddlers are still present but not used, a few for long texts will remain).

- reduce the space required for translation

- being able to remove a translation (plugins cannot be removed shadows remain present)

- instant language translation switching without reload (partial implemented, not yet for already displayed buttons)

- make code more readable by including original english text in the code instead of lingo-base and lingo macro (done)

- easier to translate, no extra tools required, checking for missing entries (one file per language), switch back to English to find the search key, for the translation)

- simply support multiple translations at the same time (Belgian Dutch/French) 

- to do: translations except for dates, numbers, colors, these also require rewrite of utils.js

- to do: all languages (partial done see list of languages below)

to accomplish this I created 9 data tiddlers one for each complete language translation called $:/trnslt-en2, $:/trnslt-nl, $:/trnslt-fr,etc

for example in the Dutch(nl/Nederlands) translation tiddler there is a line with the tranlation for 'close others' 

 close others:sluit anderen 

instead of

\define lingo-base() $:/language/Buttons/
<<lingo CloseOthers/Caption>>

the tiddlers will then contain a query on a datatiddler $:\trnslt and contains the query key of the text

{{$:/trnslt##close others}}

this then tranlates to

sluit anderen

in your tiddlers

You can have several translation present, by copying the language translation file to $:/trnslt all translation will use that translation

Please try the result at http://tw5multilanguage.tiddlyspot.com 

I  came up with the idea because I wanted to easily allow for translations for my new edit diff plugin (based on jsdiff).

This is WORK IN PROGRESS there is still a lot to do, so do not use this TW  for storing imported data 

Is this the way to go? pull request?

Njoy

wimm

[Jeremy Ruston]

Hi Wimm

That’s an ambitious and interesting piece of work, thank you for undertaking it!

- reduce the number of tiddlers (core plugin was getting very long list of tiddlers) (partial done, language tiddlers are still present but not used, a few for long texts will remain).

I’m not sure that I see the number of shadow tiddlers as a great disadvantage. Are you concerned with performance or something else?

- immediate language translation switching without reload (partial done, not yet for already present buttons)

Doesn’t that already work? As far as I know a restart is needed after installing a new language plugin, but not to switch between them.

- make code more readable by including original english text in the code instead of lingo-base and lingo macro (done)

I would need persuading that this is an improvement. It leads to much longer identification strings, and will surely lead to complexity if we want to change the *English* translation of a string?

- easier to translate, checking for missing entries (one file per language), switch back to English to find the search key, for the translation)

My thinking was that we could build tools to make it easier to translate, hence http://tiddlywiki.com/editions/translators/

This is WORK IN PROGRESS there is still a lot to do do not use this TW  for storing imported data 

Is this the way to go? pull request?

It’s very interesting; I hope my comments don’t put you off. I’d certainly find it easier to review your changes if you could make a fork on GitHub.

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/6a9e4ee7-67b8-4ef4-9945-9e025e889044%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jed Carty]

I haven't looked through the code so I am going by your description for this.

For anything like this my two concerns would be that languages need to be able to be updated independently and that plugins should be able to be translated also.
It sounds like you have the first one without any trouble, but if each language is in a single data tiddler can we still have optional additions for plugins using this system?

I also agree with Jeremy that using the english translation of a string as the key isn't necessarily a good idea. Problems with changing the english translation are only one part of that though.
Currently english is the common language for the core development, this isn't a problem because it is one of the most common languages in the world, but I would rather avoid anything that would make it a requirement or discourage non-english speakers from thinking they could contribute.
I am not certain that using english as the key would be a problem in either of those cases, but that was my first reaction to the idea.

[wimm]

Hi Jed

I haven't looked through the code so I am going by your description for this.

the TW gives more information

 

For anything like this my two concerns would be that languages need to be able to be updated independently and that plugins should be able to be translated also.

It sounds like you have the first one without any trouble, but if each language is in a single data tiddler can we still have optional additions for plugins using this system?

Separate tiddlers for a plugin translation are possible, that was the original problem I tried to resolve, before refactoring the complete TW

$:/trnslt-pluginname-en

$:/trnslt-pluginname-nl

etc...

 

I also agree with Jeremy that using the english translation of a string as the key isn't necessarily a good idea. Problems with changing the english translation are only one part of that though.

I thought of that.. There is an English to English translation tiddler, to allow for modification of the English txt, as long as you leave the part before the : untouched, so the same mechanism works for all languages.

 

Currently english is the common language for the core development, this isn't a problem because it is one of the most common languages in the world, but I would rather avoid anything that would make it a requirement or discourage non-english speakers from thinking they could contribute.

I am not certain that using english as the key would be a problem in either of those cases, but that was my first reaction to the idea.

 

thanks for your concerns

KR 

WiM

 

[wimm]

Hi Jeremy,

 

That’s an ambitious and interesting piece of work, thank you for undertaking it!

That's alright, most of the work was done by curl, regular expressions using gvim and tiddlywiki to create the new translation tiddlers.

- reduce the number of tiddlers (core plugin was getting very long list of tiddlers) (partial done, language tiddlers are still present but not used, a few for long texts will remain).

I’m not sure that I see the number of shadow tiddlers as a great disadvantage. Are you concerned with performance or something else?

- immediate language translation switching without reload (partial done, not yet for already present buttons)

Doesn’t that already work? As far as I know a restart is needed after installing a new language plugin, but not to switch between them.

this is  just a feature that should still exist, adding a new language would now not require a reload anymore

- make code more readable by including original English text in the code instead of lingo-base and lingo macro (done)

I would need persuading that this is an improvement.

My main concern was the indirection, I tried to add a translation of a diff view plugin (using jsdiff.js) when editing tiddlers, and came to this solution.

When I search for a system tiddler I normally use the span title (it is hard to search for an icon) in the current implementation you first find the translation tiddler, then you have to select ''part'' of the translation tiddler path, to find the tiddler you were looking for.

 

Using the new trnslt tiddlers, you will find the tiddler (plus the trnslt tiddlers), if you only find trnslt tiddlers the text was modified in the En-En translation, in this tiddler you can find the original key to search again. . 

 

It leads to much longer identification strings,

That is true, but it will result in a smaller TW file (trnslt json's for 8 languages are 215k)

 

and will surely lead to complexity if we want to change the *English* translation of a string?

 

English translation strings can be modified  in the English to English translation tiddler. Other translations still work if you do not change the key. If you modify the text a lot, then you maybe should opt for a new entry, so everybody knows that their translations probably also need updating..

- easier to translate, checking for missing entries (one file per language), switch back to English to find the search key, for the translation)

My thinking was that we could build tools to make it easier to translate, hence http://tiddlywiki.com/editions/translators/

I tried the translator GUI, but when importing all translations, I noticed many translations are missing.

tooling to find missing translations you would need a tiddler containing something like

\define maketranslt( tid) {{$:/trnslt-en2##$(currentTiddler)$}}===={{$:/trnslt-nl##$(currentTiddler)$}}

<$list filter="[[$:/trnslt-en2]indexes[]]"> <br><<maketrans>>

</$list>

and search for ==== at the end o the line in your editor or your browser)

if you need multiple translations for one text e.g. Belgium using Dutch and French

\define maketranslt( tid) {{$:/trnslt-en2##$(currentTiddler)$}}:{{$:/trnslt-nl##$(currentTiddler)$}} ({{$:/trnslt-fr##$(currentTiddler)$}})

<$list filter="[[$:/trnslt-en2]indexes[]]"> <br><<maketrans>>

</$list>

and copy the output to a $:/trnslt-be tiddler, and extend the choose language tiddler with one button 

This is WORK IN PROGRESS there is still a lot to do do not use this TW  for storing imported data 

Is this the way to go? pull request?

It’s very interesting; I hope my comments don’t put you off. I’d certainly find it easier to review your changes if you could make a fork on GitHub.

It's just a proof of concept, it is one of many local forks, There are only a few new tiddlers : choose language, translations data tiddlers, small flags.svg, the remainder is a "global replace" of the  lingo macros..(I'm not a fan of github yet).

 

Best wishes

Jeremy

Thanks for your reply

Best regards

WiM 

[Mat]

On mobile so brief;

Regarding transl: it is frustrating to search some string and not find the tid because string was in lingo thing.

can the technique seen in aliases.tiddlyspot.com be used to hide inline strings, targetedr via span IDs, i.e hide the english and display the translation? It relies on css pseudo elements if wanting to do it purely css.

<:-)

[Jeremy Ruston]

Hi Mat

> can the technique seen in aliases.tiddlyspot.com be used to hide inline strings, targetedr via span IDs, i.e hide the english and display the translation? It relies on css pseudo elements if wanting to do it purely css.

The translation mechanism must work on the server as well as in the browser, so it has to be done in pure JavaScript.

Best wishes

Jeremy

>
> <:-)

>
> --
> You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
> To post to this group, send email to tiddl...@googlegroups.com.
> Visit this group at https://groups.google.com/group/tiddlywiki.

> To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/6b1fef1f-4e71-43d7-866f-b334e6e8f32d%40googlegroups.com.

[PMario]

Hi Wimm,

It seems you put a lot of work into it. ... but I have some concerns if it actually is an improvement, for me it looks like the opposite.

On Wednesday, July 6, 2016 at 12:23:17 PM UTC+2, wimm wrote:

The current implementation for interface language rquires 600+ separate tiddlers for each piece of text for each language. 

If you want to translate TW, you just need to checkout the github repo and modify: 62 different files. They create a lot of shadow tiddlers, that's right. But TW is optimized for small chunks of text.
IMO TW translations are already packed quite a lot.

my goal

 

- reduce the number of tiddlers (core plugin was getting very long list of tiddlers) (partial done, language tiddlers are still present but not used, a few for long texts will remain).

why? There are a lot of tiddlers that also create dynamic lists. eg: tiddler-types dropdown. or Fields.multids ...

 - If you throw all of them into one big unmanageable file you remove the dynamic features.

 - Every shadow tiddler can be modified separately by the user. eg: You can add your own comments, tags and what not ....
   - If we remove this possibility, imo it's a "no go" regression.
   - With the existing mechanism, one tiddler will be affected by user editing.

If a user edits the "big" translation tiddler, she basically sticks with it for ever, since it is not updated by the core update mechanism anymore ..

- immediate language translation switching without reload (partial done, not yet for already present buttons)

This is status quo.
 

- make code more readable by including original english text in the code instead of lingo-base and lingo macro (done)

Making the core UI code more readable, would be nice, but having a translate "english to English" table (aka hash table) imo will increase complexity over time a lot. Not now but in the future.

Eg: just think about this hash table in 5 years from now.
For TW backwards compatibility is important. So you have to stick forever with the English keys from now on, even if it changes meaning in the future.

There is a reason, why the tiddler names use namespaces. Many filtered lists rely on those tiddler namespaces.

Your suggestion{{$:/trnslt##close others}}  imo removes the namespace concept, which influences a lot of filter operators and a core concept.

 

- easier to translate, checking for missing entries (one file per language), switch back to English to find the search key, for the translation)

As I wrote there are only 62 files to manage and the shorter they are, the easier it is to find the differences. I use the Meld program to check the differences between eg: english and german

So using Meld and the github commit history makes it incredibly easy to follow the language changes. If following the changes is hard, we need better documentation.

just some thoughts.
-mario

[wimm]

jeremy, mario, jed & mat

maybe just to clarify,
All above changes were made to a single tiddlywiki.html file using some smart vi regular expressions and the internal editor of tw.

The last time I used nodes.js for tw was at least two and a half years ago. That might explain why I don't use github ;-)

long keys:
if the keys get too long, you can easily abbreviate them as long they contain the start of the sentence. please note the translation can contain wikitext including transclusions for e.g. short sentences in the same or other trnslt file.

sv source code (for educational purposes, school..):
@mat if you want the strings in the html source to be translated . you could use for instance a vim command file (generated by a simple tiddler) to replace all english key texts by Svenska/Swedish(?) an additional  sv 2 sv trnslt is required, it would then show the texts. (append the sv mutids and tid files, see tw use en2 and the file to create english to sv translation. use that translation to create sv to sv like the belgium replace nl and fr by sv)

replace the macro for the Belgium nl fr trnslt -from previous post - in something like (not tested I'm om my phone now)
:%s/trnslt##{{trnslt##\$:\/trnslt-en2##\$(currentTiddler)\$/trnslt##{{trsnlt-sv##$(currentTiddler)/g

because all translations for one language are together in one tiddler(strong cohesion) a 4 line tiddler can generate the vim command to do a full translation into svenska. copy command to a file called sv.vim
gvim -s sv.vim mytiddlywiki.html will   creare the file you created . use gvim file menu to save and open new  html file in browser

kr WiM

[wimm]

To remove your concern about the contents of the English to English translation in let's say 5 years. We can use the power of Tiddlywiki and gvim to update all in less than 60 minutes.

 

So when the translation En-En contains too many modified translations A vim command very similar to the Svenka translation can be created using TW to update the English keys in the html source (not tested yet)

create a tiddler (working versions see http://mutlilanguage.tiddlyspot.com)

\define maketranslt( tid) :%s/trnslt##{{trnslt##\$(currentTiddler)\$/trnslt##{{trsnlt-en2##$(currentTiddler)/g

<$list filter="[[$:/trnslt-en2]indexes[]]"> <br><<maketrans>>

</$list>

Copy the output to a file called en.vim and use vim-s en.vim tiddywiki.html to update the html source

At the same time you then need to update the translation files for each language

For the updated English translation file

\define maketranslt( tid) {{$:/trnslt-en2##$(currentTiddler)$}}{{$:/trnslt-en2##$(currentTiddler)$}}

<$list filter="[[$:/trnslt-en2]indexes[]]"> <br><<maketrans>>

</$list>

copy the preview to a new data tiddler e.g. translt-en2  (don't use $:/trnslt-en2 at this moment)

for Dutch

\define maketranslt( tid) {{$:/trnslt-en2##$(currentTiddler)$}}{{$:/trnslt-nl##$(currentTiddler)$}}

<$list filter="[[$:/trnslt-en2]indexes[]]"> <br><<maketrans>>

</$list>

for all other languages replace nl in the Dutch example by the appropriate language abbreviation.

finally rename all trnslt-xx to $:/trnslt-xx