Website/documentation translation progress (was: Re: test branch)

[Marek Marczykowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

On Sat, Dec 28, 2019 at 10:29:12AM +0100, Maya wrote:
> good morning ...! :)
>
> name of test branch is translation-wf-test

After embarrassingly long delay, I've finally finished review of this
and made some modifications. The most important is adding
qubes-translated as a collection to _config.yml, which allowed to get
rid of all the symlinks!
Also, I've squashed the whole history of that branch to just few
commits. BTW, Maya, please tell me what author (name + email) should I set
there, you seems to use different ones for different commits.

I've pushed the result here:
https://github.com/marmarek/qubesos.github.io/tree/translation

Then I've executed all the translation steps from the README (including
push + pull to transifex), built the site and uploaded it here:

https://wwwtest.qubes-os.org/

Generated changes in sources pushed to the translation-output branch in
qubesos.github.io, qubes-doc, qubes-translated.

I've downloaded only some languages (de, fr, es), we have way more
of them enabled (most with minimal or no translations). Bulgarian is not
one of them...
Also, while downloading from Transifex, I've used --minimum-perc=5 option, but
I don't think it really worked. I was expecting to download only files
with at least 5% content translated.

Menu and other main website parts are setup only for English
(obviously), German, and Bulgarian, so on other languages things are
missing.

I've tried to run html-proofer locally, but ran out of memory :/
Will get travis into shape for that (now it is confused with different
branches).

There are various issues to be solved:
- sometimes 'layout' in frontmatter gets translated, breaking
pages (example: https://wwwtest.qubes-os.org/de/donate/)
- on some (most?) pages, translation disclaimer is included twice
- some not translated pages (those where transifex didn't returned
anything, maybe because of --minimum-perc=5) are returning 404
- I think in such cases there should be a different disclaimer for not
translated page (maybe inviting to help with translation?) and the
original page should be included (or redirected to?)
- documentation index will fairly easy get out of sync - generated
one that Andrew done for version-specific documentation would be very
useful here
- translation of base site elements (top menu, footer menu etc) isn't
integrated with transifex - it's about files in _data/
- language selection IMO should be a drop-down instead of a flat list
(especially when it'll get longer) - does anybody know how to do
that?

Next steps:
- solve critical issues (first: define which are critical)
- remove all resources from transifex
- decide what languages should we enable and generally cleanup
transifex
- upload fresh content (I believe "translation memory" will remain
there, so some content will automatically get translated)
- enable automated translation in transifex for initial draft (google
translate)
- enable calling "the script" (I give up waiting for having proper "github
integration" enabled) - this can be done by the same script downloads
other submodules
- open transifex for translators

Anything missing?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl4mheUACgkQ24/THMrX
1ywbqAf/a45dArgdi4yqkXCDtR65m/6MKDvb382is2w2FwyEjFG1De8GgugHSV3F
vYbKtWtoJZb8kgaXhP5jBWtR7g0hDtFxBpumglF974tgfwpqEp9nDVdPJdOO5RLm
cICK17OxDK5XDmfSIbHMqERp+ejan6o/qMDOocGyyIZu0XkAcwsWx2q0699Qao0p
uCIraJC0pJ0gfl2cryR5Ak9v9o2Q/xUERhz4zXjHOkjQB+rJUmaPgImn1DSfPJdM
1wmcESfNvH5Q3ckp67aqSbfjqod9l0ARoX3mTDTekM9BIgsmx9tLFTL8dLsNvWTE
7742WBoAg5MFVjZUdZcknxtUiDGjYw==
=KnT4
-----END PGP SIGNATURE-----

[Tobias Killer]

Hi,

Am 21.01.20 um 06:02 schrieb Marek Marczykowski:

> Hi,
>
> On Sat, Dec 28, 2019 at 10:29:12AM +0100, Maya wrote:
>> good morning ...! :)
>
>> name of test branch is translation-wf-test
>
> After embarrassingly long delay, I've finally finished review of this
> and made some modifications. The most important is adding
> qubes-translated as a collection to _config.yml, which allowed to get
> rid of all the symlinks!

Nice!

> Also, I've squashed the whole history of that branch to just few
> commits. BTW, Maya, please tell me what author (name + email) should I set
> there, you seems to use different ones for different commits.
>
> I've pushed the result here:
> https://github.com/marmarek/qubesos.github.io/tree/translation

Some translation-related files seem to be in `_util/` but not in
`_util/_translation_utils/`. These files should be moved.

>
> Then I've executed all the translation steps from the README (including
> push + pull to transifex), built the site and uploaded it here:
>
> https://wwwtest.qubes-os.org/

Great for testing, thank you!

>
> Generated changes in sources pushed to the translation-output branch in
> qubesos.github.io, qubes-doc, qubes-translated.
>
> I've downloaded only some languages (de, fr, es), we have way more
> of them enabled (most with minimal or no translations). Bulgarian is not
> one of them...
> Also, while downloading from Transifex, I've used --minimum-perc=5 option, but
> I don't think it really worked. I was expecting to download only files
> with at least 5% content translated.

Maybe we shouldn't take care of the fraction of translation? IMHO it
could be easier to provide always *all* links to *all* supported
languages for *every* page. A button "Translate this page" could be
added to the sidebar, for example.

>
> Menu and other main website parts are setup only for English
> (obviously), German, and Bulgarian, so on other languages things are
> missing.
>
> I've tried to run html-proofer locally, but ran out of memory :/
> Will get travis into shape for that (now it is confused with different
> branches).
>
> There are various issues to be solved:
> - sometimes 'layout' in frontmatter gets translated, breaking
> pages (example: https://wwwtest.qubes-os.org/de/donate/)

That's one reason why IMHO the frontmatter should be stripped off before
uploading to Transifex. It's just too critical.

> - on some (most?) pages, translation disclaimer is included twice

Not critical but should be avoided.

> - some not translated pages (those where transifex didn't returned
> anything, maybe because of --minimum-perc=5) are returning 404
> - I think in such cases there should be a different disclaimer for not
> translated page (maybe inviting to help with translation?) and the
> original page should be included (or redirected to?)

Untranslated parts should always be visible since those could make the
difference. Maybe, we could highlight those parts in screaming red and
add a disclaimer saying that red means untranslated or not up-to-date?

> - documentation index will fairly easy get out of sync - generated
> one that Andrew done for version-specific documentation would be very
> useful here

Yes, indeed!

> - translation of base site elements (top menu, footer menu etc) isn't
> integrated with transifex - it's about files in _data/

No spontaneous idea, sorry. :(

> - language selection IMO should be a drop-down instead of a flat list
> (especially when it'll get longer) - does anybody know how to do
> that?

Do you mean something like this?:
https://www.w3schools.com/Css/css_dropdowns.asp

>
> Next steps:
> - solve critical issues (first: define which are critical)
> - remove all resources from transifex
> - decide what languages should we enable and generally cleanup
> transifex
> - upload fresh content (I believe "translation memory" will remain
> there, so some content will automatically get translated)

Yes, I guess, too. Otherwise, maybe the translations will just be
proposed and the top proposal can be pulled in via a keyboard shortcut.

> - enable automated translation in transifex for initial draft (google
> translate)

Could help but also make things more difficult. Let's try and see what
happens!

> - enable calling "the script" (I give up waiting for having proper "github
> integration" enabled) - this can be done by the same script downloads
> other submodules

What do you mean? I don't understand.

> - open transifex for translators
>
> Anything missing?
>
>

What about fragments in URLs (the part beyond `#`)? They are used to
link to the sections of a page.

Thank you for your work!

Cheers,
Tobias

[Marek Marczykowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Jan 21, 2020 at 08:45:19PM +0100, Tobias Killer wrote:
> Hi,
>
> Am 21.01.20 um 06:02 schrieb Marek Marczykowski:
> > Hi,
> >
> > On Sat, Dec 28, 2019 at 10:29:12AM +0100, Maya wrote:
> >> good morning ...! :)
> >
> >> name of test branch is translation-wf-test
> >
> > After embarrassingly long delay, I've finally finished review of this
> > and made some modifications. The most important is adding
> > qubes-translated as a collection to _config.yml, which allowed to get
> > rid of all the symlinks!
>
> Nice!
>
> > Also, I've squashed the whole history of that branch to just few
> > commits. BTW, Maya, please tell me what author (name + email) should I set
> > there, you seems to use different ones for different commits.
> >
> > I've pushed the result here:
> > https://github.com/marmarek/qubesos.github.io/tree/translation
>
> Some translation-related files seem to be in `_util/` but not in
> `_util/_translation_utils/`. These files should be moved.

I've done that on purpose, with intention of removing _translation_utils
subdir completely (currently only some test scripts remains there).
There aren't that much scripts there (3) to create overly long path for
them.

> >
> > Then I've executed all the translation steps from the README (including
> > push + pull to transifex), built the site and uploaded it here:
> >
> > https://wwwtest.qubes-os.org/
>
> Great for testing, thank you!
>
> >
> > Generated changes in sources pushed to the translation-output branch in
> > qubesos.github.io, qubes-doc, qubes-translated.
> >
> > I've downloaded only some languages (de, fr, es), we have way more
> > of them enabled (most with minimal or no translations). Bulgarian is not
> > one of them...
> > Also, while downloading from Transifex, I've used --minimum-perc=5 option, but
> > I don't think it really worked. I was expecting to download only files
> > with at least 5% content translated.
>
> Maybe we shouldn't take care of the fraction of translation? IMHO it
> could be easier to provide always *all* links to *all* supported
> languages for *every* page. A button "Translate this page" could be
> added to the sidebar, for example.

Yes, such button is a good idea. And even for fully translated pages it
makes sense, for improving/updating the translation.

> > Menu and other main website parts are setup only for English
> > (obviously), German, and Bulgarian, so on other languages things are
> > missing.
> >
> > I've tried to run html-proofer locally, but ran out of memory :/
> > Will get travis into shape for that (now it is confused with different
> > branches).
> >
> > There are various issues to be solved:
> > - sometimes 'layout' in frontmatter gets translated, breaking
> > pages (example: https://wwwtest.qubes-os.org/de/donate/)
>
> That's one reason why IMHO the frontmatter should be stripped off before
> uploading to Transifex. It's just too critical.

On the other hand, some attributes should be translated - for example
title.
I think the post-processing script can easily access both source and
translated frontmatter and combine the final one from them (for example
take title from translated and the rest from source).

> > - on some (most?) pages, translation disclaimer is included twice
>
> Not critical but should be avoided.
>
> > - some not translated pages (those where transifex didn't returned
> > anything, maybe because of --minimum-perc=5) are returning 404
> > - I think in such cases there should be a different disclaimer for not
> > translated page (maybe inviting to help with translation?) and the
> > original page should be included (or redirected to?)
>
> Untranslated parts should always be visible since those could make the
> difference. Maybe, we could highlight those parts in screaming red and
> add a disclaimer saying that red means untranslated or not up-to-date?

I have no idea how to do that.
I was rather expecting that the whole file would be skipped if it isn't
translated in at least X percents. The we could fill such missing files
with some placeholders or such.
But apparently it isn't how it works.

> > - documentation index will fairly easy get out of sync - generated
> > one that Andrew done for version-specific documentation would be very
> > useful here
>
> Yes, indeed!
>
> > - translation of base site elements (top menu, footer menu etc) isn't
> > integrated with transifex - it's about files in _data/
>
> No spontaneous idea, sorry. :(
>
> > - language selection IMO should be a drop-down instead of a flat list
> > (especially when it'll get longer) - does anybody know how to do
> > that?
>
> Do you mean something like this?:
> https://www.w3schools.com/Css/css_dropdowns.asp

Yes!

> > Next steps:
> > - solve critical issues (first: define which are critical)
> > - remove all resources from transifex
> > - decide what languages should we enable and generally cleanup
> > transifex
> > - upload fresh content (I believe "translation memory" will remain
> > there, so some content will automatically get translated)
>
> Yes, I guess, too. Otherwise, maybe the translations will just be
> proposed and the top proposal can be pulled in via a keyboard shortcut.
>
> > - enable automated translation in transifex for initial draft (google
> > translate)
>
> Could help but also make things more difficult. Let's try and see what
> happens!
>
> > - enable calling "the script" (I give up waiting for having proper "github
> > integration" enabled) - this can be done by the same script downloads
> > other submodules
>
> What do you mean? I don't understand.

The steps mentioned in README. I've executed them manually by copy&paste
into terminal. This needs to be called automatically on some schedule
(or detected change).

> > - open transifex for translators
> >
> > Anything missing?
> >
> >
>
> What about fragments in URLs (the part beyond `#`)? They are used to
> link to the sections of a page.

That's a good point. When section title gets translated, such URL
fragments will change too. Do we have any plan what to do about that?

> Thank you for your work!
>
> Cheers,
> Tobias
>

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl4nYegACgkQ24/THMrX
1yzuswf/esfWnIDiexhSCVWt4uo5VDJRWDCx5KLOOhb0GdRv4dSZavfS2FxhAO5U
4+vKf/MC7xPLJMupoNCzuKsirr0ErdtFjJydKS2aJrMFgjOgjG0xTikeIeYeVoPH
e2ZqLJ/PIrv0RdLSJzCxa/qiRdGvh9fPRTF/SVDsbRYSjm1qGuhX8E9Yl5+D4J7+
KwvE7g4rmDa0VzHfh+I7PBb0AxoEHN46f3hTWLvcUszgwPJCD8mEFTBiTTptasYa
iwEIBdGEpfoHgixaXCQRD4eT/9zbCMvV5pENBaGHh3QOfcZ8Tw4/OX2xezxPw2xY
X75wtU/YXS9UZqeqA9+0Ub0ZR/OSaA==
=VplX
-----END PGP SIGNATURE-----

[Tobias Killer]

Am 21.01.20 um 21:41 schrieb Marek Marczykowski:
> On Tue, Jan 21, 2020 at 08:45:19PM +0100, Tobias Killer wrote:
>> Hi,
>
>> Am 21.01.20 um 06:02 schrieb Marek Marczykowski:
>>> Hi,
>>>
>>> On Sat, Dec 28, 2019 at 10:29:12AM +0100, Maya wrote:
>>>> good morning ...! :)
>>>
>>>> name of test branch is translation-wf-test
>>>
>>> After embarrassingly long delay, I've finally finished review of this
>>> and made some modifications. The most important is adding
>>> qubes-translated as a collection to _config.yml, which allowed to get
>>> rid of all the symlinks!
>
>> Nice!
>
>>> Also, I've squashed the whole history of that branch to just few
>>> commits. BTW, Maya, please tell me what author (name + email) should I set
>>> there, you seems to use different ones for different commits.
>>>
>>> I've pushed the result here:
>>> https://github.com/marmarek/qubesos.github.io/tree/translation
>
>> Some translation-related files seem to be in `_util/` but not in
>> `_util/_translation_utils/`. These files should be moved.
>
> I've done that on purpose, with intention of removing _translation_utils
> subdir completely (currently only some test scripts remains there).
> There aren't that much scripts there (3) to create overly long path for
> them.
>

Ah, now I see. :)

>>>
>>> Then I've executed all the translation steps from the README (including
>>> push + pull to transifex), built the site and uploaded it here:
>>>
>>> https://wwwtest.qubes-os.org/
>
>> Great for testing, thank you!
>
>>>
>>> Generated changes in sources pushed to the translation-output branch in
>>> qubesos.github.io, qubes-doc, qubes-translated.
>>>
>>> I've downloaded only some languages (de, fr, es), we have way more
>>> of them enabled (most with minimal or no translations). Bulgarian is not
>>> one of them...
>>> Also, while downloading from Transifex, I've used --minimum-perc=5 option, but
>>> I don't think it really worked. I was expecting to download only files
>>> with at least 5% content translated.
>
>> Maybe we shouldn't take care of the fraction of translation? IMHO it
>> could be easier to provide always *all* links to *all* supported
>> languages for *every* page. A button "Translate this page" could be
>> added to the sidebar, for example.
>
> Yes, such button is a good idea. And even for fully translated pages it
> makes sense, for improving/updating the translation.
>

Exactly.
I've just found out that we can link to the page on Transifex where a
specific file can be translated into a specific language. The URL is
built like

https://www.transifex.com/otf/qubes/translate/#LANG_CODE/RESOURCE_NAME/

For example:
https://www.transifex.com/otf/qubes/translate/#de/_doc_user_common-tasks_copy-paste/

>>> Menu and other main website parts are setup only for English
>>> (obviously), German, and Bulgarian, so on other languages things are
>>> missing.
>>>
>>> I've tried to run html-proofer locally, but ran out of memory :/
>>> Will get travis into shape for that (now it is confused with different
>>> branches).
>>>
>>> There are various issues to be solved:
>>> - sometimes 'layout' in frontmatter gets translated, breaking
>>> pages (example: https://wwwtest.qubes-os.org/de/donate/)
>
>> That's one reason why IMHO the frontmatter should be stripped off before
>> uploading to Transifex. It's just too critical.
>
> On the other hand, some attributes should be translated - for example
> title.
> I think the post-processing script can easily access both source and
> translated frontmatter and combine the final one from them (for example
> take title from translated and the rest from source).

Okay, let's try!

>
>>> - on some (most?) pages, translation disclaimer is included twice
>
>> Not critical but should be avoided.
>
>>> - some not translated pages (those where transifex didn't returned
>>> anything, maybe because of --minimum-perc=5) are returning 404
>>> - I think in such cases there should be a different disclaimer for not
>>> translated page (maybe inviting to help with translation?) and the
>>> original page should be included (or redirected to?)
>
>> Untranslated parts should always be visible since those could make the
>> difference. Maybe, we could highlight those parts in screaming red and
>> add a disclaimer saying that red means untranslated or not up-to-date?
>
> I have no idea how to do that.
> I was rather expecting that the whole file would be skipped if it isn't
> translated in at least X percents. The we could fill such missing files
> with some placeholders or such.
> But apparently it isn't how it works.

The highlighting thing seems to be difficult. I thought the XLIFF files
could help but I don't know how to download and read them.

Apart from the highlighting thing, it should be possible to download the
untranslated parts of a file, too. For example, on the dashboard at
Transifex, I select a resource, e.g. "_doc" and I get the opportunity to
"Download for use". This way, we get the file as translated as possible.
Untranslated parts are included. Is this what we need?

>
>>> - documentation index will fairly easy get out of sync - generated
>>> one that Andrew done for version-specific documentation would be very
>>> useful here
>
>> Yes, indeed!
>
>>> - translation of base site elements (top menu, footer menu etc) isn't
>>> integrated with transifex - it's about files in _data/
>
>> No spontaneous idea, sorry. :(
>
>>> - language selection IMO should be a drop-down instead of a flat list
>>> (especially when it'll get longer) - does anybody know how to do
>>> that?
>
>> Do you mean something like this?:
>> https://www.w3schools.com/Css/css_dropdowns.asp
>
> Yes!
>

Ok, let's try, too!

>>> Next steps:
>>> - solve critical issues (first: define which are critical)
>>> - remove all resources from transifex
>>> - decide what languages should we enable and generally cleanup
>>> transifex
>>> - upload fresh content (I believe "translation memory" will remain
>>> there, so some content will automatically get translated)
>
>> Yes, I guess, too. Otherwise, maybe the translations will just be
>> proposed and the top proposal can be pulled in via a keyboard shortcut.
>
>>> - enable automated translation in transifex for initial draft (google
>>> translate)
>
>> Could help but also make things more difficult. Let's try and see what
>> happens!
>
>>> - enable calling "the script" (I give up waiting for having proper "github
>>> integration" enabled) - this can be done by the same script downloads
>>> other submodules
>
>> What do you mean? I don't understand.
>
> The steps mentioned in README. I've executed them manually by copy&paste
> into terminal. This needs to be called automatically on some schedule
> (or detected change).
>

Okay, I see.

>>> - open transifex for translators
>>>
>>> Anything missing?
>>>
>>>
>
>> What about fragments in URLs (the part beyond `#`)? They are used to
>> link to the sections of a page.
>
> That's a good point. When section title gets translated, such URL
> fragments will change too. Do we have any plan what to do about that?
>

The IMHO best solution I could find is this:
https://github.com/QubesOS/qubes-issues/issues/3547#issuecomment-374270155
The question is how to write the script correctly. I've an idea how to
do this, but it's not proven if it will work.

Another thing to mention: There is that script which (?joins all lines
and then?) inserts a line break after each period in a text file.
https://github.com/gsganden/sentence_separator
Since it doesn't seem to take care of the Markdown syntax, we shouldn't
use it untested. Maybe we could write something on our own? The thing is
that it could help translation since the Translation Memory can work
better then.

Now, there is some work to do. Should we "fork" your `translation`
branch in order to implement the missing scripts?

[Marek Marczykowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Jan 22, 2020 at 01:14:30PM +0100, Tobias Killer wrote:
> Now, there is some work to do. Should we "fork" your `translation`
> branch in order to implement the missing scripts?

Yes, I think that would be the easiest way.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl4odR4ACgkQ24/THMrX
1ywZPwgAlLx9wK3shfrC8v0zKVqngWiqYEE+GQNkynUtecMy5ZScsCfi7revEDAQ
nOpHCuK0jdGTArZDoctZElU+wHvk+EpAodmjxKN7yWBMg42wL9TIVXClgiS+Oci3
igA3sSJJA+iHIYAwfxwfBpF74msgvqSJ94a+9Ml5zGIbrkFwkdP29BUrnfMbyeoY
GoW5sjoZqyQRVsZBaOFuPI6ZKUJGs0Yp+FwAyHgfe1cJ4qDep/u9kqQWwyjYOCAJ
yoK4ceihyBlBlOImrsfEhW/zQvlk6g5f+2nNAaSwFhXPw+aC5eBZsCBwnh8w1xbs
AXUM2raHp2LfBX9lpjrDFMZy9yc+vg==
=KUs5
-----END PGP SIGNATURE-----

[Tobias Killer]

Am 21.01.20 um 21:41 schrieb Marek Marczykowski:

> On Tue, Jan 21, 2020 at 08:45:19PM +0100, Tobias Killer wrote:
>> Do you mean something like this?:
>> https://www.w3schools.com/Css/css_dropdowns.asp
>
> Yes!

Hello, Marek,

See the attachment for a patch for this. The patch is based on commit
ab6325e (`translation-output`).

But be warned:

- I couldn't reproduce the given output on https://wwwtest.qubes-os.org/
so, I couldn't test it there. Instead, I got different output and
couldn't test it as I wished but I think the patch is more correct than
wrong.
- I couldn't "fork" your repo
https://github.com/marmarek/qubesos.github.io/ since I've already one
with the same name on my GitHub account. That's why I'm sending you a patch.
- I hope that submodules won't break since the original diff contains
the fact that the submodules have been modified (due to recursive clone
of your repo). The patch in the attachment of this email has been
manually reduced by these facts.

Also, you may have a look at the simple example (see attachment) for the
language switch slightly adapted for the Qubes OS website. The patch is
based on it.

I hope that this email is useful for you.

Cheers,
Tobias Killer

[Tobias Killer]

Am 21.01.20 um 21:41 schrieb Marek Marczykowski:

> On Tue, Jan 21, 2020 at 08:45:19PM +0100, Tobias Killer wrote:
>> Am 21.01.20 um 06:02 schrieb Marek Marczykowski:

>>> - language selection IMO should be a drop-down instead of a flat list
>>> (especially when it'll get longer) - does anybody know how to do
>>> that?
>
>> Do you mean something like this?:
>> https://www.w3schools.com/Css/css_dropdowns.asp
>
> Yes!
>

Since a language switch as of our current version, having only the
abbreviated name of a potentially unknown language ("en"), is of limited
obvious use, we should add an expressive symbol like this one:
https://icons8.com/icon/111495/translation
next to or in combination with it.

What do you think?

If yes, should we download or embed it? I prefer downloading due to
independency.

Cheers,
Tobias

[Maya]

Hello Marek, hello all,

i was off the grid for this time due to private issues, just now reading
this :)

I will go over the email thread first thing tomorrow eve and see how I
can contribute further.

If there is smth clear to do asap, just say so. If not will proceed as
planned.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Jan 26, 2020 at 02:11:41PM +0100, Tobias Killer wrote:
> Am 21.01.20 um 21:41 schrieb Marek Marczykowski:
> > On Tue, Jan 21, 2020 at 08:45:19PM +0100, Tobias Killer wrote:
> >> Do you mean something like this?:
> >> https://www.w3schools.com/Css/css_dropdowns.asp
> >
> > Yes!
>
> Hello, Marek,
>
> See the attachment for a patch for this. The patch is based on commit
> ab6325e (`translation-output`).
>
> But be warned:
>
> - I couldn't reproduce the given output on https://wwwtest.qubes-os.org/
> so, I couldn't test it there. Instead, I got different output and
> couldn't test it as I wished but I think the patch is more correct than
> wrong.

I've applied the patch and updated wwwtest.qubes-os.org

> - I couldn't "fork" your repo
> https://github.com/marmarek/qubesos.github.io/ since I've already one
> with the same name on my GitHub account. That's why I'm sending you a patch.

All forks are connected in a sense you can create pull request between
them. To download the right branch you just need to add remote:

git remote add marmarek https://github.com/marmarek/qubesos.github.io

Then you can do git fetch marmarek.

On Sun, Jan 26, 2020 at 04:57:27PM +0100, Tobias Killer wrote:
> Am 21.01.20 um 21:41 schrieb Marek Marczykowski:
> > On Tue, Jan 21, 2020 at 08:45:19PM +0100, Tobias Killer wrote:
> >> Am 21.01.20 um 06:02 schrieb Marek Marczykowski:

> >>> - language selection IMO should be a drop-down instead of a flat list
> >>> (especially when it'll get longer) - does anybody know how to do
> >>> that?
> >

> >> Do you mean something like this?:
> >> https://www.w3schools.com/Css/css_dropdowns.asp
> >
> > Yes!
> >
>

> Since a language switch as of our current version, having only the
> abbreviated name of a potentially unknown language ("en"), is of limited
> obvious use, we should add an expressive symbol like this one:
> https://icons8.com/icon/111495/translation
> next to or in combination with it.
>
> What do you think?

Yes, adding some indicator what is that is a good idea. This icon looks
good. On some websites I see globe icon for that, but I think both are
good.

> If yes, should we download or embed it? I prefer downloading due to
> independency.

Yes, I'd prefer downloading and adding into our repository.

Another thing is, I think the menu needs some background. Otherwise the
text overlaps with site content and looks ugly (see doc index for
example).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl4uH+cACgkQ24/THMrX
1yy8nAf/eX1PmGfuivo3bHya3RonvkVOsuGvExzAhp4pCtDACrCiOxc07PtoaRhY
nmTXEwTIXavvXQ5s9xP3wrjzicYp0+bOcYhqWkaIopS41iQcKE+RxioctnmCMHTg
LttVXsQgAqqlnL2cM0HyBhf94wZDC1W81K/83l1Pzzb7ME4qVQTRyBRfcmZZQQ+1
S6yo2AcigJ0rkrYQN12prDVfR4BeddHOY2+CpgXjJWVhYR2JNakBcVh81n0q1tdJ
VhQGPOug3/RktK8w0GjhMuSBqqeQsgmcWQI7nUv+N/3cqDNrYOkbQLhr0LBVDKkC
+YjJjYWEzkFS0ki12s1BNfXXSCHVgQ==
=euNA
-----END PGP SIGNATURE-----

[Tobias Killer]

Hi,

Am 21.01.20 um 06:02 schrieb Marek Marczykowski:

> On Sat, Dec 28, 2019 at 10:29:12AM +0100, Maya wrote:
> Menu and other main website parts are setup only for English
> (obviously), German, and Bulgarian, so on other languages things are
> missing.
>

> [...]

> There are various issues to be solved:

> [...]> - translation of base site elements (top menu, footer menu etc) isn't

> integrated with transifex - it's about files in _data

I found out how we can store data in `_data` into files named like
'de-DE.md' and read those data. This way, we can keep an "original" file
`architecture.yml` in English and just add other files named like
"architecture_de-DE.yml" in German into `_data`. I guess it's also
possible to move them to a dedicated directory `de-DE`.

Do you want to see a demo? You can checkout my good old repo
https://github.com/tokideveloper/langswitch-prototype
and have a look at
http://127.0.0.1:4000/test_data/
and the files
/pages/test_data/test_data.md among the files in /_data/l10n/


> [...]
> Next steps:
> [...]

> - decide what languages should we enable and generally cleanup
> transifex

I suggest to have a file `_data` with just a list of all provided
languages. This way, we can simply turn languages on and off and the
pure existence of a translated `architecture_de-DE.yml` file wouldn't be
an indicator for provision. For my example, I need such a list and I'm
not sure if it is possible to make it work without it.

What do you think?

Cheers,
Tobias

[Marek Marczykowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Mon, Jan 27, 2020 at 05:15:47PM +0100, Tobias Killer wrote:
> Hi,
>
> Am 21.01.20 um 06:02 schrieb Marek Marczykowski:
> > On Sat, Dec 28, 2019 at 10:29:12AM +0100, Maya wrote:
> > Menu and other main website parts are setup only for English
> > (obviously), German, and Bulgarian, so on other languages things are
> > missing.
> >
> > [...]
> > There are various issues to be solved:
> > [...]> - translation of base site elements (top menu, footer menu etc) isn't
> > integrated with transifex - it's about files in _data
> I found out how we can store data in `_data` into files named like
> 'de-DE.md' and read those data. This way, we can keep an "original" file
> `architecture.yml` in English and just add other files named like
> "architecture_de-DE.yml" in German into `_data`. I guess it's also
> possible to move them to a dedicated directory `de-DE`.
>
> Do you want to see a demo? You can checkout my good old repo
> https://github.com/tokideveloper/langswitch-prototype
> and have a look at
> http://127.0.0.1:4000/test_data/
> and the files
> /pages/test_data/test_data.md among the files in /_data/l10n/

I see you use site.data.l10n[l], I didn't know you can use this syntax
instead of just attributes ('.' separated). This sounds like a good
idea.
Ideally we'd have translated data files in qubes-translated repository,
is it possible this way?

> > [...]
> > Next steps:
> > [...]
> > - decide what languages should we enable and generally cleanup
> > transifex
>
> I suggest to have a file `_data` with just a list of all provided
> languages. This way, we can simply turn languages on and off and the
> pure existence of a translated `architecture_de-DE.yml` file wouldn't be
> an indicator for provision. For my example, I need such a list and I'm
> not sure if it is possible to make it work without it.
>
> What do you think?

Yes, explicit list of languages in _data is a good idea.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl4w5DQACgkQ24/THMrX
1yxvnwf+KGt0QCeL0/JQkSjRNrPo9RIdrhQaUeehcuyM1BmNLmuNcnPacwnkhuiE
Zx8pC3pXWTafP60KBx4AzW2jBoxKlFo0/zbIpTMSLqXJqyZ2hh988bAwng9cOw+w
NMoPm1fWpmF6uMF2Sj9WLBPEBW70HsRzTcGoaDPKHIa2LKTpQMte23BrCFyqnTsC
u8LkyqWks4qwDb4+EMJ92g7fCn98HNhUKnAttDnBROWIb8Autljbd80NlzreQUuv
HBLWr+osqnrmuW3yWYhEWABfeJtzO20zatO7O4FSXPYutSkBcdChnIjND8XFpcdk
nc+N8RomZVVShu31Uj26NHR9qd4K9w==
=SNgy
-----END PGP SIGNATURE-----

[Tobias Killer]

Am 29.01.20 um 02:47 schrieb Marek Marczykowski:

I'm not sure if cross-referencing data is possible. Maybe we could do
this by using submodules? Like linking to a submodule
`qubes-website-translated-data`?

Also, I'm not sure how the whole thing shall work. Is there already a
description for that?

IMHO I'd prefer the following (albeit a little bit "big"):

1. Create a new repo `qubes-website` containing the whole content of the
current `qubesos.github.io`. Assuming `qubesos.github.io` is empty
afterwards.
2. Regularily upload untranslated content from `qubes-website`,
`qubes-doc` etc. up to Transifex.
3. Regularily download translated content from Transifex to
`qubes-translated`.
4. Regularily merge `qubes-website`, `qubes-doc` etc. and
`qubes-translated` to `qubesos.github.io`.

Advantages:

- All and only the online content (whether translated or not) is located
at `qubesos.github.io`. No missing parts or submodules anywhere else. No
split data or content.
- (add some here)

Disadvantages:

- A modification on `qubes-website` etc. does not go online immediately
(if not scripted otherwise).
- (add some here)

[Tobias Killer]

Am 29.01.20 um 15:04 schrieb Tobias Killer:

I forgot to say that in the new `qubesos.github.io`, there should be a
directory for every language – even for `en`. All languages shall be
equal in every way and have the same structure everywhere.

[Tobias Killer]

Am 29.01.20 um 02:47 schrieb Marek Marczykowski:

> On Mon, Jan 27, 2020 at 05:15:47PM +0100, Tobias Killer wrote:
>> Hi,
>
>> Am 21.01.20 um 06:02 schrieb Marek Marczykowski:

>>> There are various issues to be solved:
>>> [...]> - translation of base site elements (top menu, footer menu etc) isn't
>>> integrated with transifex - it's about files in _data
>> I found out how we can store data in `_data` into files named like
>> 'de-DE.md' and read those data. This way, we can keep an "original" file
>> `architecture.yml` in English and just add other files named like
>> "architecture_de-DE.yml" in German into `_data`. I guess it's also
>> possible to move them to a dedicated directory `de-DE`.
>
>> Do you want to see a demo? You can checkout my good old repo
>> https://github.com/tokideveloper/langswitch-prototype
>> and have a look at
>> http://127.0.0.1:4000/test_data/
>> and the files
>> /pages/test_data/test_data.md among the files in /_data/l10n/
>
> I see you use site.data.l10n[l], I didn't know you can use this syntax
> instead of just attributes ('.' separated). This sounds like a good
> idea.
> Ideally we'd have translated data files in qubes-translated repository,
> is it possible this way?
>

After some research, I come to the conclusion that we can simply use
page files (via collections also across repos) as data files by storing
the data to the YAML front matter. We already use it here:

{% assign posts = site.pages | concat: site.doc | concat:
site.qubes-translated | where:'ref', page.ref | sort: 'lang' %}

source:
https://github.com/marmarek/qubesos.github.io/blob/2eb3a55b1fb963468c1baf3d4af8fac7fb9ed741/_includes/header.html#L32

And here is the manual:

https://jekyllrb.com/docs/collections/#output

We could have files like this one:

```
---
lang: en
is_data_file: true
data: "Good morning!"
---
```

… and let it translate to this:

```
---
lang: de
is_data_file: true
data: "Guten Morgen!"
---
```

The `is_data_file` item is for marking the file as a data file. Useful
for filtering.

What do you think about this idea?

Best regards,
Tobias Killer

[Tobias Killer]

Am 22.01.20 um 13:14 schrieb Tobias Killer:

> Am 21.01.20 um 21:41 schrieb Marek Marczykowski:
>> On Tue, Jan 21, 2020 at 08:45:19PM +0100, Tobias Killer wrote:
>>> Am 21.01.20 um 06:02 schrieb Marek Marczykowski:

>>>> There are various issues to be solved:
>>>> - sometimes 'layout' in frontmatter gets translated, breaking
>>>> pages (example: https://wwwtest.qubes-os.org/de/donate/)
>>> That's one reason why IMHO the frontmatter should be stripped off before
>>> uploading to Transifex. It's just too critical.
>>
>> On the other hand, some attributes should be translated - for example
>> title.
>> I think the post-processing script can easily access both source and
>> translated frontmatter and combine the final one from them (for example
>> take title from translated and the rest from source).
>
> Okay, let's try!

Here is the resulting script:

https://github.com/tokideveloper/langswitch-prototype/blob/master/_utils/merge_md_file_with_its_translated_version.py

It depends on PyYAML

https://pyyaml.org/

and must be invoked with exactly 3 parms:

```sh
$ merge_md_file_with_its_translated_version.py ORIGINAL_FILE.md
TRANSLATED_FILE.md LANGCODE
```

with LANGCODE being the code used in paths, `de`, for example.

Then it will print the merged file to stdout.

What do you think?

Best regards,
Tobias Killer

[Marta Marczykowska-Górecka]

Hi all, I've made an updated list of critical issues with the website
(based on the  https://wwwtest.qubes-os.org/de/intro/ ):

- Menu does not work in all languages: when I switch to documentation
and select 'es' language, most of the top bar menu vanishes

- Some links are missing (404) in the de version: Systemvoraussetzungen,
Zeritfizierte Hardware, Compatibility List, Installationsguide,
Verifizieren von Signaturen, Version Scheme, Source Code, News, Hilfe
und Support, Loslegen, OS architektur, Screenshots, FAQ, Video Tours,

- some pages in translated version seem to be missing CSS, e.g. "Von den
Experten", Forschung, Spenden

- "Spenden" site also has a problem with encoding, resulting in a
painful "Wie Sie spenden können"

- in DE version, "Melde ein Sicherheitsproblem" and Security Center are
completely broken and link to a catalog without a html file

Stay safe!
Marta Marczykowska-Górecka

[Tobias Killer]

Hi all,

Am 24.03.20 um 15:52 schrieb Marta Marczykowska-Górecka:

> Hi all, I've made an updated list of critical issues with the website
> (based on the  https://wwwtest.qubes-os.org/de/intro/ ):

Nice! Thank you!
Here are my assumptions:

> - Menu does not work in all languages: when I switch to documentation
> and select 'es' language, most of the top bar menu vanishes

... since `lang: es` support is not yet in `/_data/architecture.yml`.

Currently, all supported languages must also exist in that
`/_data/architecture.yml`.
It would be better to have different files (one for each language) since
it fits the Transifex workflow better.

> - Some links are missing (404) in the de version: Systemvoraussetzungen,
> Zeritfizierte Hardware, Compatibility List, Installationsguide,
> Verifizieren von Signaturen, Version Scheme, Source Code, News, Hilfe
> und Support, Loslegen, OS architektur, Screenshots, FAQ, Video Tours,

Seems to be a subrepo or Jekyll config problem for `wwwtest`. But I'm
not sure.

> - some pages in translated version seem to be missing CSS, e.g. "Von den
> Experten", Forschung, Spenden

Unsure where the errors come from (maybe from different sources?). Maybe
because the value of the key `permalink` has not been translated? Or of
`layout`?

The resulting HTML codes of

https://wwwtest.qubes-os.org/de/donate/
https://wwwtest.qubes-os.org/de/research/

have no `html` tags etc. at all. Maybe a layout problem?

> - "Spenden" site also has a problem with encoding, resulting in a
> painful "Wie Sie spenden können"

Maybe this mistake will vanish when the layout problem above has been
solved (leading to a charset definition in the HTML header).

> - in DE version, "Melde ein Sicherheitsproblem" and Security Center are
> completely broken and link to a catalog without a html file

"Melde ein Sicherheitsproblem" and "Security Center" point to the same page
https://wwwtest.qubes-os.org/de/security/
The translated `permalink` is empty for that page (as seen in
Transifex). That's why they are broken, I guess.



I like to add:

- The big blue `Q` always links to the English version. Should point to
the current selected lang IMHO.

> Stay safe!
> Marta Marczykowska-Górecka

Stay healty!
Tobias Killer

[mm]

Hello all, hello Marta, hello Tobi!

Thank you for the testing and the report!

I am aware of the bugs, and they've been fixed but currently frozen on
my overwhelmed

laptop.

There are several issues I tackled apart from fixing it and running
checks to see if

everything is at last OK, such as adjusted and enhanced existing
scripts, new ones

to tackle notranslate tags via transifex REST API and responsive layout
and such.

A mail draft for the qubes-translation list with status and all the
flavours and

a few suggestions is also in laptop lockdown :D

Right now, I have a lot of workload at my current job,

a nice exam next week and do not have time to work on further how to
tackle the

lvm problem I faced with my laptop and the several tips I got :)

posted on qubes-user and lvm-devs and redhat mailing lists:
https://groups.google.com/forum/#!topic/qubes-users/TiN6kRtTPkk

As I see it, realistically I can have some free time after the 1.st of April

to tackle the issue full time and focus and recover the data and push to
the repo.

(I know, always f*** push, I know ...:/)

I would kindly ask for some patience :)

It would be a waste of time to make the work double,

instead of upgrading it afterwards :)

Stay safe and healthy everyone!

Warm greets,

m

[Tobias Killer]

Am 29.01.20 um 02:47 schrieb Marek Marczykowski:

I have written a simple example that "mimicks" the architecture data
multilingually.

You may run my good old prototype

https://github.com/tokideveloper/langswitch-prototype

via

$ bundle exec jekyll serve

and call the page

http://127.0.0.1:4000/

Now, you will see a bold "architecture" string in the current lang just
below the bottom lang switch. Note that I served the appropriate data
files only for English and German for this example.

This is where that "architecture" string is being produced:

https://github.com/tokideveloper/langswitch-prototype/blob/master/_layouts/doc.html#L125-L140

So, it is a little bit cumbersome (okay, the filters can be written in a
single line) but possible to have translated data in a separate
folder/repo, in this case:

https://github.com/tokideveloper/langswitch-prototype/tree/master/_data-translated

The main difference from the normal `_data` folder is that my
implementation uses collections. A collection is a set of files but the
file hierarchy is being ignored. So, we cannot point to a specific
path/to/file but have to filter out which file is the correct one only
because of its content (the YAML front matter, respectively). That is
why I have `type` and `lang` in both translated and non-translated data
files, besides `data` containing the actual data.

Another thing is that Jekyll's config file must contain the name of the
collection (`data-translated`) with `output` set to `false` since there
is no Markdown file to render. This leads to the conclusion that we
should have another repo named like `qubes-data-translated` or
`qubes-website-data-translated` since the `qubes-translated` repo must
have set `output` to `true`.

By the way, the HCL, which already exists, is a good example for such a
collection containing data.

So, what do you think?

@Maya: Do you already have an appropriate solution for that?

Stay healthy!

Best regards
Tobias Killer

[Tobias Killer]

Am 19.02.20 um 12:00 schrieb Tobias Killer:

Hello,

is there someone who can and likes to answer?

Do I have to do something to make the script (for merging YAML front
matters) acceptable?

Best regards,
Tobias Killer