qubes-doc & rtd

81 views
Skip to first unread message

mm

unread,
Dec 21, 2021, 2:57:32 PM12/21/21
to qubes-devel
Hello all,

in September there was a thread about migrating the documentation to another platform,
and the silent winner was RTD.
Let me say up front: For me localization, release specific docs, automation & easy and transparent maintenance of the documentation
are one of the most important criteria for this.

I took a closer look at RTD wtr to localisation and this is what I found out:


TL;DR:

####

Here is a preliminary test version of the qubes documentation as in _doc, uploaded to the RTD, using sphinx with markdown support and transifex fake translations:

https://qubes-doctestrun.readthedocs.io/en/latest/README.html

& https://qubesosgithubiotestrun.readthedocs.io/en/latest/doc.html


And a sample wannabe german translation at:

https://qubes-doctestrun.readthedocs.io/de/latest/README.html

& https://qubes-doctestrun.readthedocs.io/de/latest/doc.html

In the bottom menu you can choose a language, specific version (right now only different html themes, pdf, epub)
As there are issues, as explained below, one can for now browse with dedication via specific links such as:

https://qubesosgithubiotestrun.readthedocs.io/en/latest/developer/building/qubes-builder.html

& https://qubes-doctestrun.readthedocs.io/de/latest/developer/building/qubes-builder.html
Of course the test repos from are behind the current state for ages.


The longer version:

#############


There are two generators that can are used by RTD -- sphinx (reST) or mkdocs (md).
MkDocs is perfect for Markdown, Sphinx - for restructured text, but can parse markdown, via an extension
but there are some problems as listed below, which need to be addressed.

## MkDocs

MkDocs writes with markdown and is more or less straightforward with qubes current state of the documentation/website.

The problem is that MkDocs does not support localization out of the box,
but through third party plugins such as [1] or [2],
which more or less defaults to the current way of localizing the website and thus also _doc. There are also not many third party plugins for MkDocs.
And most important - localization (in RTD) is only supported for sphinx documentation, which settles it if the docs should be moved to RTD at least for now:

"This feature only applies to Sphinx documentation. We are working to bring it to our other documentation backends." See [3]


## SPHINX:

The default generator used by RTD, which supports all the features by RTD, including localization.

### MYST

To use sphinx with markdown I tried MyST

(see [4], [5], [6]. MyST specific configuration can be seen at [7])

### HTML themes

There are some built-in themes ([8]), configuration options ([9] or [10])
Other themes can be found at [11].

Or one could overwrite and customize a theme with [12] or

by following some tutorials ( which should the way to go for Qubes I think )

#### BUILD

is as simple as `sphinx-build -b html _doc/ _build/html`

for latex:

```
sphinx-build -b latex _doc/ _build/latex
cd _build/latex
make # needs latexmk
```

### LOCALIZATION


Localization for sphinx is done via gettext as simple as:


```
sphinx-build -b gettext . _build/gettext
```

for the configuration the following options are good as per documentation:

```
gettext_uuid=True
gettext_compact=False
```





#### TRANSIFEX

There is a neat way to handle transifex and sphinx.


Due to a problem, which I am currently trying to solve, the sample translations are due to a transifex trick, downloaded via tx in the root repo and manually added to the qubestranslation repo,

which is not in an optimal state right now.


There is the possibility to have all the translations in one repo as in [13] and this is also the goal.

Problem: The problem is that for common users in RTD once a person has imported a repository there is no way to import it again, and for different translations,

RTD needs different projects, that exist in RTD and are marked as translations for the main one ... As till now I do not know how to tackle this issue.


## PANDOC
 
And of course there is the option to convert all the markdown documentation files
to restructured text (w pandoc f.x.),
which is native for Sphinx, and thus, many of the current rendering issues
should be non-existent.

One could convert the md to rst as in

```
pandoc doc.md --from markdown --to rst -s -o doc.rst
```

one can view it with restview (pip install restview)


## RTD problems/TODOs


1. Rendering

There are warnings that have to be adressed such as:

```
/home/user/qubes/qubesos.github.io.testrun/_doc/docs/source/user/advanced-topics/awesomewm.md:22: WARNING: Pygments lexer name 'shell_session' is not known
/home/user/qubes/qubesos.github.io.testrun/_doc/docs/source/project-security/verifying-signatures.md:150: WARNING: 'myst' reference target not found: /support/
/home/user/qubes/qubesos.github.io.testrun/_doc/docs/source/user/troubleshooting/vpn-troubleshooting.md: WARNING: document isn't included in any toctree
...
```

2. Many broken links
3. PDF  - the documentation is not being rendered correctly
4. EPUB - the documentation is not being rendered correctly
5. Sphinx uses the Jinja templating engine for its HTML templates. Meaning for files like

https://github.com/maiska/qubes-doc.testrun/blob/49253a1f5a24a9a9f5c1c42e2395ca494a1e4b4b/developer/general/style-guide.md

this is the result: https://qubes-doctestrun.readthedocs.io/de/latest/developer/general/style-guide.html

I have not found a converter from liquid to jinja, so there should be another solution

6. External files: can be just listed in the doc.md and the markdown files can be deleted. But the redirects are to be handled
7. Warning: RTD is somewhat buggy if one pushes immediately one after another several branches for one project that are activated
8. Find out how to handle multiple translations with one repo in RTD
9. Uploading to transifex problem (wip)
10. Analytics (JS in general in RTD): script as in [14]  with a RTD section regarding this [15]


What are your thoughts on this?


All the best,

m

Andrew David Wong

unread,
Dec 22, 2021, 5:10:30 AM12/22/21
to mm, qubes-devel
On 12/21/21 11:57 AM, mm wrote:
> Hello all,
>
> in September there was a thread about migrating the documentation to
> another platform,
> and the silent winner was RTD.
> Let me say up front: For me localization, release specific docs,
> automation & easy and transparent maintenance of the documentation
> are one of the most important criteria for this.
> I took a closer look at RTD wtr to localisation and this is what I found
> out:
>
> [...]
>
> What are your thoughts on this?
>

Thank you very much for investigating this so deeply and sharing your
findings! To be honest, a lot of this is over my head, but I'm very much
open to hearing everyone's opinions about the best way to proceed. I'll
support whatever path forward is best for the project.

On a personal note, I've been the designated "website and documentation
maintainer" for a while now, but, for me, the key word is "maintainer."
I'm not a professional developer, and my self-taught skills are very
limited. I'm better suited to editing content and administering an
established system rather than attempting to build and debug new
technical systems.

As we've grown as a project, I've been feeling the pressure as our
website and documentation needs have outstripped my abilities. One of
the main reasons I wanted to look into something like RTD was to find a
more "turnkey" solution to alleviate this pressure [*]. We need new
features that I simply don't have the ability to build, which is a major
limitation of our current custom system.

I would be happy to step aside or share responsibilities with anyone who
has the needed skills and is willing to step in, whether to take the
reins or just to fill certain gaps. After all, as the project gets
larger, it would make sense that we have more specialized roles.


[*] However, it was just an idea that's still being hashed out, not a
plan set in stone! This thread is a welcome and appreciated step forward
in that process.

--
Andrew David Wong
Community Manager
The Qubes OS Project
https://www.qubes-os.org

Manuel Amador (Rudd-O)

unread,
Dec 25, 2021, 3:16:39 PM12/25/21
to Andrew David Wong, mm, qubes-devel


Honestly if I were directing the project I would set up a Plone instance, and use its excellent i18n to write docs. Additionally, if wanted, I would set up two way sync between a github repo and the site. Ask me for more details if interested.

mm

unread,
Dec 25, 2021, 4:37:09 PM12/25/21
to qubes...@googlegroups.com

On 12/25/21 21:16, Manuel Amador (Rudd-O) wrote:
>
> Honestly if I were directing the project I would set up a Plone instance, and use its excellent i18n to write docs. Additionally, if wanted, I would set up two way sync between a github repo and the site. Ask me for more details if interested.


Thank you very much for taking the time to reply and also for your
suggestion.

In my personal opinion Plone as a CMS is much too heavy for the purposes
of the user documentation of Qubes.

I have experience with it, though a decade ago probably, and it is not
as intuitively as one might think.

Perhaps things changed and is a straightforward easy to use idk.

If you would like and have the time, could you outline how would you
proceed with markdown - Plone integration (or if it works w/o issues),

automation of translation/localization wrt to transifex and release
specific docs,

so one could weight the pro and contra of the different approaches till now.

All the best,

m



Manuel Amador (Rudd-O)

unread,
Dec 26, 2021, 7:47:01 AM12/26/21
to qubes...@googlegroups.com, mm


On December 25, 2021 10:37:04 PM GMT+01:00, mm <mai...@maiski.net> wrote:
>
>On 12/25/21 21:16, Manuel Amador (Rudd-O) wrote:
>>
>> Honestly if I were directing the project I would set up a Plone instance, and use its excellent i18n to write docs. Additionally, if wanted, I would set up two way sync between a github repo and the site. Ask me for more details if interested.
>
>
>Thank you very much for taking the time to reply and also for your
>suggestion.
>
>In my personal opinion Plone as a CMS is much too heavy for the purposes
>of the user documentation of Qubes.
>
>I have experience with it, though a decade ago probably, and it is not
>as intuitively as one might think.
>
>Perhaps things changed and is a straightforward easy to use idk.

It's very straightforward to use now.

>
>If you would like and have the time, could you outline how would you
>proceed with markdown - Plone integration (or if it works w/o issues),

Plone supports Markdown out of the box.

>
>automation of translation/localization wrt to transifex and release
>specific docs,

This would require a bit of code that extracts the strings to translate (using the exportimport framework) from the original language folder, and deploys the translated content (using the exportimport framework) to the respective language folders (linking the translation to the original).

E.g. article /en/howtos/vpn would get exported, posted to Transifex (possibly through a Git repo), and then once the German translation is available, /de/howtos/vpn could be posted, linking it to /en/howtos/vpn so the UI can show all translations for the same content item.

The items could then be published by the same pipeline doing the exportimport, or left to be reviewed prior to publishing by people.

mm

unread,
Dec 26, 2021, 11:33:20 AM12/26/21
to qubes...@googlegroups.com
Hi again

On 12/26/21 13:46, Manuel Amador (Rudd-O) wrote:
> aps things changed and is a straightforward easy to use idk.
> It's very straightforward to use now.
ok, nice to hear that
> Plone supports Markdown out of the box.
good
>
>> automation of translation/localization wrt to transifex and release
>> specific docs,
> This would require a bit of code that extracts the strings to translate (using the exportimport framework) from the original language folder, and deploys the translated content (using the exportimport framework) to the respective language folders (linking the translation to the original).
>
> E.g. article /en/howtos/vpn would get exported, posted to Transifex (possibly through a Git repo), and then once the German translation is available, /de/howtos/vpn could be posted, linking it to /en/howtos/vpn so the UI can show all translations for the same content item.
>
> The items could then be published by the same pipeline doing the exportimport, or left to be reviewed prior to publishing by people.

I think you are referring to smth like
https://docs.plone.org/develop/plone/content/importexport.html or
https://docs.plone.org/develop/plone/i18n/internationalisation.html

I see.

Maintenance as you say should not be a problem, so let's stay at that
for now.

I can imagine that release specific docs could be branch based and
configured.

Still my opinion is that it is not light weight enough for the purpose
of the documentation,

would require another set of tools/knowledge/migration from an already
established static html generator to a non static CMS,

could and would introduce security concerns

(see
https://www.cvedetails.com/vulnerability-list/vendor_id-4313/Plone.html
vs
https://www.cvedetails.com/product/56912/Readthedocs-Read-The-Docs.html?vendor_id=20051)


Perhaps others would like to share further views on the overall topic?


All the best,

m


p.s.

imho RDT would still be the way to go



Marek Marczykowski-Górecki

unread,
Sep 2, 2022, 11:31:30 AM9/2/22
to mm, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Let me try to resurrect this thread :)

First of all, thanks again for all this analysis. I think indeed RTD is
the way to go. We'll have occasion to talk about details on the
upcoming Qubes summit. Personally, I wouldn't exclude an option to
convert documentation to ReST format, especially since we get several
more features then (better internal links tracking, PDF/ePUB generation,
built-in macros for notices, warnings etc, and several more). But of
course this needs to be weighted against other factors like complexity
of the conversion itself, or ease of use for maintainers and
contributors.

See you soon!
> --
> You received this message because you are subscribed to the Google Groups "qubes-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/f24cb2b0-9465-ef3c-c71f-afb71248ab9f%40maiski.net.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmMSIcwACgkQ24/THMrX
1yw03Qf/c63FuOGz6zsBwRWMLabRM3JxCA05ILFu76bg3/MI68DtbiRcYEBnCekR
CbvNIxTLq077X4N58HiIlsunhrln+n2eLgsD6H/ljp3ZGmbNdzfZhz2WdbWMaKjK
4dzdxEb6pUfrDqXrNOrxDOxRHBJYLR6mjr2ih211iPVaJccgIny8l8zSkWpTmG6S
JmowbLeaxfNXsv00yODQ9Hv/7hTZj5xd0Kntq17FGrL9BWLFad0fVYxzZ93Dh6NI
g6KgSnwWAvLqUZjQ7BSrjhbXuKUOeIHJ+luA23aqKioWHNcA9SXvtz2bUZyYVNeM
Eve3568YYj12PcZIjbZH+3leDZ+EbA==
=kYM7
-----END PGP SIGNATURE-----

mm

unread,
Sep 4, 2022, 4:05:49 PM9/4/22
to Marek Marczykowski-Górecki, qubes-devel
Hello Marek,


thank you for resurrecting the thread. I wanted to write like a week ago,

but yet, it lasted longer.

I am glad to announce that the initial hurdle for Qubes OS docs
RST-conversion,
localization, hosting on RtD is behind us and you can see a preliminary
result on

https://current-qubes-docrtd.readthedocs.io/en/latest

This is the current status of the branch rtd-test-default-html-theme  in
the repo https://github.com/maiska/qubesos.github.io.rtd + submodules

To see the partially translated docs in german you can go
down to the left, click on the arrow right to 'v:latest' and choose the
language.

There are further tweeks to be done, such as epub and pdf, but the big
step is taken (including automated conversion via script and
postprocessing cross-referencing in sphinx, but still needing minor
manually fixes after that, documentation follows) and translation
workflow tested.

I will not bother you with details at that stage but with questions.

The outline of the questions is as follows:

1. Easily fixable dead end urls as found in the official docs
2. Non easily fixable links
3. Website content vs RtD documentation content
4. Hidden documentation files

1. These are the commits that show what can be fixed imho.

https://github.com/maiska/qubes-doc.rtd/commit/0b8529a026da0523355eec55c57c1da2aa110d07
https://github.com/maiska/qubes-doc.rtd/commit/2b05ac2d8418adbc1d4215d3d54195b3ca8e7f58
https://github.com/maiska/qubes-doc.rtd/commit/6772392f3e2e682d953d7e3b8d9f870867de202a
https://github.com/maiska/qubes-doc.rtd/commit/538bac0e0c368cd54e5acb0b5d1ffb77e06e5045
https://github.com/maiska/qubes-doc.rtd/commit/eb4d1dade1697d4b9e725db8871b1c54ec8c61d6
https://github.com/maiska/qubes-doc.rtd/commit/a8884c8e11f91458bb8ea30d6bbea49e25db48b2

Question: Should I just create Pull Requests for the proposed fixes

2. At least one broken for me unfixable link

in
https://github.com/QubesOS/qubes-doc/edit/master/developer/releases/2_0/release-notes.md
l 41

Empty Response or No Access-Control-Allow-Origin
https://wiki.qubes-os.org/query?status=accepted&status=assigned&status=new&status=reopened&type=defect&milestone=Release+2.1+(post+R2)&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&order=priority
http://localhost:5656/developer/releases/2_0/release-notes.html

Question: Could you fix that? :)

3. My suggestion is to keep the documentation index and some Markdown
files in the changed qubes os website repo,
the converted RST files in a new one with two submodules: attachment and
_translated

A new sample Qubes RST documentation repository (without any
translations yet) can be found here:

git clone -b qubes-rst-doc https://github.com/maiska/qubes-rst-doc.git

A new sample Qubes Website incorporating RST documenation can be found here:

git clone -b new-master https://github.com/maiska/qubesos.github.io.rtd.git

The following files will be retained at this stage, due to liquid
templating (needs some tweeking if
one wishes to work with sphinx templating) and also these are perhaps
also just part of the official website not the documenation as I see it
at least atm

- https://www.qubes-os.org/hcl/
- https://www.qubes-os.org/downloads/
- https://www.qubes-os.org/security/xsa/
- https://www.qubes-os.org/security/qsb/
- https://www.qubes-os.org/security/canary/
- https://www.qubes-os.org/security/pgp-keys/

Question: Are you OK with this? Or should I leave the question for the talk?

4. Hidden files not yet deleted or referenced in the index are to be
found here (not in the german version of the index rn):

https://current-qubes-docrtd.readthedocs.io/en/latest/#hidden-gems

Question: Should the index be populated with them?

Regarding the old translation markdown workflow - it is still there, and
can be brushed off the dust.

P.S. I just realized that my .vimrc config was broken. Please excuse the
tabs...!

Marek Marczykowski-Górecki

unread,
Sep 5, 2022, 7:55:47 AM9/5/22
to mm, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Sep 04, 2022 at 10:05:42PM +0200, mm wrote:
> Hello Marek,
>
>
> thank you for resurrecting the thread. I wanted to write like a week ago,
>
> but yet, it lasted longer.
>
> I am glad to announce that the initial hurdle for Qubes OS docs
> RST-conversion,
> localization, hosting on RtD is behind us and you can see a preliminary
> result on
>
> https://current-qubes-docrtd.readthedocs.io/en/latest
>
> This is the current status of the branch rtd-test-default-html-theme  in the
> repo https://github.com/maiska/qubesos.github.io.rtd + submodules

This looks very promising!

> To see the partially translated docs in german you can go
> down to the left, click on the arrow right to 'v:latest' and choose the
> language.
>
> There are further tweeks to be done, such as epub and pdf, but the big step
> is taken (including automated conversion via script and postprocessing
> cross-referencing in sphinx, but still needing minor manually fixes after
> that, documentation follows) and translation workflow tested.
>
> I will not bother you with details at that stage but with questions.
>
> The outline of the questions is as follows:
>
> 1. Easily fixable dead end urls as found in the official docs
> 2. Non easily fixable links
> 3. Website content vs RtD documentation content
> 4. Hidden documentation files
>
> 1. These are the commits that show what can be fixed imho.
>
> https://github.com/maiska/qubes-doc.rtd/commit/0b8529a026da0523355eec55c57c1da2aa110d07
> https://github.com/maiska/qubes-doc.rtd/commit/2b05ac2d8418adbc1d4215d3d54195b3ca8e7f58
> https://github.com/maiska/qubes-doc.rtd/commit/6772392f3e2e682d953d7e3b8d9f870867de202a
> https://github.com/maiska/qubes-doc.rtd/commit/538bac0e0c368cd54e5acb0b5d1ffb77e06e5045
> https://github.com/maiska/qubes-doc.rtd/commit/eb4d1dade1697d4b9e725db8871b1c54ec8c61d6
> https://github.com/maiska/qubes-doc.rtd/commit/a8884c8e11f91458bb8ea30d6bbea49e25db48b2
>
> Question: Should I just create Pull Requests for the proposed fixes

Yes, makes sense. I wonder why URL checker hasn't found them... Or maybe
it did but we ignored it?
Since that's about Qubes R2, I think the link can be simply dropped.
(otherwise it's probably possible to write similar github issues filter,
but I wouldn't bother)

> 3. My suggestion is to keep the documentation index and some Markdown files
> in the changed qubes os website repo,
> the converted RST files in a new one with two submodules: attachment and
> _translated

Yes, this sounds like a good plan.

> A new sample Qubes RST documentation repository (without any translations
> yet) can be found here:
>
> git clone -b qubes-rst-doc https://github.com/maiska/qubes-rst-doc.git
>
> A new sample Qubes Website incorporating RST documenation can be found here:
>
> git clone -b new-master https://github.com/maiska/qubesos.github.io.rtd.git

So, it references _doc submodule that basically is just about redirects
to the new doc. Makes sense to keep links working.

> The following files will be retained at this stage, due to liquid templating
> (needs some tweeking if
> one wishes to work with sphinx templating) and also these are perhaps also
> just part of the official website not the documenation as I see it at least
> atm
>
> - https://www.qubes-os.org/hcl/
> - https://www.qubes-os.org/downloads/
> - https://www.qubes-os.org/security/xsa/
> - https://www.qubes-os.org/security/qsb/
> - https://www.qubes-os.org/security/canary/
> - https://www.qubes-os.org/security/pgp-keys/

Sounds fine. IMO those should be moved to the main repo. In fact, actual
data for those pages is already in the main repo (_data/), just not the
.md file...

> Question: Are you OK with this? Or should I leave the question for the talk?
>
> 4. Hidden files not yet deleted or referenced in the index are to be found
> here (not in the german version of the index rn):
>
> https://current-qubes-docrtd.readthedocs.io/en/latest/#hidden-gems
>
> Question: Should the index be populated with them?

You mean pages that are only linked from some sub-section but not the
main index? Or not linked at all?

Some of those are normally linked from the website footer,
https://www.qubes-os.org/intro/ or similar.
But generally, IMO better have every doc page linked to the main index.


> Regarding the old translation markdown workflow - it is still there, and can
> be brushed off the dust.
>
> P.S. I just realized that my .vimrc config was broken. Please excuse the
> tabs...!

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmMV47oACgkQ24/THMrX
1yw1sgf/fW9F3s/Bp1/NmqH4lBH/9jy/p49/lYs8UTU9qWWw8sJY9w6v8rVZ/N3F
okWW30Pjw8j1DPx6zY9JWdvzYu9jZAABgHOLNrYY9u4O2WD9V0CFUbZNxezqDyGR
rNe8XGdWrWoPDylRW7EYw4BTC6m7RPHK7h3GuyoP0YyTyxDim0+9I4vzgq304uBB
NiUJLCdzRaW2x1S5ulJbgSd6iDYdiz2mGuvkaS/hbqv+OnaMkHLaOJmKaLShWY6g
Z5dU2tDprpx0ohkXIfd9hTvijrDrb981qDP3pjFY2bXoFFrTaHgUn6myXFCKl5tW
F728mHFxzq2LxH1XzCiFeneJ+Qg47Q==
=eLlL
-----END PGP SIGNATURE-----

Andrew David Wong

unread,
Sep 7, 2022, 6:34:58 PM9/7/22
to Marek Marczykowski-Górecki, mm, qubes-devel
Please do submit a PR.

> I wonder why URL checker hasn't found them... Or maybe it did but we ignored it?

No, we aren't ignoring any. I always immediately fix all the ones I see notifications about.

It looks like most of them are broken *external* links, whereas our URL checker only checks for broken *internal* links (which is partially why adding lots of external links increases the maintenance burden on the documentation over time).

Also, some of these are not really broken. For example, `/doc/disk-trim` is a working redirect. Try it: https://www.qubes-os.org/doc/disk-trim

Of course, replacing the redirect with the redirect's target is also fine.

>> 2. At least one broken for me unfixable link
>
>> in https://github.com/QubesOS/qubes-doc/edit/master/developer/releases/2_0/release-notes.md
>> l 41
>
>> Empty Response or No Access-Control-Allow-Origin https://wiki.qubes-os.org/query?status=accepted&status=assigned&status=new&status=reopened&type=defect&milestone=Release+2.1+(post+R2)&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&order=priority
>> http://localhost:5656/developer/releases/2_0/release-notes.html
>
>> Question: Could you fix that? :)
>
> Since that's about Qubes R2, I think the link can be simply dropped.
> (otherwise it's probably possible to write similar github issues filter,
> but I wouldn't bother)
>
>> 3. My suggestion is to keep the documentation index and some Markdown files
>> in the changed qubes os website repo,
>> the converted RST files in a new one with two submodules: attachment and
>> _translated
>
> Yes, this sounds like a good plan.
>

Sorry, I don't understand. Can anyone explain?

>> A new sample Qubes RST documentation repository (without any translations
>> yet) can be found here:
>
>> git clone -b qubes-rst-doc https://github.com/maiska/qubes-rst-doc.git
>

This appears to be empty for me.

>> A new sample Qubes Website incorporating RST documenation can be found here:
>
>> git clone -b new-master https://github.com/maiska/qubesos.github.io.rtd.git
>

This appears to just be a clone of the existing website, with the _doc submodule containing Markdown files.

> So, it references _doc submodule that basically is just about redirects
> to the new doc. Makes sense to keep links working.
>

Wouldn't converting all Markdown files to RST result in still using the _doc submodule, just with all the files converted? Maybe I'm not understanding something here.

>> The following files will be retained at this stage, due to liquid templating
>> (needs some tweeking if
>> one wishes to work with sphinx templating) and also these are perhaps also
>> just part of the official website not the documenation as I see it at least
>> atm
>
>> - https://www.qubes-os.org/hcl/
>> - https://www.qubes-os.org/downloads/
>> - https://www.qubes-os.org/security/xsa/
>> - https://www.qubes-os.org/security/qsb/
>> - https://www.qubes-os.org/security/canary/
>> - https://www.qubes-os.org/security/pgp-keys/
>
> Sounds fine. IMO those should be moved to the main repo. In fact, actual
> data for those pages is already in the main repo (_data/), just not the
> .md file...
>

I think this goes back to the previous discussions we've had over the years about how to organize website vs. doc content, and we landed on organizing it this way for various reasons. I think one of those reasons was to have everything that should be included in offline documentation in qubes-doc. (So, even if the data is in `_data/`, the offline doc generation tool might decide whether to include it based on whether the page referencing that data is in qubes-doc or not.)

Also, it might be a bit weird to have a page (/security/) in qubes-doc, while its "subpages" are in the more general parent repo "above" it. So, organizing based on semantic content vs. source file/data attributes will get you different results.

I'm sure there are many other considerations buried in our archives that I'm forgetting now.

>> Question: Are you OK with this? Or should I leave the question for the talk?
>
>> 4. Hidden files not yet deleted or referenced in the index are to be found
>> here (not in the german version of the index rn):
>
>> https://current-qubes-docrtd.readthedocs.io/en/latest/#hidden-gems
>
>> Question: Should the index be populated with them?
>
> You mean pages that are only linked from some sub-section but not the
> main index? Or not linked at all?
>
> Some of those are normally linked from the website footer,
> https://www.qubes-os.org/intro/ or similar.
> But generally, IMO better have every doc page linked to the main index.
>

Well, most of these are intentionally not linked from the main index for a reason.

Qubes builder -- probably a subpage linked from a main page
Join -- intentionally hidden when active recruiting stopped (to be unhidden if/when it resumes)
Admin API Table -- it's an "embed" on the Admin API page (just a hack to allow opening that huge table by itself in the "fullscreen" layout)
Disposable VM Implementation -- not sure
Qrexec2 -- not sure
Qfileexchgd -- not sure
Code of Conduct -- linked from footer, as Marek said; would be okay to link from index
Intro -- linked from "nav bar" at the very top of the website
Privacy -- intentional footer link, like every other website
Screenshots -- linked from Intro; would be okay to link from index
Statistics -- linked from News page; would be okay to link from index
Video Tours -- linked from Intro; would be okay to link from index
QSB Checklist -- checklist for internal use; probably not of interest to users; would just be clutter in the index
Canary Checklist -- same
Download mirrors -- table embedded on Downloads page
Custom install -- linked from Installation Guide; would be okay to link from index
Install security -- linked from Installation Guide; would be okay to link from index
How to use the hcl -- linked from HCL page; would be okay to link from index
How to reinstall a template -- subpage linked from Templates page
Debian template -- this page is already linked from the index; not sure why it was miscategorized as a hidden gem
Debian upgrade -- in-plage upgrade guide subpage linked from Debian page
Fedora upgrade -- in-plage upgrade guide subpage linked from Fedora page
Fedora -- this page is already linked from the index; not sure why it was miscategorized as a hidden gem
Update Debian Whonix -- looks like this got missed in the contributor overhaul of troubleshooting documentation that occurred a while back; it might just be outdated now

IMO:

- If a page exists purely as a technical workaround (hack), then it makes little sense to list it in the index. Pages that exist purely to serve as embeds in other pages probably fall under this. The content is already presented somewhere in a rational way, and this just creates a confusing duplicate pointer to part of it without its intended context.

- Internal docs like maintainer checklists would probably just be confusing/useless for readers, but I suppose it's okay to list them if you want. (If we don't really use these, then we can just remove them.)

- In a few places, a subpage only exists because people thought that including its content "in-line" in its parent page would make the resulting page "too long." I guess it's okay to list these, but it might help to at least update the titles to make the parent/child relationship clear.

mm

unread,
Sep 7, 2022, 9:30:53 PM9/7/22
to Andrew David Wong, Marek Marczykowski-Górecki, qubes-devel
Hello Marek, hello Andrew,


thank you for your input!! Sorry for answering this late!

As a disclaimer: the talk in an ideal world will be packed and last 20
min, so that there would be 10 min for discussion.
OK

>> I wonder why URL checker hasn't found them... Or maybe it did but we ignored it?
> No, we aren't ignoring any. I always immediately fix all the ones I see notifications about.
>
> It looks like most of them are broken *external* links, whereas our URL checker only checks for broken *internal* links (which is partially why adding lots of external links increases the maintenance burden on the documentation over time).
>
> Also, some of these are not really broken. For example, `/doc/disk-trim` is a working redirect. Try it: https://www.qubes-os.org/doc/disk-trim
>
> Of course, replacing the redirect with the redirect's target is also fine.

I think all of my fixes were external ones. Only one is internal, cause
it interferes with my scripting and post-fixing cross-reference in sphinx.

>
>>> 2. At least one broken for me unfixable link
>>> in https://github.com/QubesOS/qubes-doc/edit/master/developer/releases/2_0/release-notes.md
>>> l 41
>>> Empty Response or No Access-Control-Allow-Origin https://wiki.qubes-os.org/query?status=accepted&status=assigned&status=new&status=reopened&type=defect&milestone=Release+2.1+(post+R2)&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&order=priority
>>> http://localhost:5656/developer/releases/2_0/release-notes.html
>>> Question: Could you fix that? :)
>> Since that's about Qubes R2, I think the link can be simply dropped.
>> (otherwise it's probably possible to write similar github issues filter,
>> but I wouldn't bother)


OK, I will just submit a PQ without this link.

>>> 3. My suggestion is to keep the documentation index and some Markdown files
>>> in the changed qubes os website repo,
>>> the converted RST files in a new one with two submodules: attachment and
>>> _translated
>> Yes, this sounds like a good plan.
>>
> Sorry, I don't understand. Can anyone explain?
>
>>> A new sample Qubes RST documentation repository (without any translations
>>> yet) can be found here:
>>> git clone -b qubes-rst-doc https://github.com/maiska/qubes-rst-doc.git
> This appears to be empty for me.

Look here: https://github.com/maiska/qubes-rst-doc/tree/qubes-rst-doc

Sorry, I did not posted the url but just the git clone

>
>>> A new sample Qubes Website incorporating RST documenation can be found here:
>>> git clone -b new-master https://github.com/maiska/qubesos.github.io.rtd.git

Here again my mistake. The URL:
https://github.com/maiska/qubesos.github.io.rtd/tree/new-master

and the clone should go recursive if you want to try it out locally

git clone -b new-master --recursive https://github.com/maiska/qubesos.github.io.rtd.git

> This appears to just be a clone of the existing website, with the _doc submodule containing Markdown files.
>
>> So, it references _doc submodule that basically is just about redirects
>> to the new doc. Makes sense to keep links working.
>>
Yes, it contains actually everything from the _doc directory in place,
only without contents and with a redirect to the testing RTD version
from above.

Only some files are retained such as downloads, hcl etc (from below)

Besides the links existing in pages or posts, that link to /doc/foo
content that must be fixed en large if the _doc directory is unpopulated,

other people's articles linking to the official Qubes OS documentation
would be broken, since I do not think that there is a proxy

to handle such requests and redirects? Or there is a simpler way?

> Wouldn't converting all Markdown files to RST result in still using the _doc submodule, just with all the files converted? Maybe I'm not understanding something here.

This could be a possibility yes, perhaps with a sphinx plugin for
redirects. It exists, haven't tried it yet.

Meaning convert qubes-doc, do not create a new doc-rst repo, move
Markdown files like hcl, download etc to the pages submodule due to the
constraints I mentioned below,

as a temp stage or not.

>
>>> The following files will be retained at this stage, due to liquid templating
>>> (needs some tweeking if
>>> one wishes to work with sphinx templating) and also these are perhaps also
>>> just part of the official website not the documenation as I see it at least
>>> atm
>>> - https://www.qubes-os.org/hcl/
>>> - https://www.qubes-os.org/downloads/
>>> - https://www.qubes-os.org/security/xsa/
>>> - https://www.qubes-os.org/security/qsb/
>>> - https://www.qubes-os.org/security/canary/
>>> - https://www.qubes-os.org/security/pgp-keys/
>> Sounds fine. IMO those should be moved to the main repo. In fact, actual
>> data for those pages is already in the main repo (_data/), just not the
>> .md file...
>>
> I think this goes back to the previous discussions we've had over the years about how to organize website vs. doc content, and we landed on organizing it this way for various reasons. I think one of those reasons was to have everything that should be included in offline documentation in qubes-doc. (So, even if the data is in `_data/`, the offline doc generation tool might decide whether to include it based on whether the page referencing that data is in qubes-doc or not.)
>
> Also, it might be a bit weird to have a page (/security/) in qubes-doc, while its "subpages" are in the more general parent repo "above" it. So, organizing based on semantic content vs. source file/data attributes will get you different results.
>
> I'm sure there are many other considerations buried in our archives that I'm forgetting now.

From my part what I see is more work. It dragged itself through the
years enough :) (Of course that is also on me)

As soon as we get the localization workflow going and RTD is the way to
go as discussed previously and is stable,

one can still play with sphinx templating and incorporate the above
files in the RST documentation. I already tried it as a simple example
it functions, but is frustrating, and rn I do not have the nerve to sit
through this.

Perhaps some volunteers will come forward at the summit :)

Why I see them as a part of the website is the following: downloads,
xsa, qsb, canary and pgp-keys are somewhat essential to security of the
project, not truly documentation as I see it atm

HCL is documentation yes, there I just make my life easier
Yep, some of the gems are referenced from the footer for example.

Why I brought them to light in the index for discussion is the weird
feeling getting to qubes os documentation on RTD and not having all the
files at one's click.

As part of the official Qubes OS website it made sense and I did not
really notice.

RTD is different story.

I think if RTD should host them, the reader should be able to find them
from the index somehow, or a subindex whatever that may mean

And it is just a dump, it should be reorganized of course :)

mm

unread,
Sep 7, 2022, 9:45:58 PM9/7/22
to Andrew David Wong, Marek Marczykowski-Górecki, qubes-devel
Hello again,

here in a clear manner some suggestions.

What do you think?

>>
>> Qubes builder -- probably a subpage linked from a main page
is actually Qubes builder details, could be hosted in the Building section?

>> Join -- intentionally hidden when active recruiting stopped (to be
>> unhidden if/when it resumes)

Should it than be just part of the Qubes OS Website and not the docs?


>> Admin API Table -- it's an "embed" on the Admin API page (just a hack
>> to allow opening that huge table by itself in the "fullscreen" layout)

Could go below Admin API in the index?


>> Disposable VM Implementation -- not sure
Somewhere in developer's sections?
>> Qrexec2 -- not sure
Somewhere in developer's sections?
>> Qfileexchgd -- not sure
Somewhere in developer's sections?
>> Code of Conduct -- linked from footer, as Marek said; would be okay
>> to link from index
Perhaps in Introduction section ?
>> Intro -- linked from "nav bar" at the very top of the website
Perhaps in Introduction section ?
>> Privacy -- intentional footer link, like every other website
Perhaps in Introduction section ? for now after that somehow in the footer
>> Screenshots -- linked from Intro; would be okay to link from index
Perhaps in Introduction section ?
>> Statistics -- linked from News page; would be okay to link from index
Perhaps in Introduction section ?
>> Video Tours -- linked from Intro; would be okay to link from index
Perhaps in Introduction section ?
>> QSB Checklist -- checklist for internal use; probably not of interest
>> to users; would just be clutter in the index
Go to the official Qubes OS Website?
>> Canary Checklist -- same
Go to the official Qubes OS Website?
>> Download mirrors -- table embedded on Downloads page
hm ok obsolete
>> Custom install -- linked from Installation Guide; would be okay to
>> link from index
in Downloading, Installing, and Upgrading Qubes section?
>> Install security -- linked from Installation Guide; would be okay to
>> link from index
in Downloading, Installing, and Upgrading Qubes section?
>> How to use the hcl -- linked from HCL page; would be okay to link
>> from index
Choosing your hardware section?
>> How to reinstall a template -- subpage linked from Templates page
Templates section?
>> Debian template -- this page is already linked from the index; not
>> sure why it was miscategorized as a hidden gem
ja, idk, mistake
>> Debian upgrade -- in-plage upgrade guide subpage linked from Debian page
Templates section?
>> Fedora upgrade -- in-plage upgrade guide subpage linked from Fedora page
Templates section?
>> Fedora -- this page is already linked from the index; not sure why it
>> was miscategorized as a hidden gem
ja, idk, mistake
>> Update Debian Whonix -- looks like this got missed in the contributor
>> overhaul of troubleshooting documentation that occurred a while back;
>> it might just be outdated now

Templates section?



Andrew David Wong

unread,
Sep 8, 2022, 9:00:22 PM9/8/22
to mm, Marek Marczykowski-Górecki, qubes-devel
On 9/7/22 6:30 PM, mm wrote:
> [...]
>>>> https://github.com/maiska/qubes-doc.rtd/commit/0b8529a026da0523355eec55c57c1da2aa110d07

By the way, with this one, the link is not actually broken. It still works, because your browser adds the missing forward slash automatically. Nonetheless, I've committed a fix to make it technically syntactically correct.

> [...]
> other people's articles linking to the official Qubes OS documentation would be broken, since I do not think that there is a proxy
>
> to handle such requests and redirects? Or there is a simpler way?
>

I don't understand why links would get broken. Let's use a concrete example to help me understand. Take this page, for example:

https://www.qubes-os.org/doc/how-to-update/

After the migration to RtD, wouldn't this still be the same URL for that page? Why would it be any different?

If you're saying it has to be a different URL for some reason, what would the new URL look like?

>> Wouldn't converting all Markdown files to RST result in still using the _doc submodule, just with all the files converted? Maybe I'm not understanding something here.
>
> This could be a possibility yes, perhaps with a sphinx plugin for redirects. It exists, haven't tried it yet.
>

Why only "a possibility"? I figured that would be the obvious thing to do, but clearly I am missing something. If there is a good reason to do otherwise, then that's totally fine! It's just not clear to me what that reason might be.

> [...]
>>>> The following files will be retained at this stage, due to liquid templating
>>>> (needs some tweeking if
>>>> one wishes to work with sphinx templating) and also these are perhaps also
>>>> just part of the official website not the documenation as I see it at least
>>>> atm
>>>> - https://www.qubes-os.org/hcl/
>>>> - https://www.qubes-os.org/downloads/
>>>> - https://www.qubes-os.org/security/xsa/
>>>> - https://www.qubes-os.org/security/qsb/
>>>> - https://www.qubes-os.org/security/canary/
>>>> - https://www.qubes-os.org/security/pgp-keys/
>>> Sounds fine. IMO those should be moved to the main repo. In fact, actual
>>> data for those pages is already in the main repo (_data/), just not the
>>> .md file...
>>>
>> I think this goes back to the previous discussions we've had over the years about how to organize website vs. doc content, and we landed on organizing it this way for various reasons. I think one of those reasons was to have everything that should be included in offline documentation in qubes-doc. (So, even if the data is in `_data/`, the offline doc generation tool might decide whether to include it based on whether the page referencing that data is in qubes-doc or not.)
>>
>> Also, it might be a bit weird to have a page (/security/) in qubes-doc, while its "subpages" are in the more general parent repo "above" it. So, organizing based on semantic content vs. source file/data attributes will get you different results.
>>
>> I'm sure there are many other considerations buried in our archives that I'm forgetting now.
>
> From my part what I see is more work. It dragged itself through the years enough :) (Of course that is also on me)
>
> As soon as we get the localization workflow going and RTD is the way to go as discussed previously and is stable,
>
> one can still play with sphinx templating and incorporate the above files in the RST documentation. I already tried it as a simple example it functions, but is frustrating, and rn I do not have the nerve to sit through this.
>
> Perhaps some volunteers will come forward at the summit :)
>

Ok, if it's too much trouble to get the tech working correctly, then that's probably a good enough reason to just move these files. I don't mind.

> Why I see them as a part of the website is the following: downloads, xsa, qsb, canary and pgp-keys are somewhat essential to security of the project, not truly documentation as I see it atm
>

Well, I don't know if that holds for all of these, because at least some parts of those pages are clearly documentation explaining what XSAs are, what QSBs are, what canaries are, how we handle them, and so on.

But the other technical reason you gave enough is a separate and probably good enough on its own.

Ok, I've just moved these pages to the main repo and deleted the ones we don't need.

> HCL is documentation yes, there I just make my life easier
>

Yes, making life easier is probably more important at this stage. :)

(Btw, I'm assuming you meant *not* documentation, since above you listed the HCL page as one that you said *should* be moved to the main repo, which makes sense because it is indeed "empty" with all the data stored elsewhere. So, I've just moved it along with the others.)
Well, I think I explained above why that should not be a weird feeling. That's a bit like saying, "I bought a house, but it's a bit weird that I can't easily see and touch the insides of my own walls." o_O

> As part of the official Qubes OS website it made sense and I did not really notice.
>
> RTD is different story.
>
> I think if RTD should host them, the reader should be able to find them from the index somehow, or a subindex whatever that may mean
>

Well, ok, if that makes things easier. I've now made sure all these pages are either incorporated into their parent pages (i.e., eliminated) or listed from the index.

> And it is just a dump, it should be reorganized of course :)
>

Indeed. :)

For reference, here are the two PRs where I do everything I said I did in this email:

https://github.com/QubesOS/qubes-doc/pull/1267
https://github.com/QubesOS/qubesos.github.io/pull/204

Andrew David Wong

unread,
Sep 8, 2022, 9:00:39 PM9/8/22
to mm, Marek Marczykowski-Górecki, qubes-devel
Thanks for the suggestions. I believe I have taken care of all of these:

https://github.com/QubesOS/qubes-doc/pull/1267
https://github.com/QubesOS/qubesos.github.io/pull/204

Please let me know if I missed any.

Andrew David Wong

unread,
Sep 8, 2022, 9:14:29 PM9/8/22
to mm, Marek Marczykowski-Górecki, qubes-devel
On 9/8/22 6:00 PM, Andrew David Wong wrote:
> On 9/7/22 6:30 PM, mm wrote:
>> [...]
Ah, I already see one side effect of moving these pages to the main repo: They can no longer appear in the doc index, probably because the script that populates the doc index only checks for files in qubes-doc, which is probably one of the reasons we had them there in the first place....

Andrew David Wong

unread,
Sep 8, 2022, 9:16:07 PM9/8/22
to mm, Marek Marczykowski-Górecki, qubes-devel
But maybe that's okay, since once we migrate to RtD, we'll no longer be relying on custom scripts like this to populate the table of contents. :)

Andrew David Wong

unread,
Sep 8, 2022, 9:43:41 PM9/8/22
to mm, qubes-devel
On 9/7/22 6:30 PM, mm wrote:
> and the clone should go recursive if you want to try it out locally
>
> git clone -b new-master --recursive https://github.com/maiska/qubesos.github.io.rtd.git
>

So, how do I serve the site locally after doing this? Can you provide step-by-step instructions?

Andrew David Wong

unread,
Sep 8, 2022, 9:49:06 PM9/8/22
to mm, Marek Marczykowski-Górecki, qubes-devel
Well, I guess I better move back at least the Downloads and HCL pages, since having those in the index is pretty important. I guess the others can stay in the main repo, since they're linked from the Security Center page.

mm

unread,
Sep 9, 2022, 3:52:16 PM9/9/22
to Andrew David Wong, Marek Marczykowski-Górecki, qubes-devel

Hey Andrew,


thank you for your work!!

So one by one:

https://github.com/maiska/qubes-doc.rtd/commit/0b8529a026da0523355eec55c57c1da2aa110d07
By the way, with this one, the link is not actually broken. It still works, because your browser adds the missing forward slash automatically. Nonetheless, I've committed a fix to make it technically syntactically correct.

With the above commit there several links. I suppose you mean the internal one: it is so, yes, but for the convenience of my stupid script..., but let me explain:

Some people, when confronted with a problem, think
“I know, I'll use regular expressions.”
Now they have two problems.

Well.... :)


      
[...]
other people's articles linking to the official Qubes OS documentation would be broken, since I do not think that there is a proxy

to handle such requests and redirects? Or there is a simpler way?

I don't understand why links would get broken. Let's use a concrete example to help me understand. Take this page, for example:

https://www.qubes-os.org/doc/how-to-update/

After the migration to RtD, wouldn't this still be the same URL for that page? Why would it be any different?

If you're saying it has to be a different URL for some reason, what would the new URL look like?

The new url when generated will look like: foo.bar/user/how-to-guides/how-to-update.html from rst will look like :doc:` foo </user/how-to-guides/how-to-update>`

Domain dependent or whatever...


Wouldn't converting all Markdown files to RST result in still using the _doc submodule, just with all the files converted? Maybe I'm not understanding something here.
This could be a possibility yes, perhaps with a sphinx plugin for redirects. It exists, haven't tried it yet.

Why only "a possibility"? I figured that would be the obvious thing to do, but clearly I am missing something. If there is a good reason to do otherwise, then that's totally fine! It's just not clear to me what that reason might be.

Nope, you are abs right!

My motivation behind this solution (meaning leave the docs as they are but empty etc.) was take it one step at a time w/o necessarily drain time in fixing possible broken links. Till the documentation is on RTD

and afterwards the website can be dealt with and properly cleared of any redundant information (such as in _includes research.html, cause i blank out translated it

into flat markdown or rst),

also than use the sphinx redirect extension, these all are nice to have and can be done afterwards.

The possibility was not the right word in this context...

The HCL and Downloads thus can be addressed later :)

About the pull requests I will look into it latest tomorrow, render and say if smth is 'missing'


One thing though which I did not think about:

The privacy policy should be 2x - one for the Qubes OS website, and one for RTD. The one for RTD should be adjusted and perhaps refer to also their privacy policy:

https://docs.readthedocs.io/en/stable/privacy-policy.html


But maybe that's okay, since once we migrate to RtD, we'll no longer be relying on custom scripts like this to populate the table of contents.

Yes!! Hopefully there will be a lot less clutter :))


git clone -b new-master --recursive https://github.com/maiska/qubesos.github.io.rtd.git

So, how do I serve the site locally after doing this? Can you provide step-by-step instructions?


On a fedora VM:

sudo dnf install ruby ruby-devel openssl-devel redhat-rpm-config @development-tools;
gem install jekyll bundler;
find . -name gem; #(in my case it was /home/user/.local/share/gem/)
git clone -b new-master --recursive https://github.com/maiska/qubesos.github.io.rtd.git; cd qubesos.github.io.rtd/
bundle config set --local path '/home/user/.local/share/gem/'
bundle exec jekyll serve


All the best,

m

mm

unread,
Sep 9, 2022, 3:58:23 PM9/9/22
to qubes...@googlegroups.com
I forgot somehting

On a fedora VM:

sudo dnf install ruby ruby-devel openssl-devel redhat-rpm-config @development-tools;
gem install jekyll bundler;
find . -name gem; #(in my case it was /home/user/.local/share/gem/)
git clone -b new-master --recursive https://github.com/maiska/qubesos.github.io.rtd.git; cd qubesos.github.io.rtd/
bundle config set --local path '/home/user/.local/share/gem/'

bundle install;

(if there is some sort of error with webrick, just try: bundle add webrick; bundle exec jekyll serve)

bundle exec jekyll serve


All the best,

m

--
You received this message because you are subscribed to the Google Groups "qubes-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.

Tobias Killer

unread,
Sep 17, 2022, 2:43:27 PM9/17/22
to mm, qubes...@googlegroups.com
Hi, m,

thank you for all the work!

As far as I can remember, you talked with Marek about i18n and l10n on
the Qubes OS summit 2022. Could you (or Marek) please give us a summary
of the results of those talks?

Thank you!
Tobias

mm

unread,
Sep 25, 2022, 5:31:28 PM9/25/22
to Tobias Killer, qubes...@googlegroups.com
Hey Tobias,


I do not know unfortunately what you are referring to?

I mean the objective is to convert the markdown documentation to rst asap

Unfortunately it did not work out out of the box, due to docutils niceties.

background: I refuse to publish my regular-expressions-monster-convert
script.

I had to write a new one using docutils, as it should be.

Status as now is:

there are some minor fixes to be dealt with, such as container for
video-tours directive, registering of doc and ref roles

and a directive for strike through handling, so that the conversion can
be done with minor headaches by the qubes os team

(hint: I just pushed master branch on qubes-translation-utilz where the
current status can be feared :) )


Regarding translation as a whole: Michael suggested weblate, as tails
have experience with it and it seems very nice as OS alternative to
transifex.
Once there are further infos on that will summarize and write.

Do you mean that or smth else I forget to mention here?

All the best,
m

Marek Marczykowski-Górecki

unread,
Sep 25, 2022, 6:01:53 PM9/25/22
to mm, Tobias Killer, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi M,

In fact, I'm working on translation-utilz right now too. Marta used her
google-foo and found this gem:
https://github.com/SamWilsn/docutils-rst-writer

So, I replaced most of the qubesrstwriter2.py with just this thing and
it's almost flawless (especially tables, images etc work out of the
box).

I massaged the links fixup a bit, and now the only thing it complains
about is arch-spec-0.3.pdf, and a single "c" lexer issue. That's all.
Lists starting with non-1 number works too.

I'll push my state in few minutes into separate branch.

BR,
Marek
> --
> You received this message because you are subscribed to the Google Groups "qubes-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/75e76fab-da0d-058f-c8e9-4db4006c15ed%40maiski.net.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmMwz8cACgkQ24/THMrX
1yx55gf+L10gTE4rcdKYNE6QVAZQNfpZvuMiCooS19Ik2hy3cAqEnVtDoxSRnuOr
jr2NY8vroAEFEoD4U1QPRhOe1HeRL81+9zfTdIVZfKXBCxXSiZMlZuh0LHJYvh5X
wE5Skry+ObVjiqA8amb/AeI2Y2LXFAWlV6k0YuwOm5J8/TFu6pe5jKQdaapg5+qa
LLtC5tO+g2BcXEhkhFhhvBS/ABJD6vUjWjvdWdQtdnfhPnOgYTKEYXFiNFMPgs8Q
6ClhzILwVghws7frJJ0eOQ8JvbC+OTHAd3KJJZe5M1kAbu6rQIS7XCSOJ222WSSQ
NhggXUNW8D1sWkLOWeCgA2QaVReNpQ==
=npHQ
-----END PGP SIGNATURE-----

Andrew David Wong

unread,
Sep 26, 2022, 4:47:52 AM9/26/22
to mm, qubes-devel
On 9/9/22 12:52 PM, mm wrote:
> Hey Andrew,
>
>
> thank you for your work!!
>

Thank *you*! :)

> So one by one:
>
> https://github.com/maiska/qubes-doc.rtd/commit/0b8529a026da0523355eec55c57c1da2aa110d07
>
>> By the way, with this one, the link is not actually broken. It still works, because your browser adds the missing forward slash automatically. Nonetheless, I've committed a fix to make it technically syntactically correct.
>
> With the above commit there several links. I suppose you mean the internal one:

Ah yes, I meant the first one.

> it is so, yes, but for the convenience of my stupid script..., but let me explain:
>
> Some people, when confronted with a problem, think
> “I know, I'll use regular expressions.”
> Now they have two problems.
>
> Well.... :)
>

Oh, no worries. It ought to be syntactically correct anyway. :)

>>> [...]
>>> other people's articles linking to the official Qubes OS documentation would be broken, since I do not think that there is a proxy
>>>
>>> to handle such requests and redirects? Or there is a simpler way?
>>>
>> I don't understand why links would get broken. Let's use a concrete example to help me understand. Take this page, for example:
>>
>> https://www.qubes-os.org/doc/how-to-update/
>>
>> After the migration to RtD, wouldn't this still be the same URL for that page? Why would it be any different?
>>
>> If you're saying it has to be a different URL for some reason, what would the new URL look like?
>
> The new url when generated will look like: foo.bar/user/how-to-guides/how-to-update.html from rst will look like :doc:` foo </user/how-to-guides/how-to-update>`
>
> Domain dependent or whatever...
>

Oh, so it will mirror the directory structure. I seem to recall that I/we once tried this and decided against it for some reason, but I can no longer remember the details. I also can't recall whether we already discussed mirroring the directory structure for the purpose of this RST conversion and decided to go ahead with it. I suppose that if it's already implemented that way, we can proceed with it and deal with any problems that might arise. (Worst case scenario, we may have to go back and re-implement this part, but hopefully we can just work around it.)

>>
>>>> Wouldn't converting all Markdown files to RST result in still using the _doc submodule, just with all the files converted? Maybe I'm not understanding something here.
>>> This could be a possibility yes, perhaps with a sphinx plugin for redirects. It exists, haven't tried it yet.
>>>
>> Why only "a possibility"? I figured that would be the obvious thing to do, but clearly I am missing something. If there is a good reason to do otherwise, then that's totally fine! It's just not clear to me what that reason might be.
>
> Nope, you are abs right!
>
> My motivation behind this solution (meaning leave the docs as they are but empty etc.) was take it one step at a time w/o necessarily drain time in fixing possible broken links. Till the documentation is on RTD
>
> and afterwards the website can be dealt with and properly cleared of any redundant information (such as in _includes research.html, cause i blank out translated it
>
> into flat markdown or rst),
>
> also than use the sphinx redirect extension, these all are nice to have and can be done afterwards.
>
> The possibility was not the right word in this context...
>
> The HCL and Downloads thus can be addressed later :)
>

Ok, one step at a time makes sense.

> About the pull requests I will look into it latest tomorrow, render and say if smth is 'missing'
>
>
> One thing though which I did not think about:
>
> The privacy policy should be 2x - one for the Qubes OS website, and one for RTD. The one for RTD should be adjusted and perhaps refer to also their privacy policy:
>
> https://docs.readthedocs.io/en/stable/privacy-policy.html
>

I guess it's not obvious to me why that would be the case since we're self-hosting (aren't we?), but if you say we need it, then I don't see why we can't have two.

>
> But maybe that's okay, since once we migrate to RtD, we'll no longer be relying on custom scripts like this to populate the table of contents.
>
> Yes!! Hopefully there will be a lot less clutter :))
>
>
> git clone -b new-master --recursivehttps://github.com/maiska/qubesos.github.io.rtd.git
>
> So, how do I serve the site locally after doing this? Can you provide step-by-step instructions?
>
>
> On a fedora VM:
>
> sudo dnf install ruby ruby-devel openssl-devel redhat-rpm-config @development-tools;
> gem install jekyll bundler;
> find . -name gem; #(in my case it was /home/user/.local/share/gem/)
> git clone -b new-master --recursive https://github.com/maiska/qubesos.github.io.rtd.git; cd qubesos.github.io.rtd/
> bundle config set --local path '/home/user/.local/share/gem/'
> bundle exec jekyll serve
>

Ah, ok, it looks similar to the standard old method of serving a Jekyll site. Thanks, I'll give it a try when I find some time.

Any chance you've gotten this to work in a Debian minimal VM (plus whatever packages are required, of course)?

mm

unread,
Sep 26, 2022, 5:33:30 PM9/26/22
to Marek Marczykowski-Górecki, Tobias Killer, qubes...@googlegroups.com

Hi Marek,


On 9/26/22 00:01, Marek Marczykowski-Górecki wrote:
Hi M,

In fact, I'm working on translation-utilz right now too. Marta used her
google-foo and found this gem:
https://github.com/SamWilsn/docutils-rst-writer

this looks very nice!! :) Thanks Marta! :)

So, I replaced most of the qubesrstwriter2.py with just this thing and
it's almost flawless (especially tables, images etc work out of the
box).

OK, thanks for the PR, I'll merge into and get rid of the obsolete stuff from my branch this days.

I have some weird customized fixes all over, but this should work.

One question though:

have you had success with parsing admin-api.rst (as converted by pandoc from md?)

I get the error when parsing with docutils in order to post-process it for qubes:

docutils.utils.SystemMessage: /home/user/testingfilez/admin-api.rst:74: (SEVERE/4) Unexpected section title

and it cannot get over the table as it is...:/

all the best,

m

Marek Marczykowski-Górecki

unread,
Sep 26, 2022, 7:16:02 PM9/26/22
to mm, Tobias Killer, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Mon, Sep 26, 2022 at 11:33:22PM +0200, mm wrote:
> Hi Marek,
>
>
> On 9/26/22 00:01, Marek Marczykowski-Górecki wrote:
> > Hi M,
> >
> > In fact, I'm working on translation-utilz right now too. Marta used her
> > google-foo and found this gem:
> > https://github.com/SamWilsn/docutils-rst-writer
> >
> this looks very nice!! :) Thanks Marta! :)
>
> > So, I replaced most of the qubesrstwriter2.py with just this thing and
> > it's almost flawless (especially tables, images etc work out of the
> > box).
>
> OK, thanks for the PR, I'll merge into and get rid of the obsolete stuff
> from my branch this days.
>
> I have some weird customized fixes all over, but this should work.
>
> One question though:
>
> have you had success with parsing admin-api.rst (as converted by pandoc from
> md?)

I think so, yes.

>
> I get the error when parsing with docutils in order to post-process it for
> qubes:
>
> docutils.utils.SystemMessage: /home/user/testingfilez/admin-api.rst:74:
> (SEVERE/4) Unexpected section title

IIUC this was issue with overly wrapped table, so '_' or '-' in some cells
landed at unfortunate locations and were mis-parsed. In my PR I set line
length much longer to avoid this table wrapping and it seems to work.
The final sphinx-generated HTML looks a bit weird (no vertical cell
borders), but I think that's up to the theme, not RST.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmMyMqwACgkQ24/THMrX
1yxvdQf/edqGoIOTCO+vUWbGhcga0TtFZ3Wusa3nhk0fAzlZeWMcAtnu002UH86R
fUwIhlLnpxGBOsbqIOVZV6LKGV86sMSAsxdkrGBil2dzdzcSk7T+c9SerI8JcTDe
T2OehXRyelbjUVCVX41gKKR2J5mFfp+5Xo7/QLhaquLC9AIZs+k6WPpcp0C8LIO6
NIrxW5uyDs2u4bf2RJaggH1U6x2RD2qQADAPP9t7DR+5lfUW841ZkPCPBC9NBNbV
dShnuSWD2XnTZPfzAhjQiSG+b196Uc8rt/lHUOJYgis7qZ438W/G5ZG/DDu+B1XG
SvF1C+1zaDwqUhpWTRZLIG2KvfRzVg==
=qAw9
-----END PGP SIGNATURE-----

Marek Marczykowski-Górecki

unread,
Oct 2, 2022, 12:24:26 PM10/2/22
to mm, Tobias Killer, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Sep 27, 2022 at 01:15:56AM +0200, Marek Marczykowski-Górecki wrote:
> On Mon, Sep 26, 2022 at 11:33:22PM +0200, mm wrote:
> > Hi Marek,
> >
> >
> > On 9/26/22 00:01, Marek Marczykowski-Górecki wrote:
> > > Hi M,
> > >
> > > In fact, I'm working on translation-utilz right now too. Marta used her
> > > google-foo and found this gem:
> > > https://github.com/SamWilsn/docutils-rst-writer
> > >
> > this looks very nice!! :) Thanks Marta! :)
> >
> > > So, I replaced most of the qubesrstwriter2.py with just this thing and
> > > it's almost flawless (especially tables, images etc work out of the
> > > box).
> >
> > OK, thanks for the PR, I'll merge into and get rid of the obsolete stuff
> > from my branch this days.
> >
> > I have some weird customized fixes all over, but this should work.

I spent some more time on polishing the converter (all pushed to the
same PR), and I think it's good enough already. There are few remaining
issues that Sphinx complains about, but at least some of them look like
issues already present in the original markdown files, and can be fixed
in the RST version manually. I'd prefer it this way (for this category
of issues), because Sphinx gives much better error reporting than
jekyll, so it's easier to iterate.

The rendered content can be seen at https://qubes-doc-rst.readthedocs.io/en/latest/
And the converted source at https://github.com/marmarek/qubes-doc/tree/work3/

Note the above has already enabled PDF+ePUB version, and German and
Spanish translations (although most content is not translated yet).

I did also used your markdownredirector to generate redirects to RTD,
here:
https://github.com/QubesOS/qubes-doc/pull/1272

It's deployed to wwwpreview.qubes-os.org. It works for most links, for
example
https://wwwpreview.qubes-os.org/doc/secondary-storage/

But, htmlproofer complains about links to specific sections - which
indeed are broken now ('#' part does not survive redirect). For example:
https://wwwpreview.qubes-os.org/doc/#troubleshooting

I don't think that's a big issue in practice, but will require adjusting
htmlproofer call.

Anyway, I see further steps as:

1. Review and merge (or reject) current PRs to qubes-doc. They will
become completely obsolete after conversion. Some of them are waiting
for my feedback...

2. Convert the whole qubes-doc repo to ReST format, and connect to
readthedocs.org as doc.qubes-os.org (any better idea for the domain?)

3. Disconnect qubes-doc submodule from qubesos.github.io repo, and
replace with generated redirects (the content of PR 1272 above).

4. Incrementally fix remaining issues in rst files (at this point, I
don't see any critical ones). Some I see Maya fixed already in the
fixed-rst-errors dir.

5. Adjust documentation contributing guidelines for the new format (this
page actually has rst format issues after conversion, but I haven't
bothered to fix them, because the content would need significant change
anyway).

And then, translation-related:

6. Cleanup our transifex project (remove previous attempts, like
doc-related markdown files)

7. Upload rst version (already done, but will need re-uploading after
final conversion).

8. Setup CI job to pull translations to qubes-translated repo (it's a
single transifex client call, I already have it ready to go from
previous attempt).

9. Adjust theme on translated pages to include warning that non-English
version may be less accurate.

10. Write/Adjust guidelines for translators, open transifex project
again.

The above might get adjusted, if we'd like to use weblate instead, but I
think that's separate discussion (both in practice require migration to
rst first).

There is also a topic of translating the website itself, which IMO will
become much easier once documentation is moved away.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmM5uy8ACgkQ24/THMrX
1yySHgf/SPQD43P6rlOVYTQ2ESs8C5i+NKl8VpanV7EOOydXCVwBSe97Liu/Z7xv
haNyGCVSPs3/SYjRMBhMrH5TbL39KSchT4v37DMuy7DeohX5tpS9XC+p/GpecFze
V6gjuEOR47wSxoo6eFYVrn74zn57fCHiOmzms7MrDH0pmUOHlTQVb5qBVlm6SAwm
tIYFmsfAzqeSYn61FgnQ11iNBXSUZVwiG5991BvkZuSkTBe6Y9HT3AJcOwrjDFh3
D79yFE+iDi4qSJHpdvOoARtHQPMXJEn+Ou9xl+g0WGzTusayB/f/2fPkhtruigtb
CLJwWw+jr0+Q5dTdc8kZuVCsENGn7g==
=Do2o
-----END PGP SIGNATURE-----

mm

unread,
Oct 4, 2022, 1:39:09 AM10/4/22
to Marek Marczykowski-Górecki, Andrew David Wong, Tobias Killer, qubes...@googlegroups.com
Hello Marek, hello Andrew,

    
I can have spare time starting Sunday to focus & reply.
(Please, Marek, recall that i have pointed out mid September that I am out off the map
basically from 15.09 till 8.10.)
Nevertheless tried to fix different issues in some spare time, such as embedding videos, toctree etc.
I will push my final version on master asap.  (Still haven't gotten to the PR/merge/refactor), suggest to wait till Sunday.
If that is not possible there are several issues to be considered:
0. please pay attention to config.toml and conf.py from master and the files in preparation.
1. which rst files are to be copied (there are 3 atm, since the most issues should be fixed via rstwriter2), which to be removed,
different extensions to be configured such as redirects (I suggest leaving an empty hcl.rst (downloads etc) with a redirect, to be populated in a 
later stage via jinja scripting), new strikeout extension and new role, video directives, which markdown files are also to be copied 
- is doc.md the right one etc? latex documents are rendered correctly (toctree req)
2. !! Problem that should have a solution:
f.ex. https://qubes-doc-rst.readthedocs.io/en/latest/user/troubleshooting/installation-troubleshooting.html 
in the sentence:
"Here are the steps to fix this. Note that this allows sys-net and sys-usb to take complete control of the system, as described in the FAQ here:"
there should be a cross reference to FAQ VT-d iommu section but is not. (with my version this is also a problem)
Unfortunately the documentation (sphinx referencing via roles, perhaps the implementation?) is not saying anything how to handle punctuation with sections and cross ref with ref,
so as of the moment I do not have an idea, apart from the basic stuff.
I also tried explicit automatic labeling above each section with a directive  (.. _id-of-section) - did not work out

3. different new files to be considered:
f.e.x 2 versions of the privacy content, how to edit the documentation - markdown & rst version, visual style guide etc
4. what is done so far on master
-- video directives 
-- toctree index, notes, schedules, upgrade, how to restore
-- section cross referencing small fixes
-- svg conversion to png for selected files and replacement in rst docs
-- convert leftover markdown links in rst files
-- markdown redirects
-- added directives for warning & note (can be used also for indicating translated content)
-- anything i forgot

5. weblate - localization platform should be the way to go imho (sry not a big fan of transifex rn, weblate is OS etc, 
some more objective analysis will follow), there are several issues to be clarified, wip, tails guys need good questions to work with 
- will be done on sunday
6. anything i forgot, did not address, mentioned too briefly or not clarified i will address on sun 
(if it is still needed)

all the best,
m 

Marek Marczykowski-Górecki

unread,
Oct 4, 2022, 6:29:25 AM10/4/22
to mm, Andrew David Wong, Tobias Killer, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

Thanks for the update. Don't worry about the schedule - while we do want
to finish this soon, it doesn't need to be yesterday.


On Tue, Oct 04, 2022 at 06:54:45AM +0200, mm wrote:
> Hello Marek, hello Andrew,
>
> I can have spare time starting Sunday to focus & reply.
>
> (Please, Marek, recall that i have pointed out mid September that I am out off the map
>
> basically from 15.09 till 8.10.)
>
> Nevertheless tried to fix different issues in some spare time, such as embedding videos, toctree etc.
>
> I will push my final version on master asap.  (Still haven't gotten to the PR/merge/refactor), suggest to wait till Sunday.
>
> If that is not possible there are several issues to be considered:
>
> 0. please pay attention to config.toml and conf.py from master and the files in preparation.
>
> 1. which rst files are to be copied (there are 3 atm, since the most issues should be fixed via rstwriter2), which to be removed,
>
> different extensions to be configured such as redirects (I suggest leaving an empty hcl.rst (downloads etc) with a redirect, to be populated in a
>
> later stage via jinja scripting), new strikeout extension and new role, video directives, which markdown files are also to be copied
> - is doc.md the right one etc? latex documents are rendered correctly (toctree req)

I took toctree-example.rst and manually synced with recent doc.md. It
seems to work, including PDF version. The PDF could use explicit TOC at
the beginning, but it is in PDF metadata, so can be seen in the side
panel of PDF viewer.


> 2. !! Problem that should have a solution:
>
> f.ex.https://qubes-doc-rst.readthedocs.io/en/latest/user/troubleshooting/installation-troubleshooting.html
>
> in the sentence:
>
> "Here are the steps to fix this. Note that this allows sys-net and sys-usb to take complete control of the system, as described in the FAQ here:"
> there should be a cross reference to FAQ VT-d iommu section but is not. (with my version this is also a problem)
>
> Unfortunately the documentation (sphinx referencing via roles, perhaps the implementation?) is not saying anything how to handle punctuation with sections and cross ref with ref,
>
> so as of the moment I do not have an idea, apart from the basic stuff.

Yes, I hit similar issues elsewhere. Most of issues like this I fixed by
importing objects.inv and taking references from there (this is also
pushed to the PR), but there are still few corner cases like the noted
above. It seems '&' is handled differently by markdown and sphinx. All
the sphinx warnings fit for me on one screen, so I think those few
remaining cases can be fixed manually, there is no need to make the
converter perfect.

>
> I also tried explicit automatic labeling above each section with a directive  (.. _id-of-section) - did not work out
>
> 3. different new files to be considered:
> f.e.x 2 versions of the privacy content, how to edit the documentation - markdown & rst version, visual style guide etc

Yes, I left some of them on the old site (with matching redirects), but
this needs to be sorted out.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmM8Cv8ACgkQ24/THMrX
1yyO1wf+K/A4ZwSRWbNKczQxphUCJawL/dT4bE4QsxxfXJ+sr1WxWdFzXaOreawz
0qvEgnyoLgvtMaiyIpddRKSDJZYT176prLjaGIbTtgcYoddldorXlgI+Tt7/TWpi
LtYp9T1LLm1+6El5IsitCR06odmStiGK/7txyZQueZMB4636D4njm9zEpeIgLg3o
yKXZxkbs3pt/Y2uqHNc9ew1vjqRdl5XhdTFxELbZ3sEt0aWpeCq9+AM3Q9TR8tbT
ZOnoH2Lta04VU2/GNdPYaI1an0XNANPX8QuGPHvXx9RWvYyoHnDh4MaJijU032Da
6nS74QEPSWuVmjOMidYdgp+TBzTuDQ==
=x7tI
-----END PGP SIGNATURE-----

Tobias Killer

unread,
Oct 4, 2022, 1:20:38 PM10/4/22
to mm, qubes...@googlegroups.com
Hi, m,

Am 25.09.22 um 23:31 schrieb mm:
> Hey Tobias,
>
>
> I do not know unfortunately what you are referring to?
>
> I mean the objective is to convert the markdown documentation to rst asap
>
> Unfortunately it did not work out out of the box, due to docutils niceties.
>
> background: I refuse to publish my regular-expressions-monster-convert
> script.
>
> I had to write a new one using docutils, as it should be.
>
> Status as now is:
>
> there are some minor fixes to be dealt with, such as container for
> video-tours directive, registering of doc and ref roles
>
> and a directive for strike through handling, so that the conversion can
> be done with minor headaches by the qubes os team
>
> (hint: I just pushed master branch on qubes-translation-utilz where the
> current status can be feared :) )
>
>
> Regarding translation as a whole: Michael suggested weblate, as tails
> have experience with it and it seems very nice as OS alternative to
> transifex.
> Once there are further infos on that will summarize and write.
>
> Do you mean that or smth else I forget to mention here?

Yes I mean that. Sorry for my late answer.

Nice to see that the work is ongoing. Thank you for your work!

Best regards,
Tobias

mm

unread,
Oct 10, 2022, 6:14:56 PM10/10/22
to Marek Marczykowski-Górecki, Andrew David Wong, Tobias Killer, qubes...@googlegroups.com
Hi Marek,

The current status is as follows:

I dummy merged rst-writer branch, doing no refactoring at all.
I am using my implementation as crude as it is, as i am always directly
writing to the body.

But as you said, it must be done only one time and I think the current
results are satisfactory enough.
For a proper merge and rewriting I would need another 2 days, or about
4-5 hours to be sure it is not broken after fixing :).

To be on the safe side I will write down in detail the steps to take if
you want to convert asap and not wait

0. branch is master

1. in the current config.toml (all the directories, redirect base url
are set, otherwise the defaults should work) set all to true for the
section 1st stage:
pypandoc = true
copy_md_files = true
copy_rst_files = true
remove_rst_directory = true
remove_rst_files = true

2. after that, set 1. to false and run the 2nd stage with true:
docutils_validate = true
qubes_rst = true
svg_png_conversion_replacement = true
markdown_links_leftover = true

the validator should output 2 files to be fixed in 5 places: just fix
the title's marking
the first time from - to ~, the second time from ~ to - as in:

Rollback template changes
-------------------------
and not

Rollback template changes
~~~~~~~~~~~~~~~~~~~~~~~~~

3. let it run after fixing, so that cross referencing and further fixes
are applied.

4. to be on the safe side do grep for SYSTEM inside the _doc directory
for system errors (will be written in the files)

5. Manually fix warnings:

5.1 nice to haves
there are several warnings that I wish to ignore, such as
Bullet list ends without a blank line; unexpected unindent. or
Title underline too short.
But actually are easy to fix, and do not take much time.

5.2 toctree

developer/general/how-to-edit-the-documentation.rst: WARNING: document
isn't included in any toctree (how-to-edit-the-documentation mirrors is
in the toctree of index.rst but it refers directly to the
https://www.qubes-os.org/doc/how-to-edit-the-documentation/ link)
developer/general/visual-style-guide.rst: WARNING: document isn't
included in any toctree (visual-style-guide mirrors is in the toctree of
index.rst but it refers directly to the
https://www.qubes-os.org/doc/visual-style-guide/ link)
developer/general/website-style-guide.rst: WARNING: document isn't
included in any toctree (website-style-guide mirrors is in the toctree
of index.rst but it refers directly to the
https://www.qubes-os.org/doc/website-style-guide/ link)

user/downloading-installing-upgrading/download-mirrors.rst: WARNING:
document isn't included in any toctree (download mirrors is in the
toctree of index.rst but it refers directly to the
qubes-os.org/downloads/mirrors link)
user/downloading-installing-upgrading/downloads.rst: WARNING: document
isn't included in any toctree (downloads is in the toctree of index.rst
but it refers directly to the qubes-os.org/downloads link)
user/hardware/hcl.rst: WARNING: document isn't included in any toctree
(hcl is in the toctree of index.rst but it refers directly to the
qubes-os.org/hcl link)

5.3 unknown label/document

The label problems are several categories:

1. non existing documents/files:
a.

developer/services/admin-api.rst:60: WARNING: unknown document:
/developer/services/admin-api-table
developer/services/admin-api.rst:591: WARNING: unknown document:
/developer/services/admin-api-table

admin-api-table is currently deleted. so admin-api.rst should be freed
form those references.

b.
user/downloading-installing-upgrading/download-mirrors.rst:13: WARNING:
undefined label: user/downloading-installing-upgrading/downloads:mirrors
user/downloading-installing-upgrading/supported-releases.rst:13:
WARNING: undefined label:
user/downloading-installing-upgrading/downloads:mirrors
introduction/privacy.rst:68: WARNING: undefined label:
user/downloading-installing-upgrading/downloads:mirrors

- the link should be static for now to the
qubes-os.org/downloads/mirrors one


2. Syntax

a. when there is & in the section title
project-security/security.rst:17: WARNING: undefined label:
introduction/faq:general security

:ref: `Security FAQ <introduction/faq:general \& security>`

b. When there is an emphasis in the section title such as ``qubes.bla``
section

user/how-to-guides/how-to-update.rst:70: WARNING: undefined label:
user/advanced-topics/salt:update.qubes-dom0

:ref:`update.qubes-dom0 <user/advanced-topics/salt:``update.qubes-dom0`\`>`

the same for:
user/how-to-guides/how-to-update.rst:71: WARNING: undefined label:
user/advanced-topics/salt:update.qubes-vm
developer/debugging/vm-interface.rst:312: WARNING: undefined label:
developer/services/qrexec-internals:qrexec policy implementation
user/advanced-topics/disposable-customization.rst:161: WARNING:
undefined label: user/how-to-guides/how-to-use-pci-devices:qvm-pci usage

c. when there is a bold emphasis
project-security/verifying-signatures.rst:311: WARNING: undefined label:
user/how-to-guides/how-to-copy-from-dom0:copying from dom0

:ref:`copied <user/how-to-guides/how-to-copy-from-dom0:copying **from**
dom0>`


d. Misc
user/security-in-qubes/split-gpg.rst:559: WARNING: undefined label:
user/templates/templates:trusting your templates

There should be a blank link before the section Trusting your templates


Final steps:


1. new rst-doc-repo bc of redirects in old markdown documentation
2. attachment should be added as a submodule to the new rst doc qubes repo
3. new privacy policy for documentation hosted on the read the docs (as
long as it is not self-hosted)
4. new how to edit the rst documentation file, there is an initial
version of it (more or less the same with the original one)
in the preparation dir in the rstwriter repo:
how-to-edit-the-rst-documentation.rst
    - Edit links of the original how to edit the documentation file
(now residing in the official website) such as in contributing.rst to
the rst version
5. new rst documentation style guide
6. I suggest to add for all the empty files (still existing but in the
official website) such as hcl.rst redirects.
Example would be:

extensions = [
    'sphinx.ext.autosectionlabel',
    'sphinxnotes.strike',
    'sphinx_reredirects'
]

redirects = {
    "user/hardware/hcl": "https://google.com"
}


7. plant the redirects in the official website. (did you encounter any
problems, have i missed smth?)

There will be a separate mail about weblate/transifex/translation etc.


All the best,
m

mm

unread,
Nov 1, 2022, 7:45:39 AM11/1/22
to Marek Marczykowski-Górecki, Andrew David Wong, Tobias Killer, qubes...@googlegroups.com
Hi Marek, hi Andrew, hi Tobias,

Marek, I merged your pull request, also merged your changes into master
and added some enhancements and created a pull request.

Here you could review it
https://github.com/maiska/qubes-translation-utilz/pull/2

Either way, one could try it out, there should be no more undefined or
unknown label warnings when rendering the rst docs
and thus could safely imho go and convert the documentation and host it
on RTD.

Here is a README.md instead of spamming via email
https://github.com/maiska/qubes-translation-utilz/blob/master/README.md

the configuration should be more readable also:

https://github.com/maiska/qubes-translation-utilz/blob/master/md2rst/config.toml

Please let me know if I have to fix further or this suffices.

Otherwise I will briefly handson evaluate weblate locally and report back.


@Andrew,

regarding the minimal debian jekyll vm, here is a way with a template
and an app vm.

qvm-clone debian-11-minimal jekyll-tvm

in jekyll-tvm:
apt install qubes-core-agent-networking
apt install ruby-full build-essential zlib1g-dev vim
apt install qubes-core-agent-passwordless-root
apt install firefox-esr git

in jekyll-app-vm:
echo '# Install Ruby Gems to ~/gems' >> ~/.bashrc
echo 'export GEM_HOME="$HOME/gems"' >> ~/.bashrc
echo 'export PATH="$HOME/gems/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
gem install jekyll bundler
find . -name gem # '/home/user/.local/share/gem/'
bundle config set --local path '/home/user/.local/share/gem/'
git clone -b new-master --recursive
https://github.com/QubesOS/qubesos.github.io.git; cd qubesos.github.io.rtd/
bundle install
bundle exec jekyll serve --incremental



All the best,

m



On 10/4/22 12:29, Marek Marczykowski-Górecki wrote:

Marek Marczykowski-Górecki

unread,
Nov 1, 2022, 8:43:12 AM11/1/22
to mm, Andrew David Wong, Tobias Killer, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Nov 01, 2022 at 12:45:33PM +0100, mm wrote:
> Hi Marek, hi Andrew, hi Tobias,
>
> Marek, I merged your pull request, also merged your changes into master and
> added some enhancements and created a pull request.
>
> Here you could review it
> https://github.com/maiska/qubes-translation-utilz/pull/2
>
> Either way, one could try it out, there should be no more undefined or
> unknown label warnings when rendering the rst docs
> and thus could safely imho go and convert the documentation and host it on
> RTD.
>
> Here is a README.md instead of spamming via email
> https://github.com/maiska/qubes-translation-utilz/blob/master/README.md
>
> the configuration should be more readable also:
>
> https://github.com/maiska/qubes-translation-utilz/blob/master/md2rst/config.toml
>
> Please let me know if I have to fix further or this suffices.

Thanks. I'm a bit busy this and the next week, but maybe I'll manage to
take a look. Otherwise, after 11th of November.

Do you have somewhere a conversion output pushed?

> Otherwise I will briefly handson evaluate weblate locally and report back.

Sounds like a good plan.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmNhFFkACgkQ24/THMrX
1yx+YAgAhwv2RfvibWgnq4kkOz74E4VEBI2p5oh7OGHDvHc1jPpQAUQfeg4ibfHy
ZKWkjwop/zTfP19i1XxC2wqAQ0bCXN0TKz4YHpAFI+xnsodFCiVMRYz4O/ShcN5i
aklA8xuu9pG7G3hwT6koVWjbASCRZowDkWGnD0n5TeoABH9DA4TX2TA51dOkZkig
ztW+MvyNAeG38Crabp4XYYugs83kE1dJKpqarN5VyYzkGycGxFcdLWu6zfwSspen
njsD8Rvp58hPc0uvA1tfp87CZmjagdZL+jOsLTP/3SCS3KK5ZzcHhMbaneVxnuDv
NvfxCkVvZUuDgvzdgAZi6nRLvSsDGA==
=/WZZ
-----END PGP SIGNATURE-----

mm

unread,
Nov 1, 2022, 4:01:31 PM11/1/22
to Marek Marczykowski-Górecki, Andrew David Wong, Tobias Killer, qubes...@googlegroups.com

On 11/1/22 13:43, Marek Marczykowski-Górecki wrote:
On Tue, Nov 01, 2022 at 12:45:33PM +0100, mm wrote:
> Hi Marek, hi Andrew, hi Tobias,

> Marek, I merged your pull request, also merged your changes into master and
> added some enhancements and created a pull request.

> Here you could review it
> https://github.com/maiska/qubes-translation-utilz/pull/2

> Either way, one could try it out, there should be no more undefined or
> unknown label warnings when rendering the rst docs
> and thus could safely imho go and convert the documentation and host it on
> RTD.

> Here is a README.md instead of spamming via email
> https://github.com/maiska/qubes-translation-utilz/blob/master/README.md

> the configuration should be more readable also:

> https://github.com/maiska/qubes-translation-utilz/blob/master/md2rst/config.toml

> Please let me know if I have to fix further or this suffices.

Thanks. I'm a bit busy this and the next week, but maybe I'll manage to
take a look. Otherwise, after 11th of November.

All right.

Here is a preview: https://github.com/maiska/qubes-rst

Marek Marczykowski-Górecki

unread,
Nov 28, 2022, 11:05:59 AM11/28/22
to mm, Andrew David Wong, Tobias Killer, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Nov 01, 2022 at 12:45:33PM +0100, mm wrote:
> Hi Marek, hi Andrew, hi Tobias,
>
> Marek, I merged your pull request, also merged your changes into master and
> added some enhancements and created a pull request.
>
> Here you could review it
> https://github.com/maiska/qubes-translation-utilz/pull/2
>
> Either way, one could try it out, there should be no more undefined or
> unknown label warnings when rendering the rst docs
> and thus could safely imho go and convert the documentation and host it on
> RTD.
>
> Here is a README.md instead of spamming via email
> https://github.com/maiska/qubes-translation-utilz/blob/master/README.md
>
> the configuration should be more readable also:
>
> https://github.com/maiska/qubes-translation-utilz/blob/master/md2rst/config.toml
>
> Please let me know if I have to fix further or this suffices.

I think the current version of scripts is good :) Thanks!

There are sphinx warning remaining, but there are not that many of them
and can be fixed manually (things like underscores for headers, or
single space at the beginning of line interpreted as unexpected
indentation).

The only other issue I consider a blocker (but hopefully should be easy
to fix) is index.rst. The sidebar index is great, but in its current
form it's hard to use - it's very long. I added section captions there
and it helped a bit, but I think those should be folded by default
(similar folding like sections of the current page). I tried setting
`collapse_navigation`[1], but it didn't work. Any other ideas?

BTW, while looking for related option, I found you can use `:glob:`
in toctree, to add everything from given subdirectory. This way we can have
some section filled automatically (as long as directory structure
reflects sections, but it is the case in most cases).

I uploaded current version to
https://qubes-doc-rst.readthedocs.io/en/latest/ (I haven't fixed all
sphinx warnings there, so few places may have wrong indentation). I
haven't updated translated pages - those are still from previous
iteration.
Source of the above is at https://github.com/marmarek/qubes-doc/tree/work4/

Another thing that IMO would be better, is to keep the "introduction" page
as part of the main website, and only link to it from the documentation.
Should be easy with the redirects extension (as already done for HCL and
few others).

The next question is what to do about qubes-doc repo. We can either
replace the current content with converted to rst (and keep markdown in
a separate branch, for historical reasons), or create new one
(qubes-doc-rst?), and basically archive qubes-doc.

In any case, the _doc submodule should be replaced with generated
redirects to the new documentation. IMO it can be put into
qubesos.github.io repository directly, as it shouldn't really change
(it's only to keep existing documentation links on mailing list, forum
etc working, shouldn't be used for new content).

[1] https://sphinx-rtd-theme.readthedocs.io/en/stable/configuring.html

> Otherwise I will briefly handson evaluate weblate locally and report back.

Do you have any update on weblate?
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmOE3F8ACgkQ24/THMrX
1yxPFwf7BeLksa45KgWFAgU2tDpIoe9Rx3YukKrtD5UtmgzUgCdQTlEUamF3OpND
lqjG5zet59CvBWFCu3wx5p1XBy4Ohp2MxwX1gK+A+XOrvmOJNSIFhLabE7mEZSul
DpLknfgZ/0dh1ITg3Qgd+79FkYqPaC2ulOUnfozoC6pP8aqzelu85W9WPtDPxU73
H3+MDoJ19zFExOBFa/21A9qrNNZvd97t/O9+hwTwXErXr2GdM3k8u0AfutUm2zHU
saEyfnSDkR1hF2ANwCVfSnNCa/99nsA2sAnuVkwbsinTDSEBpkWcDw5PhOwBQvuz
3zFhHBZ0VuSKlDX3Wjb6sLJmhk8LZw==
=sxMo
-----END PGP SIGNATURE-----

Andrew David Wong

unread,
Nov 28, 2022, 7:51:51 PM11/28/22
to Marek Marczykowski-Górecki, mm, Tobias Killer, qubes...@googlegroups.com
Git already keeps everything for historical reasons, so I don't think it's necessary to keep a separate branch around *forever* just for historical reasons, which would create extra clutter and get in the way, but keeping the historical branch for, say, a year would make sense. For a permanent marker, it seems cleaner just use a tag to indicate where the transition occurred.

As far as I can tell, the main difference between reusing the same repo and creating a new repo is simply that, in the latter case, history would be split across two repos, and we'd have to update all of our references to "qubes-doc" everywhere. So, I'm inclined to say we should keep using the existing repo.

Tobias Killer

unread,
Aug 2, 2023, 1:50:30 PM8/2/23
to Andrew David Wong, Marek Marczykowski-Górecki, mm, qubes...@googlegroups.com
Hi, folks,

after a long time of silence (sorry for that!), I'd like to participate
in the translation issue again. :-)

What is the current state? What is already there and what is still missing?

I don't know what you think, but for me it's hard to see the whole. The
translation issue started small (and I thought it would remain small),
but now, I see many repositories, open pull requests and emails with
dozens of different links to previews etc.

No offense! I'm very happy to see that the work had made progress by m
(Thank you, m!). But now, I also feel like a stranger and I see a lot of
work to read in order to setup my small brain to understand the huge
current state (which is widely spreaded and doesn't seem to be
documented up-to-date).

Am I exaggerating?

Maybe, I should first summarize the current state and write it down
somewhere. Would that be useful? Or is there an up-to-date documentation?

Best regards,
Tobias Killer

Tobias Killer

unread,
Nov 29, 2023, 1:03:25 PM11/29/23
to mm, Andrew David Wong, Marek Marczykowski-Górecki, qubes...@googlegroups.com
Am 02.11.23 um 19:25 schrieb mm:
> Hey Tobi,

Hey m,

> thank you for your email and the re-aliving the thread! :)
>
> So, the current status is - the final rehearsal for the conversion of
> the documentation from markdown to rst was I think 2 weeks ago? It went
> well, meaning, at least from my pov there were no errors, that one
> should have dealt with.

Great!

> When will it be completely finalized? - up to @marek and Qubes
> release/other time schedule?

The earlier the better.

> The markdown to rst documentation can be found partly here:
> https://github.com/maiska/qubes-translation-utilz#readme and
> https://github.com/maiska/qubes-translation-utilz/blob/main/md2rst/config.toml. There is no real write up :/

The code in this repo looks comprehensive. I didn't test it yet on my
machine (TBD).

> As of now there is no written up documentation about weblate, but there
> is/was a demo and I have a lot of mini how tos, for everything I tried
> out. It would be cumbersome to write it down :)  I could enhance the
> presentation pdf to incorporate more information, and publish it though
> if there is the need/wish to do so.

I think we only need to document Qubes-specific details about weblate.

> Tbh, I would like to do that when the Qubes documentation is in
> Restructured text format and there is a decision - ok, Qubes is going to
> use weblate with an account or self-hosted, and then i would suggest a
> meeting online or live to discuss things, for which I would gladly
> prepare a README

I prefer email over online meetings but I'm open to be overruled.

>
> What do you think?
>
> All the best,
>
> m

Best Regards,
Tobias
Reply all
Reply to author
Forward
0 new messages