Should we migrate the documentation to another platform?

22 views
Skip to first unread message

Andrew David Wong

unread,
Sep 27, 2021, 7:46:44 AM9/27/21
to qubes-devel
## Summary

Currently, the Qubes documentation is written in plain text Markdown
files, version controlled with Git, rendered using Jekyll, and hosted
from GitHub Pages. This setup has served us well over the years, but as
the demands on the documentation have grown, it has become increasingly
difficult for the current setup to meet our needs. Therefore, it may be
time to consider migrating to a dedicated platform, such as Read the
Docs or a standard wiki. The purpose of this thread is to help us make
the best decision for our users and for the project as a whole.


## Pros and cons of the status quo

There are good reasons for our current setup:

1. History of changes is transparent and immutable thanks to Git.
2. Authenticity can be verified thanks to PGP-signed commits and tags.
3. Unauthorized changes are prohibited thanks to signature-checking.
4. Plain text files can be read directly, offline, even in a terminal.
5. Full control over web-view styling with custom CSS.
And more...

However, there are also challenges:

6. Difficult for many users to contribute. Requires use of GitHub's web
editor at the very least or local command-line usage for full control.
No WYSIWYG editor. (GitHub offers a Markdown preview, but you have to
serve the site locally to see how your changes will really look on the
live website.)

7. Serving the website on one's local machine is cumbersome and
error-prone. Upstream changes often break the workflow. Even users who
have all the required software installed and follow the exact same steps
we do sometimes can't get it to work.

We used to instruct users to install Jekyll and run it directly to serve
the website, but they often couldn't get it to work. We thought that
switching to Docker/Podman would solve this "works on my machine but
doesn't work on yours" problem, but it still often doesn't work out in
practice. (Currently, even I'm having trouble getting it to work
consistently on my own development machine.)

8. Requires a complex backend to achieve basic functionality that's
standard elsewhere. Since we're hosted from GitHub Pages, we can't use
most Jekyll plugins, which means we have to write and maintain custom
code for basic things like per-page tables of contents, language
switching, and version switching.

To support translations, we've had to drastically increase the
complexity of all the layouts, includes, and data files that constitute
the guts of a Jekyll website. We've had to add a lot of Liquid code in
these files to do the kinds of processing we need, and we have to hook
it all up to complicated series of Git repos, subrepos, and branches.
This makes it very difficult to understand and maintain.

Anyone who wants to contribute or help out with maintenance faces a
steep learning curve, which discourages contributions and changes. Even
an existing expert in Git, Jekyll, Liquid, and Markdown would have to
become acquainted with all of our custom code. And most of it remains
undocumented, as many of the people who built it had to rush off to
other tasks due to lack of time.

9. Related to the previous point, adding new functionality becomes
increasingly difficult as the system grows more complex. The latest
example is release-specific documentation. There are many small design
decisions that have to be made, and it all has to be done in a way
that's compatible with translations. We can't simply create a new branch
and call it a day, because all the customized guts of the Jekyll system
have to be changed to account for multiple versions of documentation,
and the documentation files themselves are a mix of release-specific and
non-release-specific files which have to be reorganized and classified
with new metadata.

Jekyll doesn't support variables in YAML, so we can't just add a release
variable to our existing YAML files and headers. We have to either
duplicate them for every release or fundamentally redesign large swaths
of the system with layers of indirection to work around such
limitations. Even bulk-manipulation of the YAML headers in all the doc
files requires us to write our own special tools, which themselves then
have to be maintained.

And all of this only gets us to the point where different doc sets can
simply *exist* without everything breaking. Exposing them to users in a
friendly way would be an entire further stage of the process, in which
we'd probably have to write a lot more custom code just to get a simple
dropdown menu that allows users to switch between versions. All of this
work would probably result in something that's still not as good as
existing off-the-shelf solutions backed by dedicated teams who
specialize in developing documentation management systems.


## Why now?

We've had the current setup for a long time. Why should we consider
migrating now? We've considered it in the past and decided against it at
the time, so what's changed? The answer is that there are new and
increasing demands on the documentation, the main two being localization
(i.e., translations) and release-specific docs.

Back before we tried to address these kinds of needs, our documentation
setup was simpler and served us well. The backend was smaller, easier to
understand, and easier to maintain, since it didn't need to do much.
There was only a single, canonical version of every documentation page.
The pros mentioned above easily outweighed the cons, since the cons list
was much shorter back then.

However, times change. As our project has matured, we've decided it was
time to acknowledge and take seriously the reality that not everyone
reads English and not everyone uses the same version of Qubes OS at the
same time. We have users and contributors from around the world, and we
have multiple versions of Qubes.

There are times when an older Qubes release is supported concurrently
with a newer release to give users time to upgrade. Even when there's
only one currently-supported release, there's usually a new one in
development and testing that folks want to start documenting. This means
that we have to support a separate version of each appropriate
documentation page for every translated language *and* for every
supported and in-development Qubes version, not to mention historical
documentation for EOL releases.


## What are our options?

Looking around at other projects similar to ours, it appears that two of
the most popular options are Read the Docs (RTD) and standard wiki
software. Here are what I currently understand to be some of the pros
and cons of each. (I haven't personally used RTD very much, so some of
this is based on hearsay. I'm sure these lists will change as we learn
more.)

RTD:
+ Built-in release-specific doc support.
+ Built-in localization support.
+ Supports Git for version control and history. (Presumably, we could
continue to use signed commits and tags for user and content
authentication.)
+ GitHub integration.
+ Supports Markdown (natively, AFAIK, though many seem to use RST). We
currently use GitHub-flavored Markdown, so some conversion may be
necessary but shouldn't be too bad.
+ Built-in support for downloading/exporting docs in multiple formats
for offline reading. Direct access to human-readable Markdown source
files in Git repos should also continue to be possible.
+ Includes search functionality.
+ Widespread use among open-source projects. Established history.
Probably not going away anytime soon (though it could eventually, e.g.,
if the project behind it ever shuts down).
+ Can be self-hosted.
? Unclear exactly how easy it'll be for users to contribute. (Almost
certainly no more difficult than it is now.)
? [Subjective] Some people think RTD looks better than a wiki.
? Might look different from the rest of our website. Might require a
subdomain.

Wiki:
+ Familiar to almost everyone thanks to Wikipedia and specialized wikis.
+ Easy for users to contribute.
+ Built-in version control and history.
+ Includes search functionality.
+ Even more widespread use throughout the web. Even longer established
history. Practically ubiquitous. Definitely not going away anytime soon.
+ Can be self-hosted.
? Unclear how good the security would be with respect to user and
content authentication.
? Unclear to what extent localization support is built-in. There are
probably plugins. Unclear how easy Transifex integration would be.
? Would probably look different from the rest of our website. Might
require a subdomain.
? [Subjective] Some people think wikis look bad.
- No built-in support for release-specific docs (AFAIK).
- No (common, standard, native) Git support.
- Native wiki syntax is different from Markdown. Markdown plugins are
available but can be problematic. Markdown can be converted into wiki
syntax using, e.g., pandoc, but will probably still require some manual
cleanup. However, once this conversion is done, contributions could be
written natively in wiki syntax with no ongoing conversions required.
- Probably wouldn't easily support direct access to human-readable
source files. Offline docs might require a local wiki instance.

## Conclusion

Overall, I'm leaning toward RTD, but I don't yet know what it'll be like
to maintain the docs there, so I'd like to experiment with it a bit
before forming an opinion. Of course, we're also open to considering
other options (including the option of not migrating at all). Finally,
although I've been saying "documentation" throughout this post, we may
also want to consider migrating the entire website if it turns out that
keeping everything together makes more sense.

So, what do you all think? Again, the purpose of this thread is to help
us make the best decision for our users and for the project as a whole.
In addition to the user-facing aspects, I'd also like us to consider
long-term maintenance requirements and the experience for contributors.

--
Andrew David Wong
Community Manager
The Qubes OS Project
https://www.qubes-os.org

OpenPGP_signature

Konstantin Ryabitsev

unread,
Sep 27, 2021, 8:58:45 AM9/27/21
to Andrew David Wong, qubes-devel
- Attracts lots of spammers, such that you either have to spend a lot of time
managing approved edits and approved editors, largely defeating the purpose
of a wiki.

> Overall, I'm leaning toward RTD, but I don't yet know what it'll be like to
> maintain the docs there, so I'd like to experiment with it a bit before
> forming an opinion.

You can self-host RTD-built documentation, but this does defeat a lot of
builtin nice-to-haves like github integration. At least with RTD you always
have an option of switching to self-rendering without any loss of
functionality.

I heartily recommend RTD. We stopped accepting new wikis on the kernel.org
side of things because they almost inevitably become a spam-haven [1].
I believe wikis as a concept are largely dead except for very large projects
(wikipedia, wikia, etc) that have the resources to keep spammers away.

-K

[1] https://korg.docs.kernel.org/docs.html

Sven Semmler

unread,
Sep 27, 2021, 12:24:17 PM9/27/21
to qubes...@googlegroups.com
> So, what do you all think?

The goal should be that a lot more of the available attention and energy can flow towards the actual content, rather than maintaining "the system".

> RTD vs. Wiki

Wiki's are ubiquitous and very easy to edit. But that could end up being a problem, as Konstantin already pointed out.

RTD appears to be purpose built for what we need and widely used. The ability to retain git version history and signed commits is a huge plus. With documentation having the wrong information displayed is worse then having nothing. So making it easy to contribute is important, but having a tight review before publishing is too. Wiki's seem to be more of a mess in that context.

/Sven

--
public key: https://www.svensemmler.org/2A632C537D744BC7.asc
fingerprint: DA59 75C9 ABC4 0C83 3B2F 620B 2A63 2C53 7D74 4BC7
OpenPGP_signature

Wojtek Porczyk

unread,
Sep 28, 2021, 8:59:08 AM9/28/21
to Andrew David Wong, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Mon, Sep 27, 2021 at 04:46:28AM -0700, Andrew David Wong wrote:
> RTD:

RTD, apart from sphinx, also supports mkdocs, though I've never used it.

> + Built-in release-specific doc support.
> + Built-in localization support.
> + Supports Git for version control and history. (Presumably, we could
> continue to use signed commits and tags for user and content
> authentication.)
> + GitHub integration.
> + Supports Markdown (natively, AFAIK, though many seem to use RST). We
> currently use GitHub-flavored Markdown, so some conversion may be necessary
> but shouldn't be too bad.
> + Built-in support for downloading/exporting docs in multiple formats for
> offline reading. Direct access to human-readable Markdown source files in
> Git repos should also continue to be possible.
> + Includes search functionality.
> + Widespread use among open-source projects. Established history. Probably
> not going away anytime soon (though it could eventually, e.g., if the
> project behind it ever shuts down).
> + Can be self-hosted.
> ? Unclear exactly how easy it'll be for users to contribute. (Almost
> certainly no more difficult than it is now.)

"Edit on Github" will be less friendly, because rST is a second-class citizen
on github. Also, markdown is second-class citizen in sphinx. There are many
things that are possible only in rST.

You can mix both markup languages in a single project (not in single page),
but that would require explicit policy, what should be written in which
language. I've once migrated one github wiki to sphinx just by copying all GFM
pages to spinx and then rewrote them one by one.

> ? [Subjective] Some people think RTD looks better than a wiki.
> ? Might look different from the rest of our website. Might require a
> subdomain.

Those two points very much depend on sphinx' theme. Wouldn't you want to port
the current layout to sphinx theme?

We already use RTD for dev.qubes-os.org (with sphinx, rST and one of the stock
themes) and it works.


With sphinx it's sometimes tricky to get rendering reproducible, i.e. `make
html` on your local system renders slightly differently than the "official"
build on rtfd.io. Those differences are largely limited to roles and
glossaries, but are still noticeable. (At least that's my experience with
sphinx-rtd-theme in another project. For compatibility with Ubuntu 18.04, said
project is still stuck with Sphinx 1.8, as available in apt repo, because we
also render manpages using sphinx).


- --
pozdrawiam / best regards
Wojtek Porczyk
Gramine / Invisible Things Lab

I do not fear computers,
I fear lack of them.
-- Isaac Asimov
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEaO0VFfpr0tEF6hYkv2vZMhA6I1EFAmFTEZYACgkQv2vZMhA6
I1FGhA/+InLJxfraA0+Ohq5EZD90Ljt4Dv+Bj/Wi9/48VkkX9LPE2teYuMJCTf+x
QsDMVz5zZopWwFwnuPN8WQJLsQ5Ah5luOM8l0i2zA5UgYdG1gPycGHnJRZTLLlkr
7GvlSmfMjtwj4AZuTqQfrVkUWTQhlBZTSLkuu7/QtJeRaSlWWb990+CDecB35gxo
x1XUE6LTbLa/QJPVhREjSMDreAoyfyQEQIGLxytj3Czkl7J1tnsrJ37k8ptHcLPC
XyYYEVYIO6tURaIEF5Clh6Rxq5vh7lA7Yjs0q7i7AxTohJQBviQMhR6xuqK0FCnN
bEPuRBnqLFK34sEnMHBCxLCRl8vfRUkU0B4WuBhxycgkDMtaF3qDyOKo/lxAn5Pq
11zsjxXvz9zx2Ujmawqi1M09POeNUjx3PfDwVGTCCFeRB0eoar0dpuLRktVarA21
wcZqFCTVy2dcIe2YDK5Hos5dQu1+bZUigSmIbYDptZ3k0ozH60D8WNx8lMa+w4Mm
xRs0m5/XPSV/hSci3T86YY2aK0w3eFNInXjbW5Cw7nWGzdqXrDGh860YUD6ip2KB
vkZfHNokO173KcwPBnaxciakYjakNMq3/bSf1PwZ01cOIcqULUwi8m3dEaAb+d2F
ghaAXQGz43LAOZmQ0MQMcKqNMHXBMtgTvQsdu52mS6aAx02kh2Q=
=jGwU
-----END PGP SIGNATURE-----

Tobias Killer

unread,
Sep 28, 2021, 2:06:47 PM9/28/21
to qubes...@googlegroups.com
Hello,

one feature that is often used in Qubes Doc is the automatic generation
of HTML IDs like, for example, if the source Markdown file contains

```
# Hello world
```

the resulting HTML file contains

```
<h1 id="hello-world">Hello world</h1>
```

which means one can link to that heading via `[Link](#hello-world)`.

Are there pendants in RTD and Wiki? In what way?

Best regards,
Tobias Killer

Marek Marczykowski-Górecki

unread,
Sep 28, 2021, 2:25:14 PM9/28/21
to Tobias Killer, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
RTD does that automatically, for example:
https://dev.qubes-os.org/projects/core-admin/en/latest/qubes-events.html#handling-events-with-variable-signature

But that's for rST input, I'm not sure if the same will work with MD
input (likely yes).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmFTXgQACgkQ24/THMrX
1yx0xggAir2EbXJT0wEfTtCYy7aT8nhSYl8tjjF2dr2gXaC51C0GZLX7+XZiUxec
5HNFoo6TTFs+51ykmsVjDuuIAa2ekmVKbyty8Kl9FXzXkWtJG774oY6UCfNM+JCd
n8jZ5MjzQv6h3adfQP6QcBayPoFnzjwzKilnwsK7c1aMsNkku/jIm8nJ3zkV6omH
h/goLMesX6H20SOjLKKsLkiHIg+p01uG6rLJDnxELBNE33Sy/rSC323hDihmr/6K
wnWJR2SElqlqpptTa8vyHi41KpVODKBJ7R0SxTKVY4d7y5TJOKyrBbbD98R/uJWm
YNTXpHyVCzfEJQ5PQgVRt6VR/+YROw==
=Te7J
-----END PGP SIGNATURE-----

Tobias Killer

unread,
Sep 28, 2021, 3:00:38 PM9/28/21
to Marek Marczykowski-Górecki, qubes...@googlegroups.com
Am 28.09.21 um 20:25 schrieb Marek Marczykowski-Górecki:
> On Tue, Sep 28, 2021 at 06:06:41PM +0000, Tobias Killer wrote:
>> Hello,
>
>> one feature that is often used in Qubes Doc is the automatic generation of
>> HTML IDs like, for example, if the source Markdown file contains
>
>> ```
>> # Hello world
>> ```
>
>> the resulting HTML file contains
>
>> ```
>> <h1 id="hello-world">Hello world</h1>
>> ```
>
>> which means one can link to that heading via `[Link](#hello-world)`.
>
>> Are there pendants in RTD and Wiki? In what way?
>
> RTD does that automatically, for example:
> https://dev.qubes-os.org/projects/core-admin/en/latest/qubes-events.html#handling-events-with-variable-signature
>
> But that's for rST input, I'm not sure if the same will work with MD
> input (likely yes).
>
>

Okay, thanks!

I think a cut now (before translations go online) is better than
afterwards. I don't know RTD and Wiki but I tend towards RTD.

I wonder which restructurings of the repos `qubesos.github.io` and
`qubes-doc` will be necessary in order to migrate to RTD. But that's a
question for later on.

Best regards,
Tobias Killer

unman

unread,
Oct 1, 2021, 12:38:03 AM10/1/21
to qubes...@googlegroups.com, Andrew David Wong
From my limited experience, it will be *more* difficult for users to
contribute, unless we accept text rewrite by email, and process it
ourselves.
I agree with Wojtek: on the basis of my limited use, there can be
significant differences between local and RTD builds.
I'd also ask Nina for her view on RTD and what theme to consider - there
is a useful comparison page at
https://rtd-sphinx-theme-sample-project.readthedocs.io/en/latest/ with
some decent themes.

awokd

unread,
Oct 4, 2021, 4:16:49 PM10/4/21
to qubes...@googlegroups.com
Andrew David Wong:

> We've had the current setup for a long time. Why should we consider
> migrating now? We've considered it in the past and decided against it at
> the time, so what's changed? The answer is that there are new and
> increasing demands on the documentation, the main two being localization
> (i.e., translations) and release-specific docs.

I remember this topic came up when discussing how best to maintain R3.2
& R4.0 document sets. Would be nice to have a smarter way to do it.

Rob Townley

unread,
Oct 4, 2021, 4:59:07 PM10/4/21
to awokd, qubes-devel
The following is more a question of my own curiosity. 
Is it possible that the existing documentation setup can be put into one or more qubes_template(s) so that it is already pre-configured?  

--
You received this message because you are subscribed to the Google Groups "qubes-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/7542225d-96b0-8ca0-ea26-dfc1d40d4a2e%40danwin1210.me.

Tobias Killer

unread,
Nov 13, 2021, 10:31:23 AM11/13/21
to Andrew David Wong, qubes-devel
Am 27.09.21 um 13:46 schrieb Andrew David Wong:
Hello,

This discussion here contains a lot of Pros and Cons but no decision. Is
there already a silent decision towards "no"? At least, I cannot see any
action towards "yes".

Best regards,
Tobias Killer

Andrew David Wong

unread,
Nov 14, 2021, 5:01:47 PM11/14/21
to Tobias Killer, qubes-devel
On 11/13/21 7:31 AM, Tobias Killer wrote:
> Am 27.09.21 um 13:46 schrieb Andrew David Wong:
>
> Hello,
>
> This discussion here contains a lot of Pros and Cons but no decision. Is
> there already a silent decision towards "no"?

Nope.

> At least, I cannot see any action towards "yes".
>

I just haven't gotten around to testing it out yet.
Reply all
Reply to author
Forward
0 new messages