Updates from CentOS/Fedora Land

[Zach Lym]

I was asked by deeplow to forward/summarize info from the "Alt RPM Distro (CentOS?) in dom0" forum thread [alt-thread] to make it more visible to devs.

CentOS Stream is an attractive candidate for use in dom0 due to its stability, LTS cycle, compatibility with Fedora, and signing of both packages and metadata.  The main concerns raised about CentOS were outdated kernels, the limited set of available userland packages, the stability of CentOS Stream, and the desire for a "minimal" distro.  Recent developments largely address these concerns:

* RHEL has moved to a 3-year cycle for major releases and there is a Hyperscaler SIG [hyper-sig] working on accelerated backports; including LTS kernel releases and core userland software such as systemd.  This is inline with Qubes' ~2±1 year release cadence.

* CentOS Stream is part of Red Hat's infrastructure modernization effort to create a continuous integration pipeline from Fedora to CentOS and its derivatives.  This effort *should* reduce the friction for getting packages available in Fedora to EPEL.

* CentOS Stream is intended to be as stable as RHEL proper.  Source and community rebuilds are available for the skiddish, however, Red Hat has expanded its free license tier to 16 production instances.  Qubes also probably qualifies for Red Hat's FOSS infrastructure program.  

* Fedora Silverblue, Fedora CoreOS, Fedora Minimal, and Red Hat's Universal Base Image are all freely available and competitive with other "minimal" distros in terms of TCB and feature set.  Rocky Linux confirmed over Matrix that RHCOS rebuilds are in the works.

While it will take time for this work to yield results, I was able to find many of the packages from fepitre’s copr repo in EPEL and AppStreams and the AltArch SIG apparently already carries a backported kernel.  I think it's worth exploring collaboration through the CentOS Special Interest Groups [sigs] and/or a custom [Fedora]/[CentOS]/[Silverblue] distro to expand the package availability and upstream maintenance. 

Sane people disagree, see [alt-thread] for caveats and spirited debate.  I hope this was helpful.

-Zach Lym

[alt-thread]: https://qubes-os.discourse.group/t/alt-rpm-distro-centos-in-dom0/2016/19

[hyper-sig]: https://wiki.centos.org/SpecialInterestGroup/Hyperscale

[sigs]: https://wiki.centos.org/SpecialInterestGroup

[Fedora]: https://fedoraproject.org/wiki/Spins_Custom

[CentOS]: https://centos.org/variants/#making-variants

[Silverblue]: https://discussion.fedoraproject.org/t/minimal-or-custom-silverblue-ostree-images/1574/20

[Demi Marie Obenour]

On 3/26/21 8:25 PM, Zach Lym wrote:
> I was asked by deeplow to forward/summarize info from the "Alt RPM Distro
> (CentOS?) in dom0" forum thread [alt-thread] to make it more visible to
> devs.

Thank you!

> CentOS Stream is an attractive candidate for use in dom0 due to its
> stability, LTS cycle, compatibility with Fedora, and signing of both
> packages and metadata. The main concerns raised about CentOS were outdated
> kernels, the limited set of available userland packages, the stability of
> CentOS Stream, and the desire for a "minimal" distro. Recent developments
> largely address these concerns:

The most important problem is not technical, but rather one of upstream
priorities. Red Hat focuses on paying customers first and foremost.
And QubesOS users are not paying customers.

Most importantly, Red Hat’s security team focuses on the impact of a
vulnerability on RHEL. Red Hat Product Security issues detailed security
advisories for RHEL, but there are no security advisories for Fedora or CentOS.
A serious vulnerability in librepo, which allowed for remote code execution,
took nearly two months to fix in Fedora. Since librepo is used by DNF to
download packages, users of the Fedora templates were affected.

Similarly, the installer (anaconda) is able to handle complex use-cases,
such as iSCSI and Fibre Channel storage. Yet it is often complained about by
those with simpler setups. For instance, it didn’t handle multiple encrypted
drives properly until Fedora 29!

> * RHEL has moved to a 3-year cycle for major releases and there is a
> Hyperscaler SIG [hyper-sig] working on accelerated backports; including LTS
> kernel releases and core userland software such as systemd. This is inline
> with Qubes' ~2±1 year release cadence.

This is definitely good news. Nevertheless, it does not solve the underlying
problem: that non-paying customers are second-class citizens in the Red Hat
ecosystem. Upstream feature development is focused on paying enterprise users
and is unresponsive to the needs of Fedora users, such as us.

> * CentOS Stream is part of Red Hat's infrastructure modernization effort to
> create a continuous integration pipeline from Fedora to CentOS and its
> derivatives. This effort *should* reduce the friction for getting packages
> available in Fedora to EPEL.
>
> * CentOS Stream is intended to be as stable as RHEL proper. Source and
> community rebuilds are available for the skiddish, however, Red Hat has
> expanded its free license tier to 16 production instances. Qubes also
> probably qualifies for Red Hat's FOSS infrastructure program.

QubesOS is very much self-hosted: the critical infrastructure of QubesOS itself
runs on QubesOS. So I am not sure how much using RHEL would help. I actually
expect that a RHEL qube would be significantly less stable than a CentOS Stream
qube, as it will have very few users and thus much less testing.

Also, CentOS Stream is a “upstream development platform” [1]. While (at
least according to Red Hat) it is sufficiently stable for production use, it
may still change frequently and without warning. This is fine for users who do
not need the ultimate in stability, or who can do their own QA. It is *not*
fine for the QubesOS dom0. The QubesOS dom0 needs to be a stable platform
for the rest of the OS, not a place for experimentation. There is a reason
that dom0 stays on a single Fedora version for an entire QubesOS release: we
(the QubesOS developers) do not want to be building on a moving foundation.
And paying for Red Hat Enterprise Linux simply is not an option.

In particular, dom0 is *not* a stock Fedora install. It would not be a stock
CentOS install either. There are *extremely* subtle interactions between our
customizations and what is provided by upstream, and some of those interactions
(such as udev rules) are security-critical. Major changes in dom0 without us
knowing is a Bad Thing. And we do not have the resources to vet each and every
update.

[1]: https://www.redhat.com/en/blog/transforming-development-experience-within-centos

> * Fedora Silverblue, Fedora CoreOS, Fedora Minimal, and Red Hat's Universal
> Base Image are all freely available and competitive with other "minimal"
> distros in terms of TCB and feature set. Rocky Linux confirmed over Matrix
> that RHCOS rebuilds are in the works.

Personally, I would be quite uncomfortable with rebuilds. This is less because
of security patch delays and more because I want something with serious effort
put into securing its infrastructure. CentOS Stream certainly qualifies, as
does the other suggestion I had for dom0: openSUSE Leap.

openSUSE Leap is a much better choice for dom0 than *any* Red Hat-based distro.
openSUSE Leap uses the same binary packages as SUSE Enterprise Linux, and
receives security fixes at the same time. No Red Hat derivative offers this
guarantee. Combined with openSUSE Leap’s release cadence, this means that we
can use a single, *supported* version of openSUSE for an entire QubesOS release
if the release schedules line up. And if they do not, we will need to upgrade
dom0 at most once.

Furthermore, openSUSE users are first-class citizens in the ecosystem.
SUSE backports fixes for packages in openSUSE Leap even if they are not part
of SUSE Enterprise Linux. Feature requests by openSUSE users that have
sufficient justification result in internal support tickets being filed.
And not only does openSUSE sign its metadata, openSUSE’s package manager
(Zypper) *requires* signed metadata by default. Only with user consent will it
fall back to trusting package signatures.

> While it will take time for this work to yield results, I was able to find
> many of the packages from fepitre’s copr repo in EPEL and AppStreams and
> the AltArch SIG apparently already carries a backported kernel. I think
> it's worth exploring collaboration through the CentOS Special Interest
> Groups [sigs] and/or a custom [Fedora]/[CentOS]/[Silverblue] distro to
> expand the package availability and upstream maintenance.

A backported kernel would very much help, but it isn’t enough. For obvious
security reasons, we cannot use third-party repositories unless we have
complete trust in their maintainers. The Silverblue approach of an immutable
base image and atomic updates is a fantastic fit for our dom0, but those two
items alone are not enough to compensate for the disadvantages.

Overall, I strongly recommend that we move away from Fedora and towards
openSUSE Leap as quickly as possible. While R4.1 is too far along to make
such a change in dom0, offering an openSUSE template for it should be possible.
And we can switch dom0 in R4.2.
--
Demi Marie Obenour
she/her/hers
QubesOS Developer, Invisible Things Lab

[JTY]

It's good to hear that you are considering SUSE for base platform.

We have already made our own Qubes system based on OpenSUSE 15.3 for both dom0 and templateVM.

Main reason for this was that SUSE has best support for security related issues (it's also possible to get enterprise-class support directly from SUSE.) and our systems are already using AutoYaST2 very heavily.

We are ready to contribute if Qubes decides to move towards SUSE.

[Frédéric Pierret]

Hi,

Le 1/13/22 à 11:37, JTY a écrit :

> It's good to hear that you are considering SUSE for base platform.
>
> We have already made our own Qubes system based on OpenSUSE 15.3 for both dom0 and templateVM.
> Main reason for this was that SUSE has best support for security related issues (it's also possible to get enterprise-class support directly from SUSE.) and our systems are already using AutoYaST2 very heavily.
> We are ready to contribute if Qubes decides to move towards SUSE.
>
>

There are WIP PRs for openSUSE template for R4.1 that deserves to be finished (see https://github.com/QubesOS/qubes-issues/issues/6567). Feel free to reuse/superseed them to make it available for Qubes R4.1.

On the dom0 side, I'm not sure if we are going to switch to another dom0 right now. I let Marek give his impression on that.

Best regards,
Frédéric