FreeBSD Jails for QubesOS isolation

[Keno Goertz]

Hello everyone,

I'm writing because I'm interested to work on FreeBSD jails as an
alternative to Xen for isolation in QubesOS. And I have some questions
about it, that someone on this list can maybe find the time to answer.

I recently learned about FreeBSD's jails, which provide isolation
similar to a chroot environment, but with proper virtualization of the
file system, the set of users and the networking subsystem.

https://docs.freebsd.org/en/books/handbook/jails/

I've also read about Joanna's vision of Qubes air here.

https://www.qubes-os.org/news/2018/01/22/qubes-air/

Joanna argues that using Xen to achieve the separation is not really at
the essence of Qubes, and she shows how the cloud or seperate raspberry
Pis could also be used for this purpose in a future version of Qubes.

Along the same train of thought, it should be possible to build on top
of FreeBSD's jails, right? This is something that I would have interest
in doing some work on, perhaps as part of my master thesis. My main
motivation is to have a version of QubesOS with lower overhead on the
virtualization technology. I'm aware that there may be differences in
the protection offered for e.g. attacks on speculative execution between
using Xen or FreeBSD's jails for the isolation. So I don't think one
technology would make the other useless, but rather that they would be
two options to choose from depending on the threat model.

My question is: Is it actually possible to start building this by
implementing the things described under "Under the hood: qubes'
interfaces" in the "Qubes Air" blogpost for FreeBSD jails? Or is there
something missing from the side of QubesOS that first needs some work?
Or do you think this is a stupid idea to begin with? :P

Best wishes
Keno

[Demi Marie Obenour]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Fri, Mar 03, 2023 at 01:41:56PM +0100, Keno Goertz wrote:
> Hello everyone,
>
> I'm writing because I'm interested to work on FreeBSD jails as an
> alternative to Xen for isolation in QubesOS. And I have some questions
> about it, that someone on this list can maybe find the time to answer.
>
> I recently learned about FreeBSD's jails, which provide isolation
> similar to a chroot environment, but with proper virtualization of the
> file system, the set of users and the networking subsystem.
>
> https://docs.freebsd.org/en/books/handbook/jails/
>
> I've also read about Joanna's vision of Qubes air here.
>
> https://www.qubes-os.org/news/2018/01/22/qubes-air/
>
> Joanna argues that using Xen to achieve the separation is not really at
> the essence of Qubes, and she shows how the cloud or seperate raspberry
> Pis could also be used for this purpose in a future version of Qubes.

Indeed it is not.

> Along the same train of thought, it should be possible to build on top
> of FreeBSD's jails, right?

It absolutely should be.

> This is something that I would have interest
> in doing some work on, perhaps as part of my master thesis. My main
> motivation is to have a version of QubesOS with lower overhead on the
> virtualization technology. I'm aware that there may be differences in
> the protection offered for e.g. attacks on speculative execution between
> using Xen or FreeBSD's jails for the isolation.

There are two major limitations:

1. FreeBSD is a monolithic kernel, so jails do nothing to sandbox device
drivers or the Wi-Fi or USB protocol stacks. One can use PCI
pass-through with bhyve or (you guessed it) Xen, though.

2. The not-very-hardened FreeBSD kernel has much more attack surface
than the Xen hypervisor. To somewhat mitigate this, I recommend:

- Considering HardenedBSD vs FreeBSD.

- Using a non-vnet jail inside a vnet jail, with the non-vnet jail
not allowed to create other jails. This avoids some of the nasty
problems with non-vnet jails without allowing the code in the jail
to reconfigure the jail’s own networking stack.

- Using strict devfs rulesets that only allow access to a bare
minimum of devices.

- Denying as many privileges from the jail’s root as is practical.

- Using a kernel compiled without e.g. COMPAT_FREEBSD32.

- Disabling support for protocols such as SCTP.

> So I don't think one
> technology would make the other useless, but rather that they would be
> two options to choose from depending on the threat model.

It could also be useful for testing purposes. FreeBSD has sufficient
Xen support to allow one to port the existing tools for Xen-based Qubes
to it. Therefore, this would allow one to run jail-based Qubes in
Xen-based Qubes, with much lower overhead than nested virtualization
(which Xen doesn’t support anyway). Right now, if one wants run the
Qubes integration tests, one must use a seperate test machine, which is
rather annoying. A jail-based Qubes would allow one to run them from
within a qube on the machine one is working on.

BTW, FreeBSD can run under Xen as either a dom0 or domU. A good first
step might be to implement a FreeBSD template for Xen-based Qubes. That
would both be useful in its own right and be a good way to get familiar
with the Qubes codebase.

> My question is: Is it actually possible to start building this by
> implementing the things described under "Under the hood: qubes'
> interfaces" in the "Qubes Air" blogpost for FreeBSD jails? Or is there
> something missing from the side of QubesOS that first needs some work?
> Or do you think this is a stupid idea to begin with? :P

It’s not a stupid idea at all! It should definitely be possible, but
you might run into various parts of Qubes OS that assume Xen+Linux even
though they do not need to. High-quality patches to fix those are
welcome.

I am also CCing Mariusz Zaborski (aka oshogbo) who is a FreeBSD
developer himself and knows way more about FreeBSD than I probably ever
will.
- --
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEdodNnxM2uiJZBxxxsoi1X/+cIsEFAmQCoIUACgkQsoi1X/+c
IsFC6g//fW7mvNqN1dAwWj8stluV2UAy6hmUHc246fbqtugtsACWZVe8ha8fDPmm
h5TiMQI23YzbgE0S/Y7jieLCWks+PF9hY+QJdceNSAFxlNeDXjokWsfRvf3PkcQD
HOp3wrQsklx4mHjiSMf/HkLypSKdeOAVA9nqECC8xZWvCiBGdkGeBXFpWy/R08WY
ChhCK6FmlK7fE93w009p/hxdcwJfFMK6vLIwpywsz/PEo+ifrujzt/LB50kHIhWW
8VWqhNisLi9uJzCEkK5XhO26QlLQlJqokz5+dxFz79mpNLUu8WjFp+uaF1femDAq
jKtl+eZZIjOLyJg8m9nn81yYWqUl6sdCa9Qeo5uEm/zwBrRhQ0RF3PnFQ6Rb7Tao
K/tZpQloikWxZdIsVBDtXswqD00LTB0MKFP5rRQWXgyfDRnJLpeuIxa5kwhaWAe6
dJGaYokVN+CFKO4eamjv9ccHJ0slWzqqdpM4M7eqamvs7+85EWlmGyv5ImL/mnV5
qmssnkFZF/Pn2Xo1kGY2PsewIltqLWW2ZenyOsPKHorcUt2lNGjL/hsXixRX0z7S
KT1VnUXjm/+GTC3l2/l96ZXWMT+6QaTwbynBKtA8OMMILtNd7gy43to4wmSvHtRQ
eMfxlkRqL91OO14UibFWsWAUY3EJx6Pqz+/ocBxT6J9DZ4ZNrG8=
=6NxH
-----END PGP SIGNATURE-----

[Keno Goertz]

Hello Demi,

thank you a lot for the very detailed and competent response!

Demi Marie Obenour writes:
> BTW, FreeBSD can run under Xen as either a dom0 or domU. A good first
> step might be to implement a FreeBSD template for Xen-based Qubes. That
> would both be useful in its own right and be a good way to get familiar
> with the Qubes codebase.

Yes, that seems like a good idea. Also to get a sense of how realistic
it is for me to spend all the necessary effort on what would be needed
for a FreeBSD based Qubes. A FreeBSD template would indeed be very
useful either way.

If/when I make progress on that, I will share it here. Thanks again for
the help :)

Best wishes
Keno

[Joe]

On 3/4/23 02:36, Demi Marie Obenour wrote:
> On Fri, Mar 03, 2023 at 01:41:56PM +0100, Keno Goertz wrote:
>> I'm writing because I'm interested to work on FreeBSD jails as an
>> alternative to Xen for isolation in QubesOS. And I have some questions
>> about it, that someone on this list can maybe find the time to answer.
>

> There are two major limitations:
>
> 1. FreeBSD is a monolithic kernel, so jails do nothing to sandbox device
> drivers or the Wi-Fi or USB protocol stacks. One can use PCI
> pass-through with bhyve or (you guessed it) Xen, though.
>

Side note: Linux has come a long way since `chroot` was the only
"freebsd jail"/"solaris zone"/"AIX WPAR"-like mechanism. These days
Linux "namespaces" let you do the same, if not more, on Linux.

I wouldn't recommend spending much time on FreeBSD-as-Xen-Dom0:
The times I've worked with FreeBSD + Xen (including porting some of the
qubes guest tooling to FreeBSD for better FBSD appvm support) it has not
been *great*. It kind of sort of works, but there is a lack of
documentation and the relevant packages are often outdated or compiled
with desirable features disabled.

The bhyve hypervisor would probably be a more natural target for such an
effort, since it would allow running both windows/linux/bsd guests (like
Xen does) as BHyve VMs, with "directed-IO"/"IOMMU" for peripherals etc.
The first and potentially biggest obstacle is th at the Qubes RPC model
is fairly married to two primitives:
- message passing between VMs (in Xen: the vchan ring-buffer mechanism,
in unix: "unix sockets" / "mqueue")
- used for executing commands, setting time,
file transfer, clipboard copy/paste support, etc
- memory sharing between VMs (in Xen: "grants", in unix terms: "shm"[ipc])
- afaik mostly used for the graphical protocol where it's shared
with dom0

Your first step in adding support for a new backend would be to figure
out how to provide similar interfaces to the guests.
The message passing could conceivably use IP communication (much larger
attack surface than vchan, but easy to get started with).
With that implemented in host+guests, you could have a non-gr aphical
qubes system.

I don't know of other users of memory sharing than the graphical
protocol, where it shares rectangular screen updates in 24-bit RGB
padded to a 32-bit word (per pixel). The message passing interface is
used to inform dom0 of the bounds and offsets. So for a full screen
update on full hd screen you need to transfer ~2 million x 32bit = 8
megabytes. This is not practical over a message passing interface, but
you could do it that way in the beginning for a PoC. Eventually you
could hack something up where you used e.g. a small `md` block device in
dom0 mapped to the guest via bhyve for this part.

Joe