Eventual Xen alternatives

[spta...@gmail.com]

I've read with some interest the discussion of Xen security downsides and the Qubes architecture requirements for hypervisors. With the upcoming move to HVM (ditching PV), how might this impact the viability of KVM as a hypervisor for Qubes?

Specifically, some of the main reasoning for preferring Xen to KVM were:
- Xen  hypervisor is  very small  compared to Linux  kernel
- Xen allows to move most of the “world-facing” code out of Dom0, including the I/O  emulator, networking code and many  drivers
- Xen supports driver  domains, KVM doesn't
- Xen has better isolation for the I/O emulator process, via two mechanisms:
  (1) PV domains just don't use it, so no IOemu at all
  (2) for each HVM, per-domain virtualized-isolated slices of IOemu

cf. https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf

As I understand it, driver domains and IOemu domains are related, but separate things. Part of Xen's IOemu-domain advantage seems to be disappearing with the move to HVM. (Is this correct or mistaken?)

It seems one major residing problem with KVM is the Linux kernel (which is large and vulnerable). A port of KVM to a thinner base layer would obviate those issues. In particular, the development of the Redox microkernel seems interesting because the community is energetic and it's written in the hot new language, Rust. It's far too early to tell whether it will be feasible, but an eventual port of KVM to Redox might be a good candidate for replacing Xen in Qubes (you get a smaller TCB + the benefits of Rust over C). At clownishly rough estimations, Redox+KVM would be (20 to 30k) + (40 to 100k) = 60 to 130k LOC, which is significantly smaller than Xen (240-300k).

Of course other microkernels are also of interest (there's been discussion on this list of Muen SK) and hopefully tooling and modularization for KVM porting could be reusable over different microkernels.

Open question: replacing Linux with e.g. Redox, what KVM downsides (vs Xen, for Qubes) would remain?

A couple specific questions that I don't know the answer to:
- Where does KVM stand today on non-privileged driver domains?
- Can KVM isolate IOemu into per-domain virtualized-isolated slices?

Recent research such as DeHype and HyperLock have enabled splitting KVM into per-VM slices, but both seem to leave QEMU untouched and monolithic. (Not good.) See https://www.internetsociety.org/sites/default/files/ndss2017_02A-4_Shi_paper.pdf (p3, fig 1)

Additionally here is my feeble attempt at answering the "Eight Questions" for KVM:

1) Has HVMs and some PV support. Needs better PV support.
2) No, unmodified usermode binaries.
3) Runs unmodified drivers. Also special paravirt drivers.
4) Yes, supports VT-d. Untrusted driver domains? Dunno, maybe not.
5) S3 sleep? Yes.
6) Yes, works on multiple chipsets.
7) Similar to Xen on most benchmarks?
8) Other special features...? Dunno. Eliminates cooperative covert channels between VMs? Dunno.

The 8 Questions: https://www.qubes-os.org/doc/user-faq/#why-does-qubes-use-xen-instead-of-kvm-or-some-other-hypervisor

Some other links of reference:

https://groups.google.com/forum/#!topic/qubes-devel/uVYowLAXd0M
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-024-2016.txt

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Jul 30, 2017 at 01:28:17PM -0700, spta...@gmail.com wrote:
> I've read with some interest the discussion of Xen security downsides and
> the Qubes architecture requirements for hypervisors. With the upcoming move
> to HVM (ditching PV), how might this impact the viability of KVM as a
> hypervisor for Qubes?
>
> Specifically, some of the main reasoning for preferring Xen to KVM were:
> - Xen hypervisor is very small compared to Linux kernel
> - Xen allows to move most of the “world-facing” code out of Dom0, including
> the I/O emulator, networking code and many drivers
> - Xen supports driver domains, KVM doesn't
> - Xen has better isolation for the I/O emulator process, via two mechanisms:
> (1) PV domains just don't use it, so no IOemu at all
> (2) for each HVM, per-domain virtualized-isolated slices of IOemu
>
> cf.
> https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf
>
> As I understand it, driver domains and IOemu domains are related, but
> separate things. Part of Xen's IOemu-domain advantage seems to be
> disappearing with the move to HVM. (Is this correct or mistaken?)
>
> It seems one major residing problem with KVM is the Linux kernel (which is
> large and vulnerable). A port of KVM to a thinner base layer would obviate
> those issues.

Yes, that's right.

> In particular, the development of the Redox microkernel seems
> interesting because the community is energetic and it's written in the hot
> new language, Rust. It's far too early to tell whether it will be feasible,
> but an eventual port of KVM to Redox might be a good candidate for
> replacing Xen in Qubes (you get a smaller TCB + the benefits of Rust over
> C). At clownishly rough estimations, Redox+KVM would be (20 to 30k) + (40
> to 100k) = 60 to 130k LOC, which is significantly smaller than Xen
> (240-300k).
>
> Of course other microkernels are also of interest (there's been discussion
> on this list of Muen SK) and hopefully tooling and modularization for KVM
> porting could be reusable over different microkernels.
>
> Open question: replacing Linux with e.g. Redox, what KVM downsides (vs Xen,
> for Qubes) would remain?
>
> A couple specific questions that I don't know the answer to:
> - Where does KVM stand today on non-privileged driver domains?

Last time I've checked it wasn't possible to have for example network
connection between two domains without either heavy host kernel usage
(including ethernet parsing, switching etc), or significant performance
impact (at least two memory copy of all the data). Or disk backend
outside of the one qemu instance. But that was some time ago, maybe
something have changed?

> - Can KVM isolate IOemu into per-domain virtualized-isolated slices?
>
> Recent research such as DeHype and HyperLock have enabled splitting KVM
> into per-VM slices, but both seem to leave QEMU untouched and monolithic.
> (Not good.) See
> https://www.internetsociety.org/sites/default/files/ndss2017_02A-4_Shi_paper.pdf
> (p3, fig 1)

As long as compromising such monolithic QEMU does not ease compromising
other domains (like other QEMU processes, host kernel etc), it shouldn't
be a big problem from security POV. On the other hand, if selected
device emulators (and/or virtio backends) could live in different qemu
instances (or even not qemu), in different domains, that would help with
the first point.

> Additionally here is my feeble attempt at answering the "Eight Questions"
> for KVM:
>
> 1) Has HVMs and some PV support. Needs better PV support.
> 2) No, unmodified usermode binaries.
> 3) Runs unmodified drivers. Also special paravirt drivers.
> 4) Yes, supports VT-d. Untrusted driver domains? Dunno, maybe not.
> 5) S3 sleep? Yes.
> 6) Yes, works on multiple chipsets.
> 7) Similar to Xen on most benchmarks?
> 8) Other special features...? Dunno. Eliminates cooperative covert channels
> between VMs? Dunno.
>
> The 8 Questions:
> https://www.qubes-os.org/doc/user-faq/#why-does-qubes-use-xen-instead-of-kvm-or-some-other-hypervisor
>
> Some other links of reference:
>
> https://groups.google.com/forum/#!topic/qubes-devel/uVYowLAXd0M
> https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-024-2016.txt
>

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZflEhAAoJENuP0xzK19csHpsH/1vJWP00I+cLrpJN4UvFotec
LUwrb+4kweodbqVau2sP06vE8c0swQ8I22PFNWB+5XUSd35hHLBkf5jo4VBocCCH
HNpZ4SuNm/58OqGz0LFs2PUq0HRMgufVTVKOSepFqr4SICvrhpLtoUP/GTp4IWtI
xZZankWPxd08VHbIn433f5rP9aTHHzDbL8uc+Udi1O3m8IR9dWBZWIMOtBMhB0/N
7ot/EjLSfMlRc0iKxwbBg20dlvJ4Yz+E64EIcafFdrNCeVWbhre3rS99FoieoURZ
3OpcIu9sO4bLsgTT+xTGe46pg8VumzSb+IjO52ez+8Xjv2tX4QYBvYcwuvcLDrw=
=9Esi
-----END PGP SIGNATURE-----

[Scott Tankard]

By the way, if anyone is interested in pursuing this topic further, I'll drop this link here where I've collected some resources and brainstorming on porting KVM to Redox:

https://github.com/KVMarx/marx-planning

[TimW]

what would KVM be bringing to the table in terms of benefits compared to Xen. I see a list of what Xen does well (why it was initially chosen) and what KVM still can not do or do as well as Xen. But what benefits does KVM bring that Xen does not? As the opening topic was security how does KVM stack up in terms of its history and prowess in security compared to Xen

[Scott Tankard]

Le jeudi 8 février 2018 22:27:58 UTC-8, TimW a écrit :
> what would KVM be bringing to the table in terms of benefits compared to Xen. I see a list of what Xen does well (why it was initially chosen) and what KVM still can not do or do as well as Xen. But what benefits does KVM bring that Xen does not? As the opening topic was security how does KVM stack up in terms of its history and prowess in security compared to Xen

So the actual replacement that I have had in mind, is Redox+KVM. (Or another microkernel similar to Redox.) I have been ignoring Linux+KVM, as Linux's security record is (compared to Xen) terrible.

The perceived security benefits of Redox+KVM over Xen would be:
- dramatically smaller overall TCB
- better technical design (more modularity, isolation, better programming languages, etc)
- better organizational prioritization of security

This is all very hypothetical (especially the third point), as Redox is not yet a mature/stable project.

Aside: I noted that many of the research papers on hardening Xen basically reimplement microkernel design considerations. (They do things like segment Xen internally, bringing it closer to a microkernel design.)

There are basically very few "industrial-grade" hypervisors (that have a large enough user base, developer base, and socioeconomic pressure and resources to maintain them secure and featureful...). There's VMWare, Xen and KVM. You could maybe add Hyper-V and Bhyve. VMWare and Hyper-V aren't libre software, so they're out. That leaves Xen, KVM, and Bhyve.

One thing I do not know (but that is important to look at) is the security history/track record of KVM vs Xen with regard to CVEs, etc. (Bearing in mind that Linux kernel should be removed from the equation.)

[Joe]

First, I'd like to say that this wmail is not meant to discourage you.
HW-assisted virtualization in Redox *would* be cool, and I hope to see
it implemented some day.

... I do have some critical points though, and I think that this
hypothetical hypervisor framework will be years in the future.

So discussion in a QubesOS context IMHO primarily makes sense to collect
ideas and feature requirements for the design.

On 04/04/2018 07:12 AM, Scott Tankard wrote:
> Le jeudi 8 février 2018 22:27:58 UTC-8, TimW a écrit :
>> what would KVM be bringing to the table in terms of benefits compared to Xen. I see a list of what Xen does well (why it was initially chosen) and what KVM still can not do or do as well as Xen. But what benefits does KVM bring that Xen does not? As the opening topic was security how does KVM stack up in terms of its history and prowess in security compared to Xen
> So the actual replacement that I have had in mind, is Redox+KVM. (Or another microkernel similar to Redox.) I have been ignoring Linux+KVM, as Linux's security record is (compared to Xen) terrible.

There's a IllumOS/SmartOS port of KVM (since 2011, so without knowing
many details I assume it's fairly mature now):
https://lwn.net/Articles/459754/

They even have package signing, so much harder to compromise than your
average Rust user. :-)

> The perceived security benefits of Redox+KVM over Xen would be:
> - dramatically smaller overall TCB

Which has seen much less review than Xen, and implemented by people with
less knowledge about x86_64 virtualization than the Xen crowd.
This will no doubt improve over the years as the Rust implementation
matures, but again, it will take some years to beat this into shape.

> - better organizational prioritization of security

This is sort of in the category "would be nice, will take a lot of work
to get there." For instance the organization to host such organizational
priorities is still just an idea, and in practice establishing a
resilient, friendly community with a focus on security has proven a
non-trivial problem to solve.

I think the QubesOS, the HardenedBSD, and the Genode communities are
worth looking at, since they have somehow managed to establish this in a
not-too-corporate setting. I don't know how the Rust/Redox camps are
doing in this regard, but I hope you have a nice community, too! :-)

Linux/KVM, on the other hand, seems kind of toxic to me (that is my
subjective impression as a passive observer), and the Xen thing seems to
have a quite corporate focus.

> This is all very hypothetical (especially the third point), as Redox is not yet a mature/stable project.
>
> Aside: I noted that many of the research papers on hardening Xen basically reimplement microkernel design considerations. (They do things like segment Xen internally, bringing it closer to a microkernel design.)
>
> There are basically very few "industrial-grade" hypervisors (that have a large enough user base, developer base, and socioeconomic pressure and resources to maintain them secure and featureful...).
> There's VMWare, Xen and KVM. You could maybe add Hyper-V and Bhyve.
VMWare and Hyper-V aren't libre software, so they're out. That leaves
Xen, KVM, and Bhyve.

I'd say Muen SK is more promising than the hypothetical changes to Redox
- the Muen SK people already have running code; boots on the ground, and
the security guarantees of Ada/SPARK are *much* stronger than what Rust
provides. I'm happy to discuss this off-list if you want to go over
details, but I fear we'd stray off-topic here on the QubesOS list.

It's also worth taking a look at Genode and their list of supported targets:
https://genode.org/documentation/platforms/index

I take it you mean for x86_64, since your list is missing stuff like
AIX/POWER which introduced the concepted of IOMMU to the PC (ported from
IBM's z/OS line of operating systems that introduced the first complete
package sof hw-assisted hypervisors in 1968).
From slightly more exotic hw (these days) there's also the old and
stable Solaris LDOM:
https://en.wikipedia.org/wiki/Oracle_VM_Server_for_SPARC

Joe

[viq]

On Wed, 4 Apr 2018, 07:12 Scott Tankard, <spta...@gmail.com> wrote:

There are basically very few "industrial-grade" hypervisors (that have a large enough user base, developer base, and socioeconomic pressure and resources to maintain them secure and featureful...). There's VMWare, Xen and KVM. You could maybe add Hyper-V and Bhyve. VMWare and Hyper-V aren't libre software, so they're out. That leaves Xen, KVM, and Bhyve.

What about OpenBSD's VMM?

One thing I do not know (but that is important to look at) is the security history/track record of KVM vs Xen with regard to CVEs, etc. (Bearing in mind that Linux kernel should be removed from the equation.)

Especially with the above ;)

[Scott Tankard]

Le mercredi 4 avril 2018 13:36:48 UTC-7, vic viq a écrit :
> On Wed, 4 Apr 2018, 07:12 Scott Tankard, <spta...@gmail.com> wrote:
>
>
>
>
> There are basically very few "industrial-grade" hypervisors (that have a large enough user base, developer base, and socioeconomic pressure and resources to maintain them secure and featureful...). There's VMWare, Xen and KVM. You could maybe add Hyper-V and Bhyve. VMWare and Hyper-V aren't libre software, so they're out. That leaves Xen, KVM, and Bhyve.
>
>
>
> What about OpenBSD's VMM?

That would be very interesting to look into. I looked into Bhyve (FreeBSD) a little bit, but not OpenBSD's VMM very much at all yet.

[Scott Tankard]

Hi Joe,

You make a lot of good points that I agree with.

You mention KVM on Illumos/SmartOS, Genode, Muen SK, among other things. Here's a big list of unsorted resources I collected on those topics and others:
https://github.com/KVMarx/marx-planning/blob/master/resources.md

Re: "primarily makes sense to collect ideas and feature requirements for the design" -- yes, absolutely agree.

Re: just focusing on x86_64, yes, i was -- because that's where Qubes' focus is.

Re: HardenedBSD -- that's something i haven't looked into much at all yet. I'll make note.

Re: Muen SK...

MuenSK+Genode+KVM i think is very interesting and I'd be happy to see it happen. I think Redox+Genode+KVM would have some advantages; Muen would also have some advantages. (The main advantage of Rust is not that it's perfect, it's that it is very *decent*, and lets you continue to do complex things quickly. Ada/SPARK provide stronger security guarantees than Rust, however as understand it, the formal methods come at a cost of longer development times.)

Re: IllumOS KVM...

The IllumOS/SmartOS port of KVM is one of the main points of research I was focusing on. Back then, I wasn't able to find much very info on it and it seemed to be unmaintained. I wasn't sure whether Joyent used it in production and considered it "stable", or whether they had abandoned the idea of using it altogether.

However, looking at the repo now, it looks like it's woken up again!
https://github.com/joyent/illumos-kvm/commits/master

Besides that repo, this is the best doc i have found on it:
https://www.linux-kvm.org/images/7/71/2011-forum-porting-to-smartos.pdf

As an aside, also interesting are crosvm and Solo5/ukvm unikernel, see here: https://github.com/KVMarx/marx-planning/issues/1

Re: Going forward...

Today I came across a new project that looks very promising in its own right, and looks like it might be relevant to this discussion, particularly considering the new Qubes admin stack and the "Qubes Air" direction: Kata Containers ("The speed of containers, the security of VMs. Kata Containers is a new open source project building extremely lightweight virtual machines that seamlessly plug into the containers ecosystem.")

But i've yet to look into it in detail, so i'm not sure yet if it's worth considering.

https://katacontainers.io
https://katacontainers.io/media/uploads/katacontainers/uploads/katacontainers/kata_containers_overview.pdf

Besides Kata, i'd say definitely Genode stands out as a promising research direction. {MuenSK,Redox,...}+Genode+{KVM,Kata,...} would seem to be something workable.

I'd definitely be interested in chatting more off the mailing list, so we don't spam people too much here.