WIP: Qubes on KVM

[Jason M]

I have begun work on porting Qubes to work within a KVM host. I need a development environment that can utilize the cuda cores on a secondary Nvidia RTX GPU and also prefer to be able to utilize the graphics card. 

For several weeks I attempted to get the GPU to successfully pass-though to a Qubes virtual machine using the Nvidia drivers without success.  I can get it to work in dom0, but how could one think to do any work there.

I looked into KVM and discovered I can pass-though the GPU with no issues, it just works.

So now I am in a dilemma.  I love Qubes, I have been using Qubes since 2014 and was having a hard time coming to terms with having to move to a plain KVM environment.  I have come to the realization that moving to KVM is the only option now.  It was time to say goodbye to Qubes and was having a difficult time dealing with that, so I again spent way too much time trying to get the Nvidia GPU working in Qubes. 

I then looked into alternatives to prevent my complete departure from Qubes.  Marek told me about DomB, which is now in its design stages.  It would allow me to statically partition my machine (like having 2 dom0 VMs - remember the Nvidia GPU with nvidia drivers works in dom0), but there is no experimental code ready yet.  So, then I attempted to get Qubes to run nested within KVM.  I was having issues with the display and then decided that instead of running Qubes nested, I should just get Qubes to run within KVM.  That's where we are today, no need to leave Qubes, invite Qubes over to KVM!

GOALS

The final goals would be to support all Qubes features and apps.

STAGE 1

The initial goal is to get Qubes to be able to manage the virtual machines (start, stop, etc) using 'qvm-*' tools and Qubes Manager.  Seamless VM video or audio will not be implemented in stage 1 so either a GPU will need to be passed through to the VM (which will also be able to provide HDMI audio), or access using spice or vnc.  Stage 1 goals include the following:

• Use same template system Qubes currently uses including settings like qvm-prefs, features, tags, etc.
• Obviously support PCI pass-through using Nvidia drivers for RTX GPU
• Support qrexec communication from host <-> vm
  
• Locking down KVM host
  
• Securing the network - look into ability to enabling sys-net and sys-firewall

FUTURE

• Seamless windows
  
• Audio
• Encrypted memory within each VM (AMD processors)
  

BUILD STATUS

I have modified where necessary all Qubes source repos to allow building for KVM within a Fedora-32 host and guest.  All build modifications used conditional tests based on the 'BACKEND_VMM' build configuration option which is set to 'kvm'.  When 'BACKEND_VMM' is set to 'xen', everything builds as normal.

• vmm-xen:  I still include this package to allow booting into KVM or XEN.  There is also one dependency on it I need to remove.
• core-libvirt:  Configured to also compile the KVM modules and any other modules provided within the Fedora 32 distribution packages.
• core-vchan-xen:  Not required.  Components that require it use the 'BACKEND_VMM' build variable.  Nice forward thinking from the Qubes developers!
• core-vchan-libkvmchan: Packaged libkvmchan code based on the work completed by @shawnanastasio.
• qubes-core-vchan-kvm: Packaged qubes-core-vchan-kvm code based on the work completed by @shawnanastasio.
• linux-utils: Removed qmemman for KVM build.  Not sure is they can be adapted for KVM.  Will revisit KVM alternative later.
• core-admin:
• Added KVM libvirt template
• Added additional conditional 'BACKEND_VMM' for Xen specific build and depends
• Still installs qubes-qmemman.service until files.  Not sure if they can be adapted to KVM.
• qubes python
• other: Minor changes here and there.

INSTALL STATUS

• dom0:
• All dom0 packages install without error (minus vmm-stubdom and iso related packages)
• All Qubes services start successfully on boot
  

• template:
• qubes-template-fedora-32 installs within kvm host.  A few manual modification were made to qubes.xml to facilitate this.

WIP

• core-admin
• qubes python package
• Added a 'hypervisor' module to detect hypervisor type (xen, kvm, etc) for cases like the following where it is expected that the hypervisor is Xen if 'xen.lowlevel' is able to be imported.  In my case the Xen module is installed since I also have a Xen boot option:
• qubes.app.VMMConnection.init_vmm_connection change. 
  
• old: if 'xen.lowlevel.{xs,xc}' in sys.modules:
• new: if hypervisor_type('xen'):
  
  
• There are a few dependencies on Xen such as:
• qubes.app.xs (xenstore)
  
  I was hoping that 'xenstore' could be used as a standalone application (without Xen being activated).  I have not yet looked at the source code but tried starting the xenstore service which failed since the '/proc/xen' directory does not exist.  Wondering a I created an procfs entry for '/proc/xen' if the store would run without Xen.
  
  If xenstore won't work without Xen then need to determine the best alternative; convert xenstore to work without Xen or some other solution?
  
  
• qubes.ext.pci.attached_devices:
  
• ls('', 'backend/pci'), ls('', 'backend/pci' + domid)
• read('', devpath + '/domain'), read('', devpath + '/num_devs'), read('', devpath + '/dev-' + str(dev))
• qubes.vm.qubesvm.stubdom_xid: if xs is None: return -1  # No issue
  
• qubes.vm.qubesvm.start_time:  read('', '/vm/{}/start_time').format(self.uuid)
• qubes.vm.qubesvm.create_qdb_entries:  set_permissions('', '/local/domain/{}/memory'.format(self.xid, [{'dom': self.xid}])
• qubes.vm.qubesvm.get_prefmem:  read('', '/local/domain/{}/memory/meminfo').format(self.xid))
• 
  
• qubes.app.xc (xen connection)
  
• qubes.app.QubesHost.get_free_xen_memory:  physinfo()['free_memory']
• qubes.app.QubesHost.is_iommu_supported:  physinfo()['virt_caps']
• qubes.app.QubesHost.get_vm_stats:  domain_getinfo(int, int)['domid', 'mem_kb', 'cpu_time', 'online_vpus']
  
  
• Added a 'hypervisor' script to '/usr/lib/qubes' for other scripts like 'startup-misc.sh'
  • if /usr/lib/qubes/hypervisor_type xen; then ...

CURRENT ISSUES TO RESOLVE

• xenstore:  Will it work without Xen?  If not convert it so it will or provide another alternative?
• qmemman:
• Provide KVM alternative
• Should components like linux-utils that provide xen only utilities have the xen utilities split into another repo like 'linux-utils-xen'?  Then when a KVM alternative can be provided it could be placed in 'linux-utils-kvm'?
• Qubes python packages:
• Not yet sure how much of it relies on any xen packages. Currently I will continue using the hypervisor check and once all python packages are functioning correctly with KVM we can look into better ways to handle xen vs kvm or other hypervisors.
• qubes-builder:
• For some reason I can not build with 'DIST_BUILD_TOOLS=1" (standard qubes xen components).  I always get an error when building dom-fc32 of "sudo: unrecognized option '--resultdir=/home/user/qubes/chroot-dom0-fc32/home/user/qubes-src/vmm-xen/pkgs/dom0-fc32'.  Am I missing another config option?
• Libvirt often fails to compile using 32 cores giving some error about some file that does not exist (when it does fail, it always fails at the same spot with same error message).  It seems to be compiling too fast or maybe has something to do with using BTRFS filesystem.  The rpm spec for libvirt uses the number of processors available (make -j32 V=1).  It will build without errors if I generate a '.rpmmacros' file containing '%_smp_mflags' -j10' to the 'chroot-dom0-fc32/home/user' directory.  Just wondering is there is a way to set number of jobs per component, or maybe switching to using 'DIST_BUILD_TOOLS' will help.
  

Comments welcome,

Jason

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Jul 31, 2020 at 02:17:05PM -0700, Jason M wrote:
> I then looked into alternatives to prevent my complete departure from
> Qubes. Marek told me about DomB, which is now in its design stages. It
> would allow me to statically partition my machine (like having 2 dom0 VMs

Not really 2 dom0 in a sense you would need. What you'd need is a 2
hardware domains (dom0 by default) and that can still be only one. One
of domB proposal goals is to make it more meaningful to have hardware
domain != dom0, by eliminating dom0.

> *GOALS*

> The final goals would be to support all Qubes features and apps.

Yes!!

> *STAGE 1*

> The initial goal is to get Qubes to be able to manage the virtual machines

> (start, stop, etc) using 'qvm-*' tools and *Qubes Manager*. Seamless VM

> video or audio will not be implemented in stage 1 so either a GPU will need
> to be passed through to the VM (which will also be able to provide HDMI
> audio), or access using spice or vnc. Stage 1 goals include the following:
>

> - Use same template system Qubes currently uses including settings like
> *qvm-prefs*, *features*, *tags*, etc.
> - Obviously support PCI pass-through using Nvidia drivers for RTX GPU
> - Support qrexec communication from host <-> vm
> - Locking down KVM host
> - Securing the network - look into ability to enabling *sys-net* and
> *sys-firewall*

One challenge with the last point is to have VM<->VM network connection
(like sys-net - sys-firewall) without exposing the host to the traffic.
Most (all?) of the traditional KVM setups assumes the host is responsible
for routing network traffic.
One idea is to use netdev socket to connect two VMs directly, but I
worry about the performance...

> *FUTURE*
>
> - Seamless windows

Yes, putting this beyond the first stage makes sense. But also it
shouldn't be that hard. Beyond vchan, you "just" need to handle
KVM-specific shared memory primitive (ivshmem device) in
gui-daemon/gui-agent.

> - Audio

Having qrexec and vchan, you will get audio for free.

> - Encrypted memory within each VM (AMD processors)
>
>
> *BUILD STATUS*

>
> I have modified where necessary all Qubes source repos to allow building
> for KVM within a Fedora-32 host and guest. All build modifications used
> conditional tests based on the 'BACKEND_VMM' build configuration option
> which is set to 'kvm'. When 'BACKEND_VMM' is set to 'xen', everything
> builds as normal.
>

> - *vmm-xen*: I still include this package to allow booting into KVM or

> XEN. There is also one dependency on it I need to remove.

> - *core-libvirt*: Configured to also compile the KVM modules and any

> other modules provided within the Fedora 32 distribution packages.

> - *core-vchan-xen*: Not required. Components that require it use the

> 'BACKEND_VMM' build variable. Nice forward thinking from the Qubes
> developers!

> - *core-vchan-libkvmchan*: Packaged *libkvmchan
> <https://github.com/shawnanastasio/libkvmchan>* code based on the work
> completed by @shawnanastasio <https://github.com/shawnanastasio>.
> - *qubes-core-vchan-kvm*: Packaged *qubes-core-vchan-kvm
> <https://github.com/shawnanastasio/qubes-core-vchan-kvm>* code based on
> the work completed by @shawnanastasio <https://github.com/shawnanastasio>
> .
> - *linux-utils*: Removed *qmemman* for KVM build. Not sure is they can

> be adapted for KVM. Will revisit KVM alternative later.

Balloon driver for KVM exists. But I don't know what interface it uses
for control. qmemman currently uses strictly Xen specific interfaces, so
disabling it for now makes sense.

One thing to consider is also enabling memory deduplication in KVM
(KSM). This should nicely save memory when running multiple similar VMs,
but at the same time is risky in light of speculative execution and also
rowhammer-style attacks.

> - *core-admin*:
> - Added KVM *libvirt* template
> - Added additional conditional 'BACKEND_VMM' for Xen specific build
> and depends
> - Still installs *qubes-qmemman.service* until files. Not sure if

> they can be adapted to KVM.

> - *qubes* python
> - *other*: Minor changes here and there.
>
>
> *INSTALL STATUS*
>
> - *dom0*:
> - All dom0 packages install without error (minus vmm-stubdom and iso
> related packages)
> - All Qubes services start successfully on boot
>
>
> - *template*:
> - qubes-template-fedora-32 installs within kvm host. A few manual

> modification were made to qubes.xml to facilitate this.

One of installation step is starting the template VM and connecting
qrexec. Do you mean that part worked already too?

> *WIP*
>
> - *core-admin*
> - *qubes python package*
> - Added a 'hypervisor' module to detect hypervisor type (xen, kvm,

> etc) for cases like the following where it is expected that the hypervisor
> is Xen if 'xen.lowlevel' is able to be imported. In my case the Xen module
> is installed since I also have a Xen boot option:

> - *qubes.app.VMMConnection.init_vmm_connection* change.
> - old: if 'xen.lowlevel.{xs,xc}' in sys.modules:
> - new: if hypervisor_type('xen'):
>
> - There are a few dependencies on Xen such as:
> -
> *qubes.app.xs (xenstore) *
> I was hoping that '*xenstore*' could be used as a standalone

> application (without Xen being activated).

Not needed. Besides qmemman, xenstore is used directly only for really
Xen specific workarounds that shouldn't be needed on KVM. You can safely
disable xenstore usage in core-admin if running on KVM.

> I have not yet looked at the

> source code but tried starting the *xenstore* service which

> failed since the '/proc/xen' directory does not exist. Wondering a I

> created an *procfs* entry for '/proc/xen' if the store would
> run without Xen.
>
> If *xenstore* won't work without X*en* then need to determine
> the best alternative; convert *xenstore* to work without Xen or
> some other solution?

As said above - you don't need that.

>
> - *qubes.ext.pci.attached_devices*:
> - ls('', 'backend/pci'), ls('', 'backend/pci' + domid)
> - read('', devpath + '/domain'), read('', devpath +

> '/num_devs'), read('', devpath + '/dev-' + str(dev))

This is a workaround for Xen toolstack limitation regarding reporting of
attached devices (extracting how device is visible inside the VM). You
can also get similar information via libvirt (and hopefully in case of
KVM it should be more accurate).

> - *qubes.vm.qubesvm.stubdom_xid*: if xs is None: return -1
> # No issue

Yes, no issue as stubdomains are not existing on KVM.

> - *qubes.vm.qubesvm.start_time*: read('',
> '/vm/{}/start_time').format(self.uuid)

I haven't found a proper libvirt API for this information. In case of
KVM, I believe you can get this by looking up qemu process id (libvirt domain
ID?) and checking its start time in /proc.

> - *qubes.vm.qubesvm.create_qdb_entries*: set_permissions('',

> '/local/domain/{}/memory'.format(self.xid, [{'dom': self.xid}])

> - *qubes.vm.qubesvm.get_prefmem*: read('',
> '/local/domain/{}/memory/meminfo').format(self.xid))

Both needed only with qmemman (and should be move to generic API, like
qubesdb if making qmemman KVM-compatible).

> -
> - *qubes.app.xc (xen connection)*
> - *qubes.app.QubesHost.get_free_xen_memory*:
> physinfo()['free_memory']

I believe it isn't used anywhere outside of qmemman. And qmemman has its
own copy of this function...

> - *qubes.app.QubesHost.is_iommu_supported*:
> physinfo()['virt_caps']

It should be possible to get it from libvirt via capabilities (see `virsh
capabilities` and `virsh domcapabilities`).

> - *qubes.app.QubesHost.get_vm_stats*: domain_getinfo(int,

> int)['domid', 'mem_kb', 'cpu_time', 'online_vpus']

There is a proper libvirt API for that (GetAllDomainStats), but it isn't
implemented in Xen driver. I hope it is in KVM.

>
> - Added a 'hypervisor' script to '/usr/lib/qubes' for other
> scripts like 'startup-misc.sh'
> - if /usr/lib/qubes/hypervisor_type xen; then ...
>
>
> *CURRENT ISSUES TO RESOLVE*
>
> - *xenstore*: Will it work without Xen? If not convert it so it will
> or provide another alternative?
> - *qmemman*:
> - Provide KVM alternative
> - Should components like linux-utils that provide xen only utilities

> have the xen utilities split into another repo like 'linux-utils-xen'?
> Then when a KVM alternative can be provided it could be placed in
> 'linux-utils-kvm'?

Ideally VM part of the qmemman could be made hypervisor-agnostic.

> - *Qubes python packages*:
> - Not yet sure how much of it relies on any xen packages. Currently I

> will continue using the hypervisor check and once all python packages are
> functioning correctly with KVM we can look into better ways to handle xen
> vs kvm or other hypervisors.

Ideally no xen-specific python modules should be required to work. There
are few Xen-specific workarounds (and I believe few KVM-specific won't
be avoidable either), but it should be easy to work just fine without
them if you are on the other hypervisor (like, using `try: import xen...;
except ImportError: ...` pattern).

> - *qubes-builder*:
> - For some reason I can not build with 'DIST_BUILD_TOOLS=1" (standard

> qubes xen components). I always get an error when building dom-fc32 of
> "sudo: unrecognized option
> '--resultdir=/home/user/qubes/chroot-dom0-fc32/home/user/qubes-src/vmm-xen/pkgs/dom0-fc32'.
> Am I missing another config option?

Missing "mock" package in your build environment?

> - Libvirt often fails to compile using 32 cores giving some error

> about some file that does not exist (when it does fail, it always fails at
> the same spot with same error message).

I guess some missing dependency in one of the Makefiles.

> It seems to be compiling too fast
> or maybe has something to do with using BTRFS filesystem. The rpm spec for
> libvirt uses the number of processors available (make -j32 V=1). It will
> build without errors if I generate a '.rpmmacros' file containing
> '%_smp_mflags' -j10' to the 'chroot-dom0-fc32/home/user' directory. Just
> wondering is there is a way to set number of jobs per component, or maybe
> switching to using 'DIST_BUILD_TOOLS' will help.

I would start with reporting this specific build issue on libvir-list.
It's worth also checking with upstream libvirt build, straight from git
clone master branch.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl8lWRUACgkQ24/THMrX
1yyeWgf9FQjfKXkTG7Rk7MQXI6g+FS+b/dWB7O7eKf8qlc3VJFe0Nfq+ov4YP7/S
i6LXv7+kD9+cgwmde+43DfuG0J3T5JOZwU4YCOL/Bp/zYTkJ+eo9sZRSraxWIdZe
wnNgHk09VnqW/VoCc7woS+5MeSk8C/ThD3ITrlIEqQssFh3WGRZUKq0DAbkSjhrL
RydrALVA9C+9LguQ8bTFpVTscj9gQN06PQQXRM/K/dMw5fI8qf3e7MpJFxl6yRKc
gWkVr7ZQA2UOzs+ENMawANlQR5pxAme2xDvx4iLfr23vDEqMJVAY4RAQP6Fb9fci
GSG8wb9Thdq2fO+Y/AAbDZ51ikb+Kg==
=t/7G
-----END PGP SIGNATURE-----

[Demi M. Obenour]

In most KVM setups that I know of, the kernel network stack is
considered trusted. That’s a reasonable assumption for production
servers, which have server-grade NICs and are behind enterprise
routers, but not for Qubes.

> One idea is to use netdev socket to connect two VMs directly, but I
> worry about the performance...

We could also reimplement the Xen netfront/netback protocols on top
of KVM shared memory. Future versions of KVM might even have direct
support for Xen paravirtualized drivers.

> One thing to consider is also enabling memory deduplication in KVM
> (KSM). This should nicely save memory when running multiple similar VMs,
> but at the same time is risky in light of speculative execution and also
> rowhammer-style attacks.

Honestly, I don’t think that deduplication is worth it, especially
in light of TRRespass. It also makes side-channel attacks far, *far*
easier to exploit.

What *might* be safe is mapping read-only data (such as dom0-provided
kernels) into multiple VMs

> Yes, no issue as stubdomains are not existing on KVM.

This could be a nasty issue for HVMs, as without stubdomains, all
emulation must be done in dom0. Google and Amazon both solved this by
writing their own VMMs. At some point, KVM might be able to move the
instruction emulation into userspace, which might be a significant win.

(What I *really* want is a version of QubesOS based on seL4.
But seL4 didn’t support 64-bit VMs last I checked, and I am not
aware of any tooling for creating and destroying VMs at run-time.
Most of the userspace tooling around seL4 is based on CAmkES, which
requires that every component be statically known at compile-time.
Furthermore, getting seL4’s hypervisor and IOMMU support verified
would either require someone to fund the project, or someone who has
sufficient skill with the Isabelle proof assistant to do it themselves.
Without verification, the benefits of using seL4 are significantly
diminished.)

Sincerely,

Demi

[Jason M]

On Saturday, August 1, 2020 at 7:59:28 a.m. UTC-4 marm...@invisiblethingslab.com wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Jul 31, 2020 at 02:17:05PM -0700, Jason M wrote:
> I then looked into alternatives to prevent my complete departure from
> Qubes. Marek told me about DomB, which is now in its design stages. It
> would allow me to statically partition my machine (like having 2 dom0 VMs

Not really 2 dom0 in a sense you would need. What you'd need is a 2
hardware domains (dom0 by default) and that can still be only one. One
of domB proposal goals is to make it more meaningful to have hardware
domain != dom0, by eliminating dom0.

Ahh, I think I understand.  I just assumed one could pass a device tree and have DomB start the other DomU's in PV mode like Dom0.  When researching DomB it appeared they would be combining the Dom0less functionality with DomB and since Dom0less seems to be able to pass a device tree to each DomU created I made the assumption that would be possible with DomB.  What I really wanted was to be able to statistic partition the computer resources where I would assign a bit of ram, network card and Nvidia GPU to one DomU and the rest to Qubes.  Will be interesting to experiment with once code become available.

 

> *GOALS*
> The final goals would be to support all Qubes features and apps.

Yes!!

> *STAGE 1*
> The initial goal is to get Qubes to be able to manage the virtual machines
> (start, stop, etc) using 'qvm-*' tools and *Qubes Manager*. Seamless VM
> video or audio will not be implemented in stage 1 so either a GPU will need
> to be passed through to the VM (which will also be able to provide HDMI
> audio), or access using spice or vnc. Stage 1 goals include the following:
>
> - Use same template system Qubes currently uses including settings like
> *qvm-prefs*, *features*, *tags*, etc.
> - Obviously support PCI pass-through using Nvidia drivers for RTX GPU
> - Support qrexec communication from host <-> vm
> - Locking down KVM host
> - Securing the network - look into ability to enabling *sys-net* and
> *sys-firewall*

One challenge with the last point is to have VM<->VM network connection
(like sys-net - sys-firewall) without exposing the host to the traffic.
Most (all?) of the traditional KVM setups assumes the host is responsible
for routing network traffic.
One idea is to use netdev socket to connect two VMs directly, but I
worry about the performance...

I have not completed any research on this yet but I will bring it up again once I start working on the networking at which point I would hope to be able to present a few alternatives for discussion.

> *FUTURE*
>
> - Seamless windows

Yes, putting this beyond the first stage makes sense. But also it
shouldn't be that hard. Beyond vchan, you "just" need to handle
KVM-specific shared memory primitive (ivshmem device) in
gui-daemon/gui-agent.

I have seen you had a discussion with @shawnanastasio in regards to this.  I should maybe add an issue in his libkvmchan repo to see if he is still interested in work on that part (and let him know I have packages his drivers and started Qubes-KVM developemnt).

> - Audio

Having qrexec and vchan, you will get audio for free.

Anything free is nice :)

Again, I have not yet researched this.  Not sure where to place it on priority list since I can just enable the virtio memballoon option within libvert xml config for now.

One thing to consider is also enabling memory deduplication in KVM
(KSM). This should nicely save memory when running multiple similar VMs,
but at the same time is risky in light of speculative execution and also
rowhammer-style attacks.

Personally I would be concerned of the risk, but I guess it could be an option for users with low memory.

 

> - *core-admin*:
> - Added KVM *libvirt* template
> - Added additional conditional 'BACKEND_VMM' for Xen specific build
> and depends
> - Still installs *qubes-qmemman.service* until files. Not sure if
> they can be adapted to KVM.
> - *qubes* python
> - *other*: Minor changes here and there.
>
>
> *INSTALL STATUS*
>
> - *dom0*:
> - All dom0 packages install without error (minus vmm-stubdom and iso
> related packages)
> - All Qubes services start successfully on boot
>
>
> - *template*:
> - qubes-template-fedora-32 installs within kvm host. A few manual
> modification were made to qubes.xml to facilitate this.

One of installation step is starting the template VM and connecting
qrexec. Do you mean that part worked already too?

No it did not.  For some reason I did not receive any qubesd errors on start-up.  After posting this information and rebooting, qubesd started but with errors.  Working on that now.

Once I can get the VM to start, I can start testing qrexec.  All Qubes services and KVM related packages have been included in the Fedora-32 template but I have not created a start-up service for the libvmchan binary for the VM.

 

> *WIP*
>
> - *core-admin*
> - *qubes python package*
> - Added a 'hypervisor' module to detect hypervisor type (xen, kvm,
> etc) for cases like the following where it is expected that the hypervisor
> is Xen if 'xen.lowlevel' is able to be imported. In my case the Xen module
> is installed since I also have a Xen boot option:
> - *qubes.app.VMMConnection.init_vmm_connection* change.
> - old: if 'xen.lowlevel.{xs,xc}' in sys.modules:
> - new: if hypervisor_type('xen'):
>
> - There are a few dependencies on Xen such as:
> -
> *qubes.app.xs (xenstore) *
> I was hoping that '*xenstore*' could be used as a standalone
> application (without Xen being activated).

Not needed. Besides qmemman, xenstore is used directly only for really
Xen specific workarounds that shouldn't be needed on KVM. You can safely
disable xenstore usage in core-admin if running on KVM.

That's a relief!

> I have not yet looked at the
> source code but tried starting the *xenstore* service which
> failed since the '/proc/xen' directory does not exist. Wondering a I
> created an *procfs* entry for '/proc/xen' if the store would
> run without Xen.
>
> If *xenstore* won't work without X*en* then need to determine
> the best alternative; convert *xenstore* to work without Xen or
> some other solution?

As said above - you don't need that.

>
> - *qubes.ext.pci.attached_devices*:
> - ls('', 'backend/pci'), ls('', 'backend/pci' + domid)
> - read('', devpath + '/domain'), read('', devpath +
> '/num_devs'), read('', devpath + '/dev-' + str(dev))

This is a workaround for Xen toolstack limitation regarding reporting of
attached devices (extracting how device is visible inside the VM). You
can also get similar information via libvirt (and hopefully in case of
KVM it should be more accurate).

Great.  I will look into it further.

> - *qubes.vm.qubesvm.stubdom_xid*: if xs is None: return -1
> # No issue

Yes, no issue as stubdomains are not existing on KVM.

> - *qubes.vm.qubesvm.start_time*: read('',
> '/vm/{}/start_time').format(self.uuid)

I haven't found a proper libvirt API for this information. In case of
KVM, I believe you can get this by looking up qemu process id (libvirt domain
ID?) and checking its start time in /proc.

Ditto.

> - *qubes.vm.qubesvm.create_qdb_entries*: set_permissions('',
> '/local/domain/{}/memory'.format(self.xid, [{'dom': self.xid}])
> - *qubes.vm.qubesvm.get_prefmem*: read('',
> '/local/domain/{}/memory/meminfo').format(self.xid))

Both needed only with qmemman (and should be move to generic API, like
qubesdb if making qmemman KVM-compatible).

Ditto.

> -
> - *qubes.app.xc (xen connection)*
> - *qubes.app.QubesHost.get_free_xen_memory*:
> physinfo()['free_memory']

I believe it isn't used anywhere outside of qmemman. And qmemman has its
own copy of this function...

> - *qubes.app.QubesHost.is_iommu_supported*:
> physinfo()['virt_caps']

It should be possible to get it from libvirt via capabilities (see `virsh
capabilities` and `virsh domcapabilities`).

Ditto.

> - *qubes.app.QubesHost.get_vm_stats*: domain_getinfo(int,
> int)['domid', 'mem_kb', 'cpu_time', 'online_vpus']

There is a proper libvirt API for that (GetAllDomainStats), but it isn't
implemented in Xen driver. I hope it is in KVM.

Lets hope!  Hopefully get this and the above sorted out over the weekend.

>
> - Added a 'hypervisor' script to '/usr/lib/qubes' for other
> scripts like 'startup-misc.sh'
> - if /usr/lib/qubes/hypervisor_type xen; then ...
>
>
> *CURRENT ISSUES TO RESOLVE*
>
> - *xenstore*: Will it work without Xen? If not convert it so it will
> or provide another alternative?
> - *qmemman*:
> - Provide KVM alternative
> - Should components like linux-utils that provide xen only utilities
> have the xen utilities split into another repo like 'linux-utils-xen'?
> Then when a KVM alternative can be provided it could be placed in
> 'linux-utils-kvm'?

Ideally VM part of the qmemman could be made hypervisor-agnostic.

I agree.  I am not sure how to do this at the moment, but once I start working on the memory management I should have a better idea and be able to discuss further.

My immediate concern was being able to completely eliminate all related qmemman code as it is spread among a few packages.  I think I temporarily left a Xen depend in place andt did not exclude the qmemman systemd unit file since another Qubes unit depends on it.  Guess we can revisit this once I start working looking it KVM memory management further.

 

> - *Qubes python packages*:
> - Not yet sure how much of it relies on any xen packages. Currently I
> will continue using the hypervisor check and once all python packages are
> functioning correctly with KVM we can look into better ways to handle xen
> vs kvm or other hypervisors.

Ideally no xen-specific python modules should be required to work. There
are few Xen-specific workarounds (and I believe few KVM-specific won't
be avoidable either), but it should be easy to work just fine without
them if you are on the other hypervisor (like, using `try: import xen...;
except ImportError: ...` pattern).

I added a simple hypervisor detection module since you can't rely on import failing for xen when Xen is also installed.  I guess it depends on how Qubes-KVM will be packaged.

I was thinking of 2 different packaging options:

1. Installing Qubes packages within an existing installation (dnf/apt install qubes-kvm) by either adding the Qubes repo or maybe get the packages included within Fedora/RpmFusion, Debian, Ubuntu, etc.  With that in mind I deliberately installed the Fedora virtualization group packages and Xen.
2. Generating an Qubes-KVM ISO image to be distributed in same manner as Qubes.

Option 2 would most likely be the ideal option since it would be easier than option 1 and having a clean install would help ensure Qubes packages are not being installed onto an already compromised system.  As with Qubes, it may not really matter what the KVM host OS distribution is as the plan, if possible, would to disable networking on the host and provide the same or similar GUI management VM.

> - *qubes-builder*:
> - For some reason I can not build with 'DIST_BUILD_TOOLS=1" (standard
> qubes xen components). I always get an error when building dom-fc32 of
> "sudo: unrecognized option
> '--resultdir=/home/user/qubes/chroot-dom0-fc32/home/user/qubes-src/vmm-xen/pkgs/dom0-fc32'.
> Am I missing another config option?

Missing "mock" package in your build environment?

You are right, the 'mock' package is not installed.  Wonder why 'make build-deps' never picked it up.

> - Libvirt often fails to compile using 32 cores giving some error
> about some file that does not exist (when it does fail, it always fails at
> the same spot with same error message).

I guess some missing dependency in one of the Makefiles.

> It seems to be compiling too fast
> or maybe has something to do with using BTRFS filesystem. The rpm spec for
> libvirt uses the number of processors available (make -j32 V=1). It will
> build without errors if I generate a '.rpmmacros' file containing
> '%_smp_mflags' -j10' to the 'chroot-dom0-fc32/home/user' directory. Just
> wondering is there is a way to set number of jobs per component, or maybe
> switching to using 'DIST_BUILD_TOOLS' will help.

I would start with reporting this specific build issue on libvir-list.
It's worth also checking with upstream libvirt build, straight from git
clone master branch.

Ok, it definitely seems like either libvirt or rpm build issue.

Thanks for quick feedback.  I will keep this thread updated with progress after hitting milestones.

Do you have any suggestion on the best way to manage this project?  Should you or I create a qubes-kvm repo or something where issues, todo lists, milestones and discussion can happen that is separate from the mainline Qubes issues?  I'm just thinking something like that may be useful to allow others to track progress and maybe offer to contribute to areas within todo list and be able to offer expertise dealing with specific task items.  Also would it be better to add a qubes-kvm-developer section in the forum to allow separate threads for kvm related discussion or keep it here?  Either option is good more me.

[Chris Laprise]

On 7/31/20 5:17 PM, Jason M wrote:
>
> I have begun work on porting Qubes to work within a KVM host. I need a
> development environment that can utilize the cuda cores on a secondary
> Nvidia RTX GPU and also prefer to be able to utilize the graphics card.

Hi Jason,

Slightly OT question: Is there a way you could use OpenCL instead? Does
that make it any easier?

When I see FOSS users doing backflips for Nvidia I get worried bc Nvidia
compatibility (of various sorts) tends to consume a lot of user
attention and effort which eventually contributes to ppl migrating away
from Linux.... meanwhile Linus Torvalds is reminding ppl that Nvidia is
an uncooperative entity.

I think its more productive to focus on what other hardware vendors
offer; Less heat more light.

-

Here are links to a CUDA-to-OpenCL tool and AMD's GPUopen site:

https://github.com/hughperkins/coriander

https://gpuopen.com/


--
Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[donoban]

WoW,

It is really nice to have someone working on this!

On 2020-08-01 17:39, Jason M wrote:
>
>
> One thing to consider is also enabling memory deduplication in KVM
> (KSM). This should nicely save memory when running multiple similar
> VMs,
> but at the same time is risky in light of speculative execution and
> also
> rowhammer-style attacks.
>
>
> Personally I would be concerned of the risk, but I guess it could be an
> option for users with low memory.

Just for mention some idea (pretty off topic), I think that this kind of
security risk could be highly reduced with a simple Qubes RPC which
pauses all (or near all) running VM's when some critical data is
unencrypted in memory.

In example, a cold electrum wallet with an offline VM (priv-vm) with
encrypted private keys and another with just public keys:

- When you want to sign a transaction on the offline VM, it requests to
dom0 to stop the another and waits.
- dom0 pauses all needed VM's (maybe asks for confirmation)
- priv-vm now asks for the password of the seed, decrypts it, signs the
transaction and wipes the memory.
- finally it tells dom0 to resume all VM's again.

I am not sure if it is a pretty crazy idea.

[Jason M]

On Saturday, 1 August 2020 12:23:21 UTC-4, Chris Laprise wrote:

On 7/31/20 5:17 PM, Jason M wrote:
>
> I have begun work on porting Qubes to work within a KVM host. I need a
> development environment that can utilize the cuda cores on a secondary
> Nvidia RTX GPU and also prefer to be able to utilize the graphics card.

Hi Jason,

Slightly OT question: Is there a way you could use OpenCL instead? Does
that make it any easier?

When I see FOSS users doing backflips for Nvidia I get worried bc Nvidia
compatibility (of various sorts) tends to consume a lot of user
attention and effort which eventually contributes to ppl migrating away
from Linux.... meanwhile Linus Torvalds is reminding ppl that Nvidia is
an uncooperative entity.

I think its more productive to focus on what other hardware vendors
offer; Less heat more light.

I totally agree.  Right now OpenCL is not an option for my current needs but may be in the future.

Corporations like Nvidia are completely motivated by greed it seems.  Now they have products that customers need, but soon the competition will be catching up and because of their business practices I see them losing the majority of market share. I have once company I have done consulting for that had just recently gone with a competitor for these very reasons, which meant Nvidia just loss of over 2 million in current sales, not to mention future sales.  Nvidia states they are a software company, so as open source software catches up they will lose their edge.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

TBH I don't think "behind enterprise routers" really helps them. In many
cases (all the cloud providers) the untrusted traffic comes from within
the VMs, not only from outside.

> > One idea is to use netdev socket to connect two VMs directly, but I
> > worry about the performance...
>
> We could also reimplement the Xen netfront/netback protocols on top
> of KVM shared memory. Future versions of KVM might even have direct
> support for Xen paravirtualized drivers.

While this would be a nice development, I think it *way more* complex
development that is realistic in the short term future and without a
specific directed funding for that.

As for the KVM upstream support for Xen PV drivers, all the plans I've
seen focus on just compatibility (run unmodified Xen VMs on KVM),
ignoring superior Xen security model. Specifically - implement backend
drivers with the assumption of full guest memory access and have them
only on the host system. There is also dual topic discussed - virtio
support on Xen, and suffers from the same issue. Everyone says avoiding
full guest memory access (and being able to have backend in another VM)
is nice to have that no one is willing to work on :/

> > One thing to consider is also enabling memory deduplication in KVM
> > (KSM). This should nicely save memory when running multiple similar VMs,
> > but at the same time is risky in light of speculative execution and also
> > rowhammer-style attacks.
>
> Honestly, I don’t think that deduplication is worth it, especially
> in light of TRRespass. It also makes side-channel attacks far, *far*
> easier to exploit.
>
> What *might* be safe is mapping read-only data (such as dom0-provided
> kernels) into multiple VMs

I don't think the gain from just this worth the effort.

> > Yes, no issue as stubdomains are not existing on KVM.
>
> This could be a nasty issue for HVMs,

You mean any VM. If not emulation, then PV backends.

> as without stubdomains, all
> emulation must be done in dom0. Google and Amazon both solved this by
> writing their own VMMs. At some point, KVM might be able to move the
> instruction emulation into userspace, which might be a significant win.
>
> (What I *really* want is a version of QubesOS based on seL4.
> But seL4 didn’t support 64-bit VMs last I checked, and I am not
> aware of any tooling for creating and destroying VMs at run-time.
> Most of the userspace tooling around seL4 is based on CAmkES, which
> requires that every component be statically known at compile-time.
> Furthermore, getting seL4’s hypervisor and IOMMU support verified
> would either require someone to fund the project, or someone who has
> sufficient skill with the Isabelle proof assistant to do it themselves.
> Without verification, the benefits of using seL4 are significantly
> diminished.)

Well, seL4 kernel is still significantly smaller and delegate much more
tasks to less trusted components. But I agree - for the current features
of Qubes, the static nature of seL4 is quite limiting.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl8l5nYACgkQ24/THMrX
1yxmzwf9Fe6P+Ar/U9NvJr7p8543XANXKHIi+Qj7wNcRSbd6cCHCkKXBoGTej+70
51PNcd7xRkYQmIwgcypgGg4Ml9GHD7GMPeP7bS73LUdMcnfa5t3xloxqJMPHGInD
/ccUCQzqAsSA3DIudO6jmKTZ07h7u+PA83axPzSxKaVExDvHLZHFXCkUJ9OcETP3
XY/4uPleCAeVCvV+udDqzv17Tun9ylwgdNpGzehqx4n1CcMh5cC7HtPnV9j1gIH/
OGL/yrIMS20vz01fJx6ic1zSTycODJA7JGT70mpEm8FcvnoIHWsLUm2GB+iF3iQo
gV6GTQYwB40sE2ki9e3LvT8ybf2xPQ==
=lh96
-----END PGP SIGNATURE-----

[Demi M. Obenour]

On 2020-08-01 11:32, Demi M. Obenour wrote:
> We could also reimplement the Xen netfront/netback protocols on top
> of KVM shared memory. Future versions of KVM might even have direct
> support for Xen paravirtualized drivers.

To avoid any confusion, I am not talking about running Xen on top of
KVM, but rather implementing the Xen paravirtualized driver protocols
(or something similar) on top of KVM.

Sincerely,

Demi

[Demi M. Obenour]

On 2020-08-01 18:02, Marek Marczykowski-Górecki wrote:
>> In most KVM setups that I know of, the kernel network stack is
>> considered trusted. That’s a reasonable assumption for production
>> servers, which have server-grade NICs and are behind enterprise
>> routers, but not for Qubes.
>
> TBH I don't think "behind enterprise routers" really helps them. In many
> cases (all the cloud providers) the untrusted traffic comes from within
> the VMs, not only from outside.

That is true. Nevertheless, RCEs in the Linux network stack are
*rare*, especially in LTS kernels. Other than GRO Packet of Death
(which didn’t affect any LTS kernel), I am not aware of a single
such RCE in the last decade. There are likely some vulnerabilities in
exotic protocols like SCTP and AX_25, but exploiting them requires
the relevant modules to be loaded, which they usually are not.
I believe most cloud hosts run very few (if any) network services
that are exposed to untrusted guests, so the userspace attack surface
is likely small as well. Furthermore, cloud hosts don’t need to
implement Wi-Fi or Bluetooth.

>>> One idea is to use netdev socket to connect two VMs directly, but I
>>> worry about the performance...
>
>> We could also reimplement the Xen netfront/netback protocols on top
>> of KVM shared memory. Future versions of KVM might even have direct
>> support for Xen paravirtualized drivers.
>
> While this would be a nice development, I think it *way more* complex
> development that is realistic in the short term future and without a
> specific directed funding for that.
>
> As for the KVM upstream support for Xen PV drivers, all the plans I've
> seen focus on just compatibility (run unmodified Xen VMs on KVM),
> ignoring superior Xen security model. Specifically - implement backend
> drivers with the assumption of full guest memory access and have them
> only on the host system. There is also dual topic discussed - virtio
> support on Xen, and suffers from the same issue. Everyone says avoiding
> full guest memory access (and being able to have backend in another VM)
> is nice to have that no one is willing to work on :/

I am not familiar with virtio internals, but the protocol might be
inherently incompatible with a de-privileged backend. If there is no
explicit setup of backend-writable memory mappings, it probably is.

It is almost certainly possible to implement Xen PV protocols on top
of KVM, but I wonder if it would be better to implement KVM-specific
protocols that map better to KVM’s primitives, in order to reduce
attack surface in the host. Nevertheless, I believe that Xen is
likely to continue to be a better fit in the future. Xen is focusing
on the security-critical embedded space, whereas KVM is mostly used
in datacenters.

>>> One thing to consider is also enabling memory deduplication in KVM
>>> (KSM). This should nicely save memory when running multiple similar VMs,
>>> but at the same time is risky in light of speculative execution and also
>>> rowhammer-style attacks.
>
>> Honestly, I don’t think that deduplication is worth it, especially
>> in light of TRRespass. It also makes side-channel attacks far, *far*
>> easier to exploit.
>
>> What *might* be safe is mapping read-only data (such as dom0-provided
>> kernels) into multiple VMs
>
> I don't think the gain from just this worth the effort.

Agreed.

>>> Yes, no issue as stubdomains are not existing on KVM.
>
>> This could be a nasty issue for HVMs,
>
> You mean any VM. If not emulation, then PV backends.

No, I meant HVMs specifically. PV backends can run in full domains,
as they do now.

>> as without stubdomains, all
>> emulation must be done in dom0. Google and Amazon both solved this by
>> writing their own VMMs. At some point, KVM might be able to move the
>> instruction emulation into userspace, which might be a significant win.
>
>> (What I *really* want is a version of QubesOS based on seL4.
>> But seL4 didn’t support 64-bit VMs last I checked, and I am not
>> aware of any tooling for creating and destroying VMs at run-time.
>> Most of the userspace tooling around seL4 is based on CAmkES, which
>> requires that every component be statically known at compile-time.
>> Furthermore, getting seL4’s hypervisor and IOMMU support verified
>> would either require someone to fund the project, or someone who has
>> sufficient skill with the Isabelle proof assistant to do it themselves.
>> Without verification, the benefits of using seL4 are significantly
>> diminished.)
>
> Well, seL4 kernel is still significantly smaller and delegate much more
> tasks to less trusted components. But I agree - for the current features
> of Qubes, the static nature of seL4 is quite limiting.

seL4 is actually fully dynamic ― it is the userspace tooling that
is static. The seL4 developers have expressed interest in using seL4
in dynamic systems, and there are others who are working on building
dynamic systems on top of seL4, so I suspect this limitation will be
lifted someday. QubesOS is explicitly listed on the seL4 website as an
area where seL4 is a much better fit than Xen, so the seL4 developers
might be willing to cooperate on a seL4-based version of Qubes.

The SMP version of seL4 also has a global lock. This should not be
much of a problem as seL4 system calls are *very* fast. The one
exception is object destruction, but it makes explicit checks to
allow preemption. It would be a problem on large NUMA systems,
but my understanding is that QubesOS is rarely used on such systems.
Future versions of seL4 will support a multikernel mode of operation
that avoids this limitation.

Sincerely,

Demi

[Andrew “Arthur” Summers]

I know this is rather old, so I apologize for the necro. Has there been any progress on this effort? There is a lot of flexibility that KVM would bring (especially for passthrough support), and I would be interested in helping test. I'm a web developer, so my ability to contribute is a bit difficult, but I'll do what I can!

[Mark Reynolds]

Several years ago I worked extensively on a KVM version of Qubes and I am interested in contributing again.

--
You received this message because you are subscribed to the Google Groups "qubes-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/b918d75e-a3cf-4b8e-a3a1-bff387b2c971n%40googlegroups.com.