Qubes-OS general usage

[Olivier Médoc]

Hello,

I'm currently trying to use Qubes-OS as my main system. Qubes-OS setup was easy and most configuration options where easily reachable, which is a good point as it is generally hard to mix ease to use and security.

So thank you a lots for this concept that was really missing. In my daily work, I'm running a VM system (actually vmware, sorry for the inapropriate advertising) and I try to segregate customer data using snapshots, but lot of problems arise when you want to update all snapshots... So Qubes-OS is almost exactly what I need.

Now I'm using it actively since two weeks, and I'm tweaking it to my needs. In this process, I faced several issues, eventually I solved it, and I also have several suggestions.

Here are my unordered through. I won't mind if you don't answer all of them, but it might be interresting for other Qubes-OS users:

• First the UserDoc wiki is good at presenting how to use main Qubes-OS functions, but I would like to see more documentation for stupid things that help users to tweak their settings. For instance, I'm a daytoday linux/gnu/xfce user, but I don't know anything about Fedora nor Gnome and KDE. Maybe a community wiki could help people like me to share tips and tricks on common topics ?

• You present Qubes-OS as a "security oriented" system, but I had to activate the screensaver screen lock function myself, which is a physical attack vector that present bigger risks than DMA hardware attacks ?

• I tried to build Nvidia drivers on Qubes-OS beta, but it failed. Now, it is unsupported by your team. Nouveau seems quite slow (I tried to use composite+desktop effects but I had to disable most of them to make it usable). Additionnaly, my system crashed when waking up from standby-mode, which could be caused by Nouveau. Now, it doesn't seem to crash when the screensaver is effective during standby.

• What are the configuration parameters that are shared between dom0 and the other VMs ? Are there any ?

• I'm currently working on localisation, I try to install most packages for my language both in dom0 (kde-i18n...) and in the template VM (kde-i18n, dictionnaries and spell systems, libreoffice and mozilla language packs...), then I have to setup /etc/sysconfig/i18n to enable my language and UTF-8 both in dom0 and template VM (by the way, is UTF-8 disabled by default in the template VM?). Is it the right process ? Maybe it could be a good topic for the user manual (or the aforementioned community wiki) ?

• Don't be stupid like me and do not share the hard disk controler your system is installed on to an appVM

• I really want to try antievil maid, but I'm currently running a dual boot system. Is it possible ?

• I would like to use exotic network configurations such as 802.1q (which does not seems to be possible in gnome network manager), or to setup a netvm with multiple interfaces bridged with eth0 (to play routing, iptable natting and man in the middle). Maybe somebody has hints to help to setting this up ?

• Setting a network printer for all VMs requires to temporarily allow network access to the template VM.

• I saw HVM is greyed out somewhere in the VM Manager menus, as soon as it is ready, I would like make a template for my favorite Linux distribution, essentially for building my own packages. Are there already any template building documentation ? Same for ... err ... windows
  

• Do you think using a virtualisation system inside an appVM is feasible ? This is essentially for the two or three stupid drivers or applications that are running on windows...
  

Sorry for the brain spamming...

Olivier

[Franz]

Simple practical interesting notes. I also would like to know the answers.
I can offer only one: I successfully installed the network printer for all VMs and the template VM has network access by default.
Best
Franz

[Marek Marczykowski]

On 20.09.2012 11:36, Olivier Médoc wrote:
> Hello,
>
> I'm currently trying to use Qubes-OS as my main system. Qubes-OS setup was
> easy and most configuration options where easily reachable, which is a good
> point as it is generally hard to mix ease to use and security.
>
> So thank you a lots for this concept that was really missing. In my daily
> work, I'm running a VM system (actually vmware, sorry for the inapropriate
> advertising) and I try to segregate customer data using snapshots, but lot of
> problems arise when you want to update all snapshots... So Qubes-OS is almost
> exactly what I need.
>
> Now I'm using it actively since two weeks, and I'm tweaking it to my needs. In
> this process, I faced several issues, eventually I solved it, and I also have
> several suggestions.
>
> Here are my unordered through. I won't mind if you don't answer all of them,
> but it might be interresting for other Qubes-OS users:
>

> * First the UserDoc wiki is good at presenting how to use main

> Qubes-OS functions, but I would like to see more documentation for
> stupid things that help users to tweak their settings. For instance,
> I'm a daytoday linux/gnu/xfce user, but I don't know anything about
> Fedora nor Gnome and KDE. Maybe a community wiki could help people
> like me to share tips and tricks on common topics ?

The are a lot of KDE and Fedora tutorials on the web...

Regarding XFCE, I'm using it on Qubes:
http://git.qubes-os.org/gitweb/?p=marmarek/xfce4-dom0.git;a=summary

It still require some noticable configuration (like application menu), but is
usable after this initial step. Unfortunately installation isn't trivial:
after building xfwm4 (in above repo, also supported by qubes-builder), you
need to manually collect required package list (based on yum groupinfo XFCE
call from VM) and then install them using qubes-dom0-update.

> * You present Qubes-OS as a "security oriented" system, but I had to

> activate the screensaver screen lock function myself, which is a
> physical attack vector that present bigger risks than DMA hardware
> attacks ?

Indeed it is a good idea to have it by default, not only when the user enables
it by hand.

> * I tried to build Nvidia drivers on Qubes-OS beta, but it failed.

> Now, it is unsupported by your team.

Yes, nvidia binary drivers do not support newer kernels with virtualization
enabled (pvops), so we've marked it as obsolete option.

> Nouveau seems quite slow (I
> tried to use composite+desktop effects but I had to disable most of
> them to make it usable). Additionnaly, my system crashed when waking
> up from standby-mode, which could be caused by Nouveau. Now, it
> doesn't seem to crash when the screensaver is effective during standby.

What exactly nvidia hardware do you have? I'm using it on two different models
and everything works. Ah, on one of them (Quadro NVS 140M) I'm using slightly
newer kernel - 3.4.4. It is in "unstable" repo, you can install it by
qubes-dom0-update:
qubes-dom0-update --enablerepo=qubes-dom0-unstable kernel-3.4.4

> * What are the configuration parameters that are shared between dom0

> and the other VMs ? Are there any ?

Keyboard layout and timezone are copied to VM from dom0. I've posted details
here sometime ago.

> * I'm currently working on localisation, I try to install most

> packages for my language both in dom0 (kde-i18n...) and in the
> template VM (kde-i18n, dictionnaries and spell systems, libreoffice
> and mozilla language packs...), then I have to setup
> /etc/sysconfig/i18n to enable my language and UTF-8 both in dom0 and
> template VM (by the way, is UTF-8 disabled by default in the
> template VM?). Is it the right process ? Maybe it could be a good
> topic for the user manual (or the aforementioned community wiki) ?

I think it is a right way. Regarding UTF-8, what country independent default
locale do you propose? I can change it to en_US.UTF-8, but without "_US" would
be nicer...

> * I really want to try antievil maid, but I'm currently running a dual

> boot system. Is it possible ?

Yes, as long as you are using grub as bootloader (which will be changed to
trustedgrub by AEM).

> * I would like to use exotic network configurations such as 802.1q

> (which does not seems to be possible in gnome network manager), or
> to setup a netvm with multiple interfaces bridged with eth0 (to play
> routing, iptable natting and man in the middle). Maybe somebody has
> hints to help to setting this up ?

You need to disable NetworkManager first, it can be done by qvm-service:
http://wiki.qubes-os.org/trac/wiki/Dom0Tools/QvmService

Then you have /rw/config/rc.local for all you need. NetworkManager have set
some qubes-specific hooks:
/etc/NetworkManager/dispatcher.d/30-qubes_external_ip:
- write external IP to xenstore, mainly to trigger firewall reload on IP change
/etc/NetworkManager/dispatcher.d/qubes_nmhook
- set up DNS forwarding (VMs are using virtual IP as DNS, which is DNATed to
real on by NetVM)

> * Setting a network printer for all VMs requires to temporarily allow

> network access to the template VM.

Yes.

> * I saw HVM is greyed out somewhere in the VM Manager menus, as soon

> as it is ready, I would like make a template for my favorite Linux
> distribution, essentially for building my own packages. Are there
> already any template building documentation ? Same for ... err ...
> windows

HVM support isn't available for now in Qubes (at least not in 1.0). Some parts
of the support are already implemented in Qubes Manager (the Qubes Manager is
the same for HVMs), but will remain disabled until we release Qubes with HVM
support.

> * Do you think using a virtualisation system inside an appVM is

> feasible ? This is essentially for the two or three stupid drivers
> or applications that are running on windows...

You will not be able to use VT-x inside of VM, so things will be terribly
slow. But should work, at least tested with qemu. Maybe VirtualBox also will
work without VT-x, but I haven't tried.

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

[Olivier Médoc]

I finally managed to install XFCE (thanks to qubes-builder). It looks
good for me as soon as qubes-manager and notifications are working
correctly. I'm still missing a good mixer appled as xfce-mixer seems to
crash.

>> * You present Qubes-OS as a "security oriented" system, but I had to
>> activate the screensaver screen lock function myself, which is a
>> physical attack vector that present bigger risks than DMA hardware
>> attacks ?
> Indeed it is a good idea to have it by default, not only when the user enables
> it by hand.
>
>> * I tried to build Nvidia drivers on Qubes-OS beta, but it failed.
>> Now, it is unsupported by your team.
> Yes, nvidia binary drivers do not support newer kernels with virtualization
> enabled (pvops), so we've marked it as obsolete option.
>
>> Nouveau seems quite slow (I
>> tried to use composite+desktop effects but I had to disable most of
>> them to make it usable). Additionnaly, my system crashed when waking
>> up from standby-mode, which could be caused by Nouveau. Now, it
>> doesn't seem to crash when the screensaver is effective during standby.
> What exactly nvidia hardware do you have? I'm using it on two different models
> and everything works. Ah, on one of them (Quadro NVS 140M) I'm using slightly
> newer kernel - 3.4.4. It is in "unstable" repo, you can install it by
> qubes-dom0-update:
> qubes-dom0-update --enablerepo=qubes-dom0-unstable kernel-3.4.4

Updating to the last kernel seems to work. I didn't experienced a crash
after resuming from sleep since the update.

>> * What are the configuration parameters that are shared between dom0
>> and the other VMs ? Are there any ?
> Keyboard layout and timezone are copied to VM from dom0. I've posted details
> here sometime ago.
>
>> * I'm currently working on localisation, I try to install most
>> packages for my language both in dom0 (kde-i18n...) and in the
>> template VM (kde-i18n, dictionnaries and spell systems, libreoffice
>> and mozilla language packs...), then I have to setup
>> /etc/sysconfig/i18n to enable my language and UTF-8 both in dom0 and
>> template VM (by the way, is UTF-8 disabled by default in the
>> template VM?). Is it the right process ? Maybe it could be a good
>> topic for the user manual (or the aforementioned community wiki) ?
> I think it is a right way. Regarding UTF-8, what country independent default
> locale do you propose? I can change it to en_US.UTF-8, but without "_US" would
> be nicer...

Is it possible to use 'en.UTF-8' alone ? I'm used to US default so I
don't really care, as soon as I can change it to 'fr' ;-). Just for
information, the file /etc/sysconfig/i18n doesn't exists at all in the
template VM. It can be created without problem.

>> * I really want to try antievil maid, but I'm currently running a dual
>> boot system. Is it possible ?
> Yes, as long as you are using grub as bootloader (which will be changed to
> trustedgrub by AEM).

I finally setup antievilmaid in dualboot (I kept the standard grub on my
hdd mbr, while trusted grub is installed on the usb stick), works nicely.
The only problem is that on boot, I'm asked three time for my encryption
passphrase now (once for swap which doesn't allow the boot process to
continue, a second time for the qubes partition, and then a third time
when mounting swap). Is it possible to improve this ?

By the way, antievilmaid would be great by loading a customized
framebuffer picture (that will be the secret) instead of the fancy qubes
password request splashscreen. I will dig into this if I find the time.
Maybe you have some tips to start ?

>
>> * I would like to use exotic network configurations such as 802.1q
>> (which does not seems to be possible in gnome network manager), or
>> to setup a netvm with multiple interfaces bridged with eth0 (to play
>> routing, iptable natting and man in the middle). Maybe somebody has
>> hints to help to setting this up ?
> You need to disable NetworkManager first, it can be done by qvm-service:
> http://wiki.qubes-os.org/trac/wiki/Dom0Tools/QvmService
>
> Then you have /rw/config/rc.local for all you need. NetworkManager have set
> some qubes-specific hooks:
> /etc/NetworkManager/dispatcher.d/30-qubes_external_ip:
> - write external IP to xenstore, mainly to trigger firewall reload on IP change
> /etc/NetworkManager/dispatcher.d/qubes_nmhook
> - set up DNS forwarding (VMs are using virtual IP as DNS, which is DNATed to
> real on by NetVM)

I found that network manager supports VLAN in the current version (only
from the nmcli). Also, I compiled a git rpm package with bridge support
both in nmcli and nm-applet (the applet is really buggy for VLAN and
Bridge).

I'm still trying to setup a bridged firewall vm with two interfaces.

Now, I've seen that if the vm config filename is changed (both filename
and reference in qubes manager xml file), it is considered as custom and
can be edited. If I add two vifs interfaces in the config file and use
the script /etc/xen/scripts/vif-bridge instead of vif-route-qubes.

I'm getting near the desired results, but currently nothing works :D . I
have to test the following:
- Defining eth0 as a bridge slave doesn't break the standard firewallvm
(it's routed so it should automatically route trafic to bridge0 instead
of eth0 ?)
- Verifying that iptable doesn't interfere with the layer2 trafic
between the bridge slaves
- Verifying that /etc/xen/scripts/vif-bridge works correctly for qubes

>> * Setting a network printer for all VMs requires to temporarily allow
>> network access to the template VM.
> Yes.
>> * I saw HVM is greyed out somewhere in the VM Manager menus, as soon
>> as it is ready, I would like make a template for my favorite Linux
>> distribution, essentially for building my own packages. Are there
>> already any template building documentation ? Same for ... err ...
>> windows
> HVM support isn't available for now in Qubes (at least not in 1.0). Some parts
> of the support are already implemented in Qubes Manager (the Qubes Manager is
> the same for HVMs), but will remain disabled until we release Qubes with HVM
> support.
>> * Do you think using a virtualisation system inside an appVM is
>> feasible ? This is essentially for the two or three stupid drivers
>> or applications that are running on windows...
> You will not be able to use VT-x inside of VM, so things will be terribly
> slow. But should work, at least tested with qemu. Maybe VirtualBox also will
> work without VT-x, but I haven't tried.

Now that I have HVM and almost a bridged firewall VM, I don't need a
virtualisation system anymore. I still see two nice things missing:
- the 'vmware' like snapshots system where a full tree of snapshot can
be used without impacting the disk space by copying the whole vm.
- an easy way to open a service/forward a port to the network (eg: a web
server in the vm "development"). What is the security standpoint on such
a feature ?

[Joanna Rutkowska]

On 11/20/12 12:04, Olivier Médoc wrote:
/.../

>>> * I really want to try antievil maid, but I'm currently running a dual
>>> boot system. Is it possible ?
>> Yes, as long as you are using grub as bootloader (which will be
>> changed to
>> trustedgrub by AEM).
> I finally setup antievilmaid in dualboot (I kept the standard grub on my
> hdd mbr, while trusted grub is installed on the usb stick), works nicely.
> The only problem is that on boot, I'm asked three time for my encryption
> passphrase now (once for swap which doesn't allow the boot process to
> continue, a second time for the qubes partition, and then a third time
> when mounting swap). Is it possible to improve this ?
>

Interesting -- if you use the same passphrase for all the Qubes
partitions (which is what the installer sets up by default), then you
should only be asked once.

> By the way, antievilmaid would be great by loading a customized
> framebuffer picture (that will be the secret) instead of the fancy qubes
> password request splashscreen. I will dig into this if I find the time.
> Maybe you have some tips to start ?
>

We don't, but I would love to accept a patch for that.

Another thing that would be cool would be to integrate tboot with Anti
Evil Maid -- one of the advanatges of tboot/TXT vs. TrustedGrub (SRTM)
is protection against cold boot attacks (via SCLEAN module and MCH
locking when TXT was not cleary exited).

/.../

> Now that I have HVM and almost a bridged firewall VM, I don't need a
> virtualisation system anymore. I still see two nice things missing:
> - the 'vmware' like snapshots system where a full tree of snapshot can
> be used without impacting the disk space by copying the whole vm.

Can you elaborate a bit more on the usage of such a feature?

> - an easy way to open a service/forward a port to the network (eg: a web
> server in the vm "development"). What is the security standpoint on such
> a feature ?

I think we could allow such a feature on an opt-in basis. Patches welcome :)

joanna.

[Marek Marczykowski]

Interesting, it's working for me. Is it repetable? Can you give some details?
Most preferably backtrace:
gdb xfce4-mixer:
> r
(wait for crash)
> bt

(perhaps you need to install gdb in dom0 first)

>>
>>> * I would like to use exotic network configurations such as 802.1q
>>> (which does not seems to be possible in gnome network manager), or
>>> to setup a netvm with multiple interfaces bridged with eth0 (to play
>>> routing, iptable natting and man in the middle). Maybe somebody has
>>> hints to help to setting this up ?
>> You need to disable NetworkManager first, it can be done by qvm-service:
>> http://wiki.qubes-os.org/trac/wiki/Dom0Tools/QvmService
>>
>> Then you have /rw/config/rc.local for all you need. NetworkManager have set
>> some qubes-specific hooks:
>> /etc/NetworkManager/dispatcher.d/30-qubes_external_ip:
>> - write external IP to xenstore, mainly to trigger firewall reload on IP
>> change
>> /etc/NetworkManager/dispatcher.d/qubes_nmhook
>> - set up DNS forwarding (VMs are using virtual IP as DNS, which is DNATed to
>> real on by NetVM)
> I found that network manager supports VLAN in the current version (only from
> the nmcli). Also, I compiled a git rpm package with bridge support both in
> nmcli and nm-applet (the applet is really buggy for VLAN and Bridge).
>
> I'm still trying to setup a bridged firewall vm with two interfaces.
>
> Now, I've seen that if the vm config filename is changed (both filename and
> reference in qubes manager xml file), it is considered as custom and can be
> edited.

Indeed, smart!

> If I add two vifs interfaces in the config file and use the script
> /etc/xen/scripts/vif-bridge instead of vif-route-qubes.
>
> I'm getting near the desired results, but currently nothing works :D . I have
> to test the following:
> - Defining eth0 as a bridge slave doesn't break the standard firewallvm (it's
> routed so it should automatically route trafic to bridge0 instead of eth0 ?)
> - Verifying that iptable doesn't interfere with the layer2 trafic between the
> bridge slaves
> - Verifying that /etc/xen/scripts/vif-bridge works correctly for qubes

Ensure also that you have still disabled scatter gather in firewallvm on eth0
(ethtool -K eth0 sg off). It doesn't work on xen PV network interfaces.

[Outback Dingo]

You might also consider using open-vswitch and integrating it as it does allow for a much more involved networking and routing configuration, extending the qubes-os to also become more viable for research, prototyping and providing flexibilities. its essentially what XEN, XenServer and XEN Cloud platform use.

[abb]

On Tuesday, November 20, 2012 8:18:33 PM UTC+1, Outback Dingo wrote:

You might also consider using open-vswitch and integrating it as it does allow for a much more involved networking and routing configuration, extending the qubes-os to also become more viable for research, prototyping and providing flexibilities. its essentially what XEN, XenServer and XEN Cloud platform use.

By the way, you can write controllers for open-vswitch in python as well, http://www.noxrepo.org/pox/about-pox/.